# Foundations for Modeling University Curricula in Terms of Multiple Learning Goal Sets

Richard
Judy
Tim

Pages: pp. 25-37

Abstract—It is important, but very challenging, to design degree programs, so that the sequence of learning activities, topics, and assessments over three to five years give an effective progression in learning of generic skills, discipline-specific learning goals and accreditation competencies. Our CUSP (Course and Unit of Study Portal) system tackles this challenge, by helping subject teachers define the curriculum of their subject, linking it to Faculty and institutional goals. The same information is available to students, enabling them to see how each subject relates to those goals. It then gives additional big-picture views of the degree for the academics responsible for the whole degree, including the ability to easily assess if a degree meets accreditation requirements. CUSP achieves this by exploiting a lightweight semantic mapping approach that gives a highly flexible and scalable way to map learning goals from multiple internal and external accrediting sources across the degree. We report its validation as used in a live university environment, across three diverse faculties, with 277 degrees and 7,810 subject sessions over a period of three years. Data from this evaluation indicates steady improvement in the documentation of the relationships between subjects, assessments, learning outcomes, and program level goals. This is driven by the reporting tools and visualizations provided by CUSP, which enable program designers and lecturers to identify parts of the curriculum that are unclear. This improved documentation of the curriculum enables more accurate and immediate quality reviews. Key contributions of this work are: a validated new approach for curriculum design that helps address the complexity of ensuring learners progressively develop generic skills; and a validated lightweight semantic mapping approach that can flexibly support visualizing the curriculum against multiple sets of learning goal frameworks.

Index Terms—Curriculum mapping, graduate attributes, accreditation competencies, learner model

## Introduction

University degree programs typically aim to build learners' generic skills, such as written and spoken communication, team work, design, and problem solving. These are highly valued both within learning institutions [ 1], [ 2] and by outside groups, notably employers [ 3], [ 4]. Learners need to develop these skills progressively, over several years, aided by a suitable sequence of learning experiences [ 5]. To achieve such long term learning over a whole degree program, designers of each subject must appreciate how their subject fits into the full curriculum.

Also, those responsible for each degree program must ensure that generic skills are developed via a series of learning activities across subjects. This is quite complex, especially where students have flexibility to select elective subjects that match their background, interests, and goals [ 6].

Despite the importance of learning generic skills, it is difficult to rigorously classify the skills learned in each subject. For this, we need to define two aspects: the generic skill; and the proficiency level of that skill. While there has been some research involving fine-grained ontological models for learning design, such as [ 7], this approach is not adequate for our goals to model long term learning of generic skills across the many subjects that span the three to five years of a degree program.

A central problem is that the semantic model describing the learning progression must be agreed upon and used by several groups of people. First, the lecturer responsible for teaching a particular subject must understand just what is required from their subject; otherwise they may fail to keep it true to the curriculum. Second, people at the faculty level must understand the curriculum design well enough to assess if it does develop the faculty's required generic attributes. Outside the university, accreditation/regulatory bodies must be convinced that their stated learning requirements are being met.

Importantly, universities, accreditation, regulatory, and professional bodies each define their own learning goal sets. While these bodies may attempt to ensure general alignment with other existing standards, the learning goal sets defined by each vary in their descriptors, granularity, specificity, and structure.

For example, the Bachelor of Engineering in Software Engineering BE(SE) degree at the University of Sydney in Australia needs to anticipate learning goal requirements as defined in:

1. University of Sydney Faculty of Engineering Graduate Attributes [ 8].
2. Engineers Australia (EA) Stage 1 Competency Standards [ 9].
3. ACM/IEEE Software Engineering Curriculum Guidelines 2004 [ 10].
4. ALTC LTAS Threshold Learning Outcomes for Engineering and ICT [ 11].
5. Tuning-AHELO Expected/Desired Learning Outcomes in Engineering [ 12].
6. Wordwide CDIO Initiative, Standard 2—Syllabus Outcomes [ 13].

Note the different terminology: attributes, competencies, learning outcomes, syllabus outcomes, etc. For the rest of the paper, we refer to these interchangeably, or simply as learning goals. These will be discussed in greater detail in the background and related work section.

Flexibility in subject choices creates another challenge for curriculum design that integrates these learning goals throughout the curriculum. The level of prescriptiveness in subject sequence varies from degree-program to degree-program and from university to university. Any allowed elective subjects in a particular degree program must enable the student to achieve the required set of learning goals, and in an effective sequence that enables progressive learning.

Our Bachelor of Engineering in Software Engineering BE(SE) degree is four years long. It requires students to complete 192 credit points of subjects, 60 credit points of which are elective choices (a subject is usually six credit points). The BE(SE) degree program coordinator must thus ensure that the set of learning goals identified above is covered and assessed at an appropriate level, in an effective progressive sequence, irrespective of which combination of elective choices students wish to make. This is a difficult task, both for curriculum design and curriculum maintenance.

At an institutional level, this curriculum mapping task would need separate replication for each of the different degree programs on offer. For example, in 2012, the University of Sydney offered over 600 degree programs and ran over 13,000 subject sessions. 1 Many of these 600 degree programs must meet distinctive learning goal requirements, stipulated by each faculty and by relevant discipline specific accreditation/regulatory bodies.

We now describe the high-level approach, we have taken to address this problem of modeling learning goals throughout entire degree programs. The next section describes related work, followed by our approach and the user view of our Course and Unit of Study Portal system (CUSP, where Course refers to a degree program, and Unit of Study refers to a semester long subject). We then report its validation and success in a live environment over a period of three years. We conclude with a discussion of the strengths and limitations of this work.

## Background and Related Work

This section describes the nature of curriculum design for generic skills through the several years of university degrees. It presents a view of this task in Fig. 1 in terms of the many drivers for the design of a degree curriculum: some are institutional and others are external. It shows the stakeholders who play a role in definition the learning goals that the curriculum must meet and the others involved in the teaching and learning. This captures the extent and complexity of the problem we aim to address in providing new support for curriculum design. We then summarize previous work in terms of the various aspects.

Figure    Fig. 1. Illustration of complexity of learning goal requirements for two degree programs.

### 2.1 Complexity of Learning Goals in University Curricula

Fig. 1 illustrates the complexity of learning goal sources and mappings to university curricula and degree programs. In the top center is a typical University which defines a University Graduate Attribute Policy statement. As an example, the Graduate Attribute Policy at the University of Sydney defines five generic skills that all graduates are expected to develop through the completion of any degree program [ 14]. For example, “Communication - Graduates of the University will use and value communication as a tool for negotiating and creating new understanding, interacting with others, and furthering their own learning.”

This University graduate attribute policy statement is then inherited by the individual faculties within the university, where each faculty contextualizes the original goals to its own discipline. As an example, the Faculty of Engineering at the University of Sydney defines seven graduate attributes [ 8], which relate to but extend upon the five University attributes from above. To illustrate, the Faculty of Engineering redefines Communication as “Proficiency in organising, presenting and discussing professional ideas and issues in oral, written and graphic formats,” and defines an additional attribute for Professional Conduct and Teamwork which is described as “Conducting oneself professionally, functioning as an effective team member and exercising appropriate values, standards and judgement, consistent with requirements of economic, social and environmental sustainability.”

Each faculty within the University offers a number of different degree programs; all should develop this set of faculty graduate attributes in all students who complete each program. Fig. 1 shows a Bachelor of Engineering in Software Engineering BE(SE), which must teach the seven Engineering graduate attributes referenced above.

Now consider the external influences represented in the left box of the Fig. 1. In Australia, all undergraduate engineering degrees are accredited by Engineers Australia against the Engineers Australia Stage 1 Competency Standards [ 9]. This document defines 16 competencies, some examples of which include: “Effective oral and written communication in professional and lay domains”; “Effective team membership and team leadership”; “Ethical conduct and professional accountability,” etc. As evident, these show some strong semantic similarities to the internal Faculty attributes from above.

All BE(SE) graduates from the University of Sydney are thus required to develop the seven Faculty of Engineering graduate attributes, and also the Engineering Australia Stage 1 Competency standards listed in [ 9].

In addition to this, in many cases, there are further pressures on the design of the curriculum, such as in-demand employability skills or other external definitions of desirable learning outcomes. For example, the ACM/IEEE/AIS Curriculum Guidelines for Undergraduate Degree Programs in Software Engineering [ 10] lists seven high-level student outcomes (which also share strong semantic similarities to the graduate attributes and accreditation competencies from above). For example, “Work as an individual and as part of a team to develop and deliver quality software artifacts” and “Demonstrate an understanding and appreciation for the importance of negotiation, effective work habits, leadership, and good communication with stakeholders in a typical software development environment.” These must also be integrated into our BE(SE) curriculum.

Progressing further down on the left in Fig. 1, we see examples of National and International bodies that may define yet more sets of learning goals. As an example, the Australian Learning and Teaching Academic Standards (LTAS) project is a new initiative funded by the Austalian Government to create a set of discipline-specific Threshold Learning Outcomes (TLOs) as part of reforms in higher education quality assurance [ 15]. The Engineering and ICT draft standards document lists a set of five high-level learning outcomes [ 16], e.g., “Coordination and communication - Communicate and coordinate proficiently by listening, speaking, reading and writing English for professional practice, working as an effective member or leader of diverse teams, using basic tools and practices of formal project management.”

Further still, there is growing international recognition of the importance of meeting these challenges of curriculum design. Projects such as AHELO are working toward creating “a robust approach to measuring learning outcomes in ways that are valid across cultures and languages, and across the diversity of institutional settings and missions” [ 17]. A list of proposed standards in the Engineering discipline has been created and published in collaboration with the European Tuning project in [ 12]. This document lists 21 learning outcomes, grouped under five top-level categories, some examples of which include: “The ability to function effectively as an individual and as a member of a team”; “The ability to use diverse methods to communicate effectively with the engineering community and with society at large,” etc.

Yet another global effort to standardize the engineering curriculum is the CDIO Syllabus [ 18], [ 19]. This Syllabus lists yet another a set of intended learning outcomes under four top-level categories, examples of which include: “Team goals and objectives”; “Team process management”; “Representing the team to others,” etc.

So far we have discussed only the Engineering discipline within the University of Sydney as a concrete example. Similar situations may be found in professional programs more broadly within universities, where division of curriculum authority and the multiplication of standards and regulatory frameworks frequently prevails. Accounting degrees for example, may be subject within Australia to the standards of CPA Certified Practicing Accountants Australia (CPA), 2 Institute of Chartered Accountants (ICAA), 3 Institute of Public Accountants (IPA), 4 Association to Advance Collegiate Schools of Business (AACSB), 5 and new TEQSA Threshold Learning Outcomes for Accounting along similar lines to the Threshold Learning Outcomes for Engineering described above. These are represented on the right of Fig. 1.

Nonprofessional university programs may not be engaged with the problems of long term learning goal specification to the same extent as the professional programs, but this does not mean that the challenge in nonprofessional degrees is any less serious than for degrees linked to professional qualifications. The weaker engagement with long-term learning goal specification in the nonprofessional degrees actually makes the problem more serious, in so far as it entails a more limited experience and awareness of the complexities involved. The challenge of learning goal specification is greater in the nonprofessional than the professional programs for a second reason: the extent of differences to be bridged in finding commonality among disciplinary cultures and values. To derive a common set of learning goals in Science for example, encompassing disciplines as distantly related as Geology and Psychology, is a much bigger task than simply trying reconcile different versions of the main learning goals for professional software engineers.

In combined or double-degree cases such the BE(SE)/BC(A) represented in Fig. 1 (e.g., Software Engineering/Commerce-Accounting), a student would be required to be taught, assessed, and accredited on the relevant learning goals from both disciplines. These further compounds the curriculum design challenges for such combined programs.

In summary, the design of a degree curriculum must take account of multiple sets of learning outcomes, defined by various independent bodies. This is very complex, in terms of the number of learning outcomes to be met and the different ontologies and descriptions defined for each by the institution or group which created it. When it comes time for accreditation, it is important to be able to show that the curriculum ensures progressive building toward the learning outcomes of each relevant body.

The bottom of the middle box shows some of the key stakeholders involved. At the left is the curriculum designer who is responsible for the whole degree and must do this, taking account of the required learning outcomes. Next, we show the subject lecturer who must decide just how they will teach that subject; they need to ensure that they do this in ways that will achieve the required long term learning goals for the degree. As shown in the figure, the subject may be part of multiple degrees, making the lecturer's task more challenging. There is also the enrolled student who needs to pick elective subjects for next semester, and the prospective student who is trying to decide which degree program to enrol in and which subjects she or he needs to study.

### 2.2 Need for Big-Picture View of Curriculum

Each stakeholder shown in the figure can benefit from an appropriate overview of the curriculum's long term skill building. Taking a student's perspective, they must enrol in the required core subjects and select from the available elective subjects. Currently, students typically have only subject level details of the different learning goals, without any indication of the different sources that they come from, and different purposes that they were created for. This can cause a disconnect between the subjects studied and student understanding of the real-world relevance of what may otherwise seem to be irrelevant topics, or complete arbitrary assignments [ 20].

From the perspective of the lecturer for each subject, the set of prescribed learning goals are essential for the task of designing the subject's learning experiences. However, even if the lecturer had a major role in designing the curriculum, it is challenging to be mindful of the university graduate attributes plus numerous other potential professional learning goal sets, accreditation competencies, etc., especially as some of these may change over time. There is a serious problem if the lecturer cannot readily see the big-picture view of where his subject fits in to the overall curriculum. This means that a subject lecturer may, for example, drop a major group assignment as he or she prefers to have an end-of-semester written exam which better assesses the theoretical aspects of the subject. However, perhaps the original intention of that group assignment was to cover some of the generic transferrable skill learning goals, such as teamwork, leadership, and communication. Dropping this component of the subject could lead to a learning goal gap in the overall curriculum.

Now consider the needs of the degree or program coordinators who oversee and approve changes to individual subjects, to allowed elective choices and overall degree sequencing. With the large number of different learning goals from different sources, and the additional complexity of cases like combined/double degrees, how is a degree program coordinator expected to perform this critical task without supporting tools?

### 2.3 Who has the Big-Picture

Responsibility for ensuring the educational quality of the university programs is shared among a number of higher agencies, the university itself in first place, along with government and the various independent accrediting bodies for professional degree programs. No existing agency, however, is in a position for systematic overview of learning goal delivery at the level of detail described above.

Universities themselves lack the infrastructural capacity for systematic monitoring of learning goal delivery in their teaching programs. Tracking of learning goals even at the most generic level, that of the graduate attributes that are supposedly acquired by all graduating students, has proven insurmountably complex for Australian universities. University documentation of graduate attribute development within degree programs is superficial at best [ 21]. External accrediting bodies, such as Engineers Australia are necessarily confined to their specific accreditation mandate. It would be unrealistic to expect accreditation reviewers to concern themselves with any other learning goals apart from those for which they are specifically commissioned. Further limitations on external accreditation bodies are the time and expense of the accreditation process and the long intervals between accreditation review cycles, typically five years for Australian engineering degrees [ 22].

The government role in formulation and administration of university learning goals has grown substantially through recent minimum standards initiatives such as Tuning and AHELO internationally and the Learning and Teaching Academic Standards (LTAS) project within Australia [ 15]. The deployment of the new minimum goal specifications emerging from these projects is at a very early stage, however, with monitoring arrangements still under exploratory discussion and no definite plans in place [ 23]. In the meantime, the practical effect of these initiatives has simply been to increase the range of learning goal specifications that university curricula may need to take into account, rather than bringing those that already exist within an integrated framework. While the integration of existing learning goal frameworks from different sources has clearly been a major consideration in the drafting of new goal specifications such as the LTAS Threshold Learning Outcomes [ 15], the connections are only loosely defined and do not allow the replacement of older frameworks to be assumed with confidence. The range of pre-existing curriculum goals and sources that are addressed through the new government sponsored specifications is, in any case, a limited one. The LTAS Threshold Learning Outcomes for Engineering and ICT [ 16], for example, address the prior learning goal frameworks of Engineers Australia (EA), the International Engineering Alliance (IEA), the Australian Computer Society (ACS), and the Australian Council of Professors and Heads of Information Systems (ACPHIS) but do not explicitly recognise those of subdisciplines such as Software, Chemical, or Civil Engineering. The perspectives of outside disciplines contributing to engineering and information technology combined degrees is also absent. Without a broad engagement across the full diversity of learning expectations facing university programs, new initiatives in this area only add to the complexity of the field rather than making it more manageable, even when coming with the highest authority and the best of intentions.

So the question remains, who has the big-picture view of the curriculum? And who can say how much learning goal coverage a particular degree program achieves? How can the curriculum be introspected to derive these answers? What happens when a new learning goal framework is introduced, such as the LTAS TLOs or the AHELO learning outcomes, and existing degree programs need to be updated and accredited against these?

### 2.4 Need for Curriculum Mapping Infrastructures

The need for better support for integrating learning goals into university degrees is recognized by many, including Mulder et al. [ 24] who discuss the growing need for standards-based design of university curricula in Europe. They report on various projects from England, Germany, France, and Netherlands, noting the need for better quality control and integration of learning goal frameworks, but the lack of supporting technology to achieve this. This is also identified as a serious problem by McKenney et al. [ 25], who reiterate the need for better tools to support curriculum designers.

Koper [ 26] explored approaches to modeling curriculum elements via a metamodel, in Educational Modeling Language 6 (EML). With an e-Learning focus, subjects were represented as collections of reusable learning objects (LOs). This is important support for fine-grained design at the subject level. But it is unclear how this approach can address the degree-level curriculum design. While various other modeling standards (e.g., IEEE LOM, IMS LIP, SCORM, HR-XML, IMS-RDCEO) deal with parts of a whole degree, they do not help with the degree design complexity problems or multiple attribute framework semantic mapping challenges. Few examples can be found of technology based on these standards that tackle the degree level and are practical in a real university environment for generating the much-needed big picture.

Ontological approaches to skill and competency mappings have been attempted in various forms by Psych et al. [ 7] (also using EML and IMS-LD), Van Assche [ 27], Paquette et al. in the LORNET TELOS project [ 28] and others. These are promising for the ontological issues that are somewhat similar to our needs to map the disparate collections of learning objectives. However, they cannot meet our goals when scaling to entire university degrees. Paquette et al. express this concern: “what is yet to be proven is that the general approach presented here can be used at different levels by average design practitioners and learners.” Kalz et al. [ 29] also share this view: “the design and implementation of competence ontologies is still a very complex and time-consuming task.”

Bittencourt et al. [ 30] explore use of semantic web technologies to improve curriculum quality and support the design process. They conclude, however, that “a large-scale use of SW for education is still a futuristic vision rather than a concrete scenario” and the implementation of ontologies is sometimes “more an art rather than technology.” Winter et al. [ 31] also realize the strengths and limitations of traditional intelligent tutoring systems with “carefully crafted” content and ontologies versus e-Learning systems that are typically standards based but have “content crafted by normal authors.” To support long-term learning, domain-specific ontologies will need to be mapped to each other but “in a realistic setting...this may be difficult to do” [ 31].

A limited implementation of attribute-to-subject mapping was employed by Calvo et al. [ 32] in their Curriculum Central (CC) system. It had a single attribute framework, to map a large set of subjects to these attributes. However, it could not deal with multiple learning goal frameworks, or with the complexity of elective subject choices.

Bull and Gardner [ 33] mapped multiple choice questions, in several subjects, to UK SPEC Standards for Professional Engineering attributes (UK-SpecIAL). As students completed online questions, the system built open learner models, enabling students to see their learning progress, identify their weaknesses based on assessment results, and identify which subjects could help them strengthen their skill portfolios. This gave students a valuable big-picture view of UK SPEC Standards. However, the system did not support complex degree program structures or multiple learning goal frameworks either. It focused on tagged multiple-choice questions in a small sequence of subjects.

## Design Hypothesis for Complex Mappings

We tackle key aspects of the complexity of curriculum design, including when it must meet both institutional and multiple sets of external goals. Our approach is based on two main strands. First, we have created a system and associated interfaces to enable a subject lecturer to define their curriculum, linking this to institutional goals and defining progression in generic skill development across subjects. Second, we hypothesise that lightweight semantic mappings can deal with multiple sets of external goals. This approach avoids the need to cross-map each subject curriculum with learning goals of every set. This is an important pragmatic concern. This approach risks introducing translation and mapping errors in cases where a subject outcome is mapped to a graduate attribute, which in turn is mapped to one or more other learning goals from different sets. These translation errors may not accurately reflect the learning outcome relationships. We evaluate our hypothesis by measuring the severity and impact of these translation errors. We assess the overall approach in terms of actual use, with data about improved curriculum documentation and maintenance. We do this via a case-study analysis of the CUSP system in a live deployment. This presents a realistic and pragmatic view of the consequences and benefits of the approach, and is a commonly accepted evaluation method in educational technology research [ 34], [ 35], [ 36], [ 37].

Our approach is to create lightweight, two part models, based on skill definitions and level definitions. These support models with the semantic relationships between any sets of skills and levels. This approach seemed promising for our multiple design goals, notably the pragmatics of meeting the needs of teaching staff, institutions, and accreditation. An early version of this approach and an initial validation was presented in [ 38]. We now present an in-depth description of the approach, followed by an evaluation that spans multiple years of use in a live university environment.

Fig. 2 shows our high-level architecture. Taking the institutional goals as the base model, we define Primary Skill Set definitions from the established set of graduate attributes. This choice of using the graduate attributes as the primary skill set is an important decision: we consider the foundation should come from the institution's own goals. In our case, this has just seven top-level attributes, most covering generic skills. For example, one of these attributes is Design: Ability to work both creatively and systematically in developing effective, sustainable solutions to complex practical problems.

Figure    Fig. 2. High-level architecture.

To model progression in learning, we considered the many approaches for describing the level of expertise, including the widely used Bloom taxonomy [ 39] as well as newer approaches, like the neo-Piagetian levels [ 40]. To meet the goals of our system, our approach is based on up to five Levels of proficiency for each attribute. This gives a coarse set of levels for key stakeholders to agree on, both for the levels and for classifying learning activities. This granularity is meaningful to model progression over the three to five years of a typical degree program (that is, each successive semester or academic year may develop skills at a higher level of proficiency).

To incorporate other learning goal frameworks into a degree program, the curriculum designer defines these as Secondary Skill Sets, and then maps them against the base model Primary Skill Set Skill/Level definitions. So, for example, the EA Accreditation Competency statement “experience in personally conducting a major design exercise to achieve a substantial engineering outcome to professional standards” maps to our faculty Design attribute at “Level 3: Engages with a whole systems design cycle in working to general technical specifications.” Additional frameworks can be systematically incorporated into the model by repeating this process. Importantly, addition of each new set of standards requires one mapping process, linked to the institutional set of learning goals.

This means that subject lecturers map their subject learning outcomes and assessments to the institutional base model. They need not be burdened with mapping each learning outcome and assessment to every other learning goal framework. This is an important aspect of our approach. It ensures modest demands on the lecturer, even if their subject is part of many degree programs, each with multiple external learning outcome sets. Our approach means that we can generate a big-picture view of a degree for any of the learning goal frameworks; our system does this by resolving the semantic relationships defined between the primary and secondary Skills and Levels. This is a key strength of the architecture which is critical for its scalability and practicality in a real university environment—that is, minimizing demands on individual subject lecturers.

## CUSP User View

CUSP 7 implements this approach, with interfaces to manage the modeling processes. The left part of Fig. 3 shows the seven high-level goal descriptors for the Faculty of Engineering and IT. The screenshot has expanded the third of these, Fundamentals of Science and Engineering. We can see the description of this attribute, the four levels that have been defined for this attribute, and the Equivalents for Level 1, which map to semantically related competencies from Engineers Australia Stage 1 accreditation standards.

Figure    Fig. 3. Faculty of engineering graduate attribute framework, with floating equivalence editor on the right.

Our approach aims to avoid restrictions on the structure of a skill set. Skills can be arbitrarily nested, or flat. Each skill can be given a code, a label and a description and it can have any number of levels, each with their own descriptions (although we chose to use only up to five levels in defining the primary skill set as discussed above).

Clicking the “E” button next to a skill or level definition brings up the floating Equivalence editor dialog (as seen on the bottom-right of Fig. 3). This enables curriculum designers to define many-to-many semantic relationships between skills or levels from different sets. So, the figure shows that Fundamentals of Science Engineering at Level 1, has 12 equivalents defined to other learning goal frameworks.

Up till now, only one type of relationship was made available, namely parent/child. This design decision was driven by the goal to minimize complexity and reduce data entry overheads. Additional semantic classes can be added as deemed necessary, depending on the granularity of mappings and accuracy required.

We now describe the lecturer view for individual subjects. A lecturer can define a high-level subject outline with information such as a handbook description, prerequisite/prohibition subject requirements, teaching methods and activities, learning outcomes, assessment tasks, resources, and scheduling information. The fields are on the tabs for easy navigation as shown in Fig. 4. This figure shows how a lecturer has linked the 2012 version of their subject to the a set of four graduate attributes, Design, Engineering/IT Specialisation, Communication and Professional Conduct, and Teamwork. Note that this attribute set is an updated version of that shown in Fig. 3, hence the slight differences in naming. This is another important feature of CUSP, as it allows versioning of learning goal frameworks, which sometimes change over a span of multiple years.

Figure    Fig. 4. Attributes linked to a subject and described by lecturers via development methods.

Each of the four attributes in Fig. 4 maps to a specific proficiency level. The lecturer has provided a free-form description stating how the attribute is supported by the subject. This appears on the left. The subject attributes are further mapped (by lecturers) to learning outcomes and indirectly to assessments (each assessment can be mapped to one or more weighted, learning outcome). From the lecturer's perspective, the work they need to do is a very small increment on the basic information they would normally provide in the course outline document for students. CUSP automatically generates this course outline document.

On the degree side, a program coordinator links a degree to any number of learning goal frameworks. Our Bachelor of Engineering in Software Engineering degree links to the Faculty of Engineering Attribute Framework, Engineers Australia Accreditation Stage 1 Competency Standards, AHELO-Tuning, CDIO, and LTAS TLOs. The degree structure is then defined in terms of core and elective subjects, streams, and recommended elective blocks.

We now have multiple learning goal frameworks captured in the system, as well as the semantic relationships between them, the mappings of attributes to subjects, learning outcomes and assessments, and the degree core/elective subject structures. These are all the pieces we need to start building our big-picture view of full three-to-five year degrees.

Fig. 5 shows the generic skills profile of our full Bachelor of Engineering in Software Engineering degree in terms of the 2012 Faculty of Engineering Attribute Framework.

Figure    Fig. 5. Overview of the BE(SE) degree in terms of planned, practiced and assessed attributes.

The left column of the matrix has the seven top-level attributes and the columns show the subjects where each level is taught. For example, for Design, students learn Level 1 aspects in the core subject ELEC1601, and also in elective subjects INFO1003 and ELEC1103.

Next to each subject are three additional columns: Pl./Pr./As. These stand for Planned, Practiced, and Assessed, and signify the different levels at which particular attributes may be incorporated within the subject studied: the “intended curriculum” level verses “delivered curriculum” verses “attained curriculum” [ 41], [ 42], [ 43].

That is, Planned means a subject has listed the attribute as an intended learning goal. Practiced means the lecturer explicitly specified a method for developing the attribute. The descriptor Assessed means that the attribute's development is specifically assessed within the subject. Attributes that are Planned for a particular subject may or may not be Practiced and/or Assessed. For example, in Fig. 5, the subject COMP2129 appears in the second row, for Engineering/IT Specialisation attribute at Level 2. It has boxes showing “Yes” (code=Y) for Planned, Practiced, and Assessed level. The attribute in this particular subject is thus Planned and Practiced but not Assessed. Clicking on a subject takes the user to the full outline describing the precise attribute mappings.

The Framework drop-down box in Fig. 5 allows the user to regenerate the above report in terms of any relevant learning goal framework. That is, a degree coordinator or lecturer preparing for accreditation can easily switch to the EA Stage 1 Competency Standards framework and use the data to identify precisely where each competency is taught and assessed, thus showing compliance to the visiting accreditation panel. The same can be done for any of the other relevant learning goal frameworks for each degree, such as the LTAS TLOs, AHELO standards, or CDIO syllabus outcomes listed previously.

These reports are generated by exercising our semantic equivalence mappings as described in the approach. To add a new framework into the system only requires defining the equivalence relationships between the primary skill set and the new skill set as shown in Fig. 2. This does not require additional data entry at the individual subject level, which is critically important for two reasons: first, subject lecturers are not burdened with having to do tedious mappings to each learning goal framework; and second, a new learning goal framework can be introduced in say 2012, and degrees/subjects from 2011 can be retrospectively visualized against this framework with minimal effort.

The chart visualization in Fig. 6 is another big-picture view of our BE(SE) degree. Along the $x$ -axis we have the seven faculty attributes again. Along the $y$ -axis we have the percentage distribution of each attribute in terms of assessments. That is, the BE(SE) degree devotes roughly 19 percent of all assessment tasks to Discipline Specific Expertise, and only 2 percent to Information Skills. Each column is further broken down into the corresponding attribute levels, which are represented in different shades of gray. A mouse-over reveals the precise percentage distribution of each level.

Figure    Fig. 6. A stacked column chart showing percentage distribution of assessed faculty attributes.

The last bar on the $x$ -axis is labelled Undetermined. This shows the percentage of assessment value across the degree as a whole that is not linked to an attribute. This undetermined component is a product of two factors. The first occurs for incomplete units of study, where details of attributes assessed and/or assessment itself are missing. In this degree, several subjects are taught by the Faculty of Science, which does not yet use CUSP. An important design constraint for the system is that such outside subjects should be handled. Our solution to this is to incorporate their extent but, without the detailed input of that part of the curriculum, it appears only in this undetermined area. An additional contributing case occurs when a subject outline has been entered in CUSP but it is not complete. That is, the subject lecturers have not yet defined the mappings between the attributes and assessment tasks. In such cases, the chart visualization and matrix report are useful tools for identifying these subjects and triggering further investigation to determine the cause of discrepancies. This promotes a natural cycle of real-time curriculum quality assurance and improvement.

## Validation

### 5.1 Deployment

CUSP was deployed at the University of Sydney toward the end of 2009, and by 2010 it was being used to officially host the degree program subject structures, subject outlines, and learning goal framework mappings for the three founding faculties—Engineering, Architecture, and Health Sciences. Since 2009, it has been populated with 30 learning goal frameworks, 1,444 individual skill definitions, 259 degree programs, 2,696 subjects, 7,810 subject sessions, 13,169 learning outcomes, and 8,004 assessment items across the three founding faculties.

The capture of outcomes, assessments, and learning goal relationships has relied upon a combination of lecturer and administrative staff input. Outcome and assessment mappings have been reviewed and adjusted by degree coordinators or other experienced staff wherever possible. Initially, the accuracy and completeness of mappings varied widely from subject to subject and degree to degree. However, the data was sufficient to start generating big-picture review reports immediately. These reports were then inspected and actions were taken to improve the accuracy of the data; the sequencing of subjects and learning activities within subjects; and the alignment of assessments to intended learning goals. New reports are then generated in real-time, and the process is repeated, each iteration leading to improved integration of learning goals and improved curriculum data integrity.

Since 2010, CUSP has been the official public website where all degree/subject outline information is offered for prospective and current students, to aid them in enrolling for each semester of study. This receives up to and over 1,000 unique student visitors per day, who view and download subject outlines, which contain mappings to all the relevant program level learning goals. This indicates substantial regular use by students. Likewise the system is used by 200+ staff members per day from the three partner faculties to update subject outlines and review program structures.

### 5.2 Light-Weight Ontology Mapping

We conducted a test to validate the equivalence mapping approach as described in Section 3. To make this test more effective, we performed it on two very different professionally accredited degrees: a two-year Masters degree in Architecture, and a four-year Bachelor degree in Engineering. Subjects for each degree were mapped against the relevant faculty's primary skill framework by subject lecturers and program designers. The faculty primary skill frameworks were in turn mapped, via equivalence relationships, to secondary frameworks comprised of competency standards required for accreditation in each discipline. These framework equivalence mappings were created by curriculum design experts from each faculty.

A report compiling subject learning outcomes under accreditation competency headings was generated for each degree. A recent graduate of each degree was asked to examine each subject outcome and determine in each case whether it represented a meaningful contribution to the competency descriptor under which it appeared. We used recent program graduates for this evaluation for two reasons: they were very familiar with the degree structure and subjects studied; the validation exercise required a substantial amount of time and focus to perform rigorously, which a program coordinator may not have had. In cases where the match between outcome and secondary competency mapping was not confirmed, the learning outcome mapping to the faculty graduate attribute framework was checked by the relevant program coordinator and the curriculum design expert who defined the equivalence mappings. This was done to determine whether the failure came from original data entry (learning outcome mapped to incorrect generic attribute/level); or from an equivalence mapping error (learning outcome mapped to correct generic attribute/level, but accreditation competency equivalence mapped to incorrect generic attribute level); or an attribute translation error (learning outcome mapped to correct generic attribute/level with correct equivalence mapping, but mismatch with learning outcome). All three failure types were found, as shown in Table 1.

Table 1. Learning Outcomes Matching Accreditation Competency Descriptors as Resolved via Attribute Equivalence Mappingsfor Two Professional Degrees

The Masters degree had a high match ratio between learning outcomes and equivalence attribute mappings (92.28 percent), with only 4.56 percent of mismatches due to attribute translation errors (i.e., loss of context in cross-mapping more granular accreditation competencies to more generic faculty attributes, which are then mapped to more granular subject learning outcomes). The Bachelor degree did not fair as well with only a 49.63 percent match ratio between learning outcomes and accreditation attributes. The primary cause of this low ratio was due to incorrect mappings between learning outcomes and the core faculty attribute framework. The attribute translation failure rate was only 6.99 percent. This degree, related subjects and core faculty attribute mappings were imported from an earlier system which had no accreditation competency equivalences defined, whereas the Masters was a newly created degree and hence had more accurate data.

This validation exercise shows our light-weight approach does not provide perfect mappings between degree subjects and multiple learning goal frameworks. Equivalence translation errors sometimes appear due to the multilevel mapping of skills at different granularities. The mappings are, however, valid to a large extent when data is correctly entered. The outcome report used to perform this evaluation has been integrated into the CUSP system interface, and can now be regenerated at will.

Program coordinators and curriculum experts are able to use these reports to perform similar validation checks when creating or changing equivalence mappings, and also on a periodic basis to ensure quality is maintained. These reports are particularly useful during accreditation rounds as they provide a very rich set of data which shows alignment between subject-level outcomes and the relevant accreditation competencies. This continual process of validation and verification is valuable for long term degree quality control and maintenance.

### 5.3 Curriculum Consistency and Data Integrity

Table 2 shows a statistical summary of data in CUSP over a three year time-span from 2010 to 2012. The statistics presented only include data from the Faculty of Engineering, to show a clearer picture of changes to the curriculum consistency over the three year period since CUSP was launched. The first row shows the total number of subject sessions. That is, each semester is treated as a different session. So 689 sessions in 2010 means 689 subjects were offered during first and second semesters, and summer/winter semesters for that year. The remaining rows show the total and average counts of five key data/relationship types (refer to Table 2 and the architecture diagram in Fig. 2 describing these relations).

Table 2. CUSP Curriculum Data/Relationship Statistics from 2010 to 2012

The averages presented indicate that since 2010, all five data/relationship metrics have increased. That is, subject outlines contain more detailed, fine-grained learning objectives. Assessments are likewise more detailed and itemized. A stronger explicit connection is being made between the learning outcomes and assessments of each unit of study. The number of primary skills mapped to each subject session has likewise increased, and so has the number of attributes mapped to learning outcomes (which are then mapped to the assessments). These increases in data integrity are further reflected in Fig. 7. The chart on the left shows the attribute assessment distribution across a full four year Engineering degree as of 2010. The chart on the right shows the distribution for the same degree program for 2011. As in Fig. 6, the final column in each chart represents the percentage of assessment across the full degree for which attribute alignments are undetermined (an undesirable property). The charts reveal that from 2010 to 2011, the curriculum data for this degree program was improved to reduce the undetermined assessment alignment by 11.3 percent. Similar positive changes can be seen in other degree programs across all three faculties where CUSP is used.

Figure    Fig. 7. Attribute assessment distribution changes from 2010 to 2011 in an engineering degree program.

This presents evidence that CUSP has First enabled us to report on the quality and learning goal framework integration of our degrees in terms of quantitative measures. Second, since CUSP was introduced in our environment, the integrity of curriculum designs has positively increased over a successive three year period. Additionally, we now have the reporting capabilities to quickly inspect our degrees in terms of any of the relevant existing learning goal frameworks. CUSP also makes us well prepared to easily report against the new AHELO standards and LTAS TLOs once they are finalized and mandated.

## Discussion

### 6.1 Summary of Findings

We have described our approach to support design of flexible degrees that are accountable in terms of ensuring that important generic skills and accreditation requirements are met over the full three-to-five year duration. We have implemented this in CUSP and reported its use to map multiple learning goal frameworks to individual degrees, and map learning goals to each core or elective subject that is part of a degree. The CUSP reporting tools give lecturers and degree coordinators a much needed big picture view of entire degrees. This helps identify knowledge gaps, accreditation requirement gaps, and progressive learning inconsistencies.

We have validated our approach by deploying the system on a large scale in a live university environment with real data. The system is in active use with 259 degrees, 7,810 subject sessions and over 30 different learning goal frameworks pertaining to the three founding faculties. From the evidence of Table 1, the simple equivalence mapping architecture used is certainly not a mechanism for eliminating all errors or weakness in curriculum design and documentation, but rather tends to amplify the impact of any errors present or any missing data. In doing so, it provides a sensitive test of quality in all the elements concerned. To this end, the big-picture reports generated by CUSP have been used to drive curriculum review within the faculties involved. The data presented shows a significant increases in the granularity and extent of learning goal integration into degree program structures as a result of these reporting capabilities. This improved documentation of the curriculum allows for more accurate and rapid changes in parts that remain unclear or parts that do not align with program level goals. This picture is supported by the views of the people who are responsible for the degree programs as CUSP has enabled them to see strengths and to identify and follow up problems in the curriculum design.

### 6.2 Limitations and Assumptions

The CUSP system described in this paper models curriculum design in university teaching programs. The paper describes the system's design, motivations, and benefits from the perspective of the large-scale curriculum management. Within the curriculum management area, the paper explains the unique capabilities of the CUSP system in enabling university programs to address diverse sets of internal and external learning goals efficiently and coherently, and shows the practical advantage of this capability for curriculum quality assurance.

This description is not intended as a comprehensive account of the CUSP systems potential as a curriculum tool, but more of an introduction. It is necessary to mention some of the important curriculum questions that could not be adequately addressed within the present paper and briefly explain where they fit into CUSP system's ongoing assumptions and strategy. Further discussion is required in particular regarding the scope for the following:

• Flexible, open-ended curriculum design verses top-down, prescriptive approaches.
• Student participation in learning goal development and progress monitoring.
• Relevance to more free-form generalist university programs as distinct from the more structured professional degree cases discussed so far.
• Tracking of actual student achievement, not just intended learning goals.

The CUSP system works from a basic assumption that clear statements of intended learning are an essential foundation for high quality student-centered curriculum design. Students will be better able to shape and negotiate their own learning expectations when starting from a clear initial idea of the kind of learning that the course provider intends them to achieve [ 44], [ 45].

In practice, some institutional users may perceive the CUSP system as favoring a more prescriptive curriculum approach, with little room for any sort of negotiation or student initiative. In a context where curriculum systems integration is a relatively new idea, and there is little experience in large scale cooperative curriculum models, other than by top-down command and control, new systems risk being stereotyped as prescriptive management tools [ 21], [ 42], [ 46]. The CUSP system assumes that improved quality of information regarding learning requirements will progressively nurture an institutional climate that is more reflective and less prescriptive on curriculum matters, but with a speed of evolution that is hard to predict and readily affected by other institutional factors. While longer-term consequences remain to be seen, a presumption in favor of transparent communication, rather than against, seems more consistent with educational values.

Student participation in the formulation and monitoring of university learning goals has not been a major question for the paper, nor a major feature of the CUSP system so far. This should not be read as any kind of reflection upon the role that students might potentially play in these areas, but more a question of the service range that the CUSP system is currently able to provide. The system's primary business is to ensure that the learning opportunities offered to students are clearly defined in terms of what students may expect to learn. Once a mechanism exists for achieving clearer, more coherent identification of university learning goals, through the CUSP system, student interaction with these goals can be discussed at a more practical level if not automatically resolved. In the meantime, the provision of clear and consistent course information, where such information did not exist before, represents an important enhancement of the student learning experience, if not a complete transformation.

The potential application of the CUSP system to more loosely prescribed nonprofessional type programs was briefly mentioned in the background section above but actual cases have yet to be found. The main obstacle to wider application in these programs, as was previously mentioned, is the lag in learning goal development relative to the professional degrees. This situation is progressively changing through initiatives such as AHELO [ 17] internationally and the Learning and Teaching Academic Standards project [ 47] in Australia, but coming off a lower base. The CUSP system could potentially assist the development of new learning frameworks through accelerated prototyping, testing and cross-disciplinary comparison of proposed draft frameworks. The system could also be supplemented by automated ontology matching and semantic similarity research such as [ 48], [ 49] in mapping of skill-knowledge divisions and priorities within disciplines to guide and stimulate the formulation of new learning goal descriptions. These methods could provide suggestions to the curriculum expert doing the mappings, or help identify mappings that may cause translation errors due to low relatedness. Additionally, student and academic feedback can be incorporated into the system to further tag and validate or flag semantic mappings over time [ 50].

Student achievement is an integral part of established curriculum mapping methods whose core question is the relationship between curriculum goals (the “intended curriculum”) and their realization in practice (the “delivered curriculum” and the “assessed curriculum”) [ 42], [ 43]. The CUSP system is less concerned with how curriculum results match against original intentions and more concerned with the problem of specifying the intended learning itself. It is the complexity of learning goal description in the university sector that imposes this more restricted focus. While specification of intended learning remains fragmentary, ill-defined, and contentious in university teaching as a whole, attempts at curriculum mapping more broadly are likely to be wasted. Effort is better focused in working through the fundamental conceptualisation of learning and teaching priorities rather than extensive audits of curriculum delivery [ 51], [ 52]. Taking a longer term view, the improved mapping of initial curriculum intentions as described in this paper may in the future provide a foundation for more effective analysis of assessment achievement. We envisage for example that CUSP could integrate with an LMS that contains itemized student grades, which could then be used to generate actual learner models showing achieved graduate attributes.

## Conclusions and Future Work

Our work was driven by the need to enable people to overcome the complexity of designing the curriculum for whole degree programs so that they develop students' skills and knowledge in the key learning areas that are defined by institutions and accreditation bodies. All of these areas involve long term learning, over several years, to learn then consolidate that learning and move to higher levels of learning. This requires that the individual subjects each play their role in building these long term broad sets of skills and knowledge. This is an ambitious and challenging goal that is important for the quality of degree programs.

Our approach to designing CUSP drew upon available research. At the same time, we needed to take care to minimize any additional load on lecturers. This led us to an approach based on a light-weight ontological model, and a small number (up to five) levels of proficiency for each learning goal. We use the institution's learning attributes as the master; each other set of learning goals is mapped to it. We have demonstrated that our approach, and its realisation in CUSP, is effective in supporting curriculum design. It enables the lecturer to define their own subject in relation to the institutional learning attributes and to see how it relates to these and their levels. CUSP gives a big picture overview of the whole of a degree and serves as a foundation for improvements to the design of the curriculum, with flexible views across any of the relevant sets of learning attributes. Our key contribution is a new mechanism for supporting the complex and important task of degree level curriculum design.

While CUSP has demonstrated the value of our approach for curriculum designers, at the level of the subject and the degree, we plan to extend our approach to incorporate available assessment data within each subject to create detailed individual student models. To do this, we will move beyond our current mapping of attributes to assessments via learning outcomes. This will allow us to explore the value of personalized attribute progress matrices for students in terms of making more informed subject enrolment decisions, personal reflection and gaining a better understanding of the governing factors influencing their degree. It will also provide a basis for longitudinal data mining of the learner models to improve understanding of the causes of student difficulties.

## Acknowledgments

The CUSP Project is a joint initiative of the Faculties of Engineering and Information Technologies, Health Sciences and Architecture, Design and Planning at the University of Sydney, undertaken with the support by the University's Teaching Improvement and Equipment Scheme (TIES).

## References

• 1. S. Barrie, “A Research-Based Approach to Generic Graduate Attributes Policy,” Higher Education Research & Development, vol. 23, no. 3, pp. 261-275, 2004.
• 2. S.C. Barrie, “A Conceptual Framework for the Teaching and Learning of Generic Graduate Attributes,” Studies in Higher Education, vol. 32, no. 4, pp. 439-458, 2007.
• 3. M. Kavanagh, and L. Drennan, “What Skills and Attributes does an Accounting Graduate Need? Evidence from Student Perceptions and Employer Expectations,” Accounting & Finance, vol. 48, no. 2, pp. 279-300, 2008.
• 4. R. Bridgstock, “The Graduate Attributes We've Overlooked: Enhancing Graduate Employability through Career Management Skills,” Higher Education Research & Development, vol. 28, no. 1, pp. 31-44, 2009.
• 5. D. Bath, C. Smith, S. Stein, and R. Swann, “Beyond Mapping and Embedding Graduate Attributes: Bringing Together Quality Assurance and Action Learning to Create a Validated and Living Curriculum,” Higher Education Research & Development, vol. 23, no. 3, pp. 313-328, 2004.
• 6. T. Lever, D. Auld, and R. Gluga, “Representing and Valuing Non-Engineering Contributions to Engineering Graduate Outcomes in Engineering Combined Degrees,” Proc. 22nd Ann. Conf. Australasian Assoc. for Eng. Education, 2011.
• 7. V. Psych, J. Bourdeau, and R. Mizoguchi, “Ontology Development at the Conceptual Level for Theory-Aware ITS Authoring Systems,” Proc. Conf. Artificial Intelligence in Education, pp. 491-493, http://hal.archives-ouvertes.fr/hal-00190661/en, 2003.
• 8. “Engineering and IT Graduate Outcomes Table,” Univ. of Sydney, Faculty of Eng., http://cusp.sydney.edu.au/attributes/view-attribute-set-pdf/competency, 2012.
• 9. “Stage 1 Competency Standard for Professional Engineering,” Engineers Australia, http://www.engineersaustralia.org.au/sites/default/files/shado/Education/Program, 2011.
• 10. “Software Engineering 2004 Curriculum Guidelines for Undergraduate Degree Programs in Software Engineering,” Assoc. for Computing Machinery, IEEE CS, http://sites.computer.org/ccse/SE2004Vol.pdf, 2004.
• 11. “Engineering and ICT Learning and Teaching Academic Standards Statement,” Australian Learning and Teaching Council, http://www.altc.edu.au/system/files/altc_standards, Dec. 2010.
• 12. “A Tuning-AHELO Conceptual Framework of Expected/Desired Learning Outcomes in Engineering,” Organisation for Economic Co-Operation and Development, http://www.oecd.org/data oecd/46/34/43160507.pdf, June 2009.
• 13. “12 CDIO Standards,” Worldwide CDIO Initiative, http://www. cdio.org/implementing-cdio/standards/12-cdio-standards, 2012.
• 15. C. Ewan, “Disciplines Setting Standards: The Learning and Teaching Academic Standards (LTAS) Project,” Proc. Australian Quality Forum Quality in Uncertain Time, http://www.auqa.edu. au/files/publications/auqf_proceedings_2010.pdf#page=11, 2010.
• 16. “Engineering and ICT: Learning and Teaching Academic Standards Statement,” Australian Learning and Teaching Council, http://www.olt.gov.au/resource-Eng.-ict-ltas-statement-altc-2010, 2010.
• 17. “Assessment of Higher Education Learning Outcomes (AHELO),” Australian Council for Educational Research (ACER), http://www.acer.edu.au/aheloau, 2012.
• 18. E. Crawley, J. Malmqvist, and S. Östlund, Rethinking Engineering Education: The CDIO Approach, vol. 133. Springer Verlag, 2007.
• 19. E. Crawley, J. Malmqvist, W. Lucas, and D. Brodeur, “The CDIO Syllabus v2.0. An Updated Statement of Goals for Eng. Education,” Proc. Seventh Int'l CDIO Conf., 2011.
• 20. P. Avitabile, J. McKelliget, and T. Van Zandt, “Interweaving Numerical Processing Techniques in Multisemester Projects,” Proc. Am. Soc. for Eng. Education Ann. Conf. (ASEE), 2005.
• 21. S. Barrie, C. Smith, C. Hughes, and K. Thomson, “The National Graduate Attributes Project: Key Issues to Consider in the Renewal of Learning and Teaching Experiences to Foster Graduate Attributes,” Australian Learning & Teaching Council, 2009.
• 22. “Accreditation Management System - Education Programs at the Level of Professional Engineer,“ Engineers Australia Accreditation Board, http://www.engineersaustralia.org.au/about-us/ accreditation-management-system-professional-engineers, 2008.
• 23. “Developing a Framework for Teaching and Learning Standards in Australian Higher Education and the Role of TEQSA,” Teaching Education Quality & Standards Agency (TEQSA), http://www.deewr.gov.au/HigherEducation/Policy/teqsa/Pages/default.aspx, 2011.
• 24. M. Mulder, T. Weigel, K. Collins, and B. Bibb, “The Concept of Competence in the Development of Vocational Education and Training in Selected EU Member States: A Critical Analysis,” J. Vocational Education and Training, vol. 59, no. 1, pp. 67-78, 2007.
• 25. S. McKenney, N. Nieveen, and J. van den Akker, “Computer Support for Curriculum Developers: CASCADE,” Educational Technology Research and Development, vol. 50, no. 4, pp. 25-35, 2002.
• 26. R. Koper, “Modeling Units of Study from a Pedagogical Perspective: The Pedagogical Meta-Model Behind EML,” Proc. Ocean Thermal Energy Conversion (OTEC), http://dspace.ou.nl/dspace/handle/1820/36, 2001.
• 27. F. Van Assche, “Linking Learning Resources to Curricula by Using Competencies,” Proc. First Int'l Workshop LO Discovery & Exchange, http://ftp.informatik.rwth-aachen.de/Publications/CEUR-WS/Vol-311/paper11.pdf, 2007.
• 28. G. Paquette, I. Rosca, S. Mihaila, and A. Masmoudi, “TELOS, a Service-Oriented Framework to Support Learning and Knowledge Management,” E-Learning Networked Environments and Architectures: A Knowledge Processing Perspective, p. 434, Springer, http://halshs.archives-ouvertes.fr/hal-00190038, 2007.
• 29. M. Kalz, J. van Bruggen, E. Rusman, B. Giesbers, and R. Koper, “Positioning of Learners in Learning Networks with Content, Metadata and Ontologies,” Interactive Learning Environments, vol. 15, no. 2, pp. 191-200, 2007.
• 30. I. Bittencourt, S. Isotani, E. Costa, and R. Mizoguchi, “Research Directions on Semantic Web and Education,” J. Scientia-Interdisciplinary Studies in Computer Science, vol. 19, no. 1, pp. 59-66, 2008.
• 31. M. Winter, C. Brooks, and J. Greer, “Towards Best Practices for Semantic Web Student Modelling,” Proc. 12th Int'l Conf. Artificial Intelligence in Education Aied (ICAIED), pp. 694-701, 2005.
• 32. R. Calvo, N. Carroll, and R. Ellis, “Curriculum Central: A Portal for the Academic Enterprise,” Int'l J. Continuing Eng. Education and Life Long Learning, vol. 17, no. 1, pp. 43-56, 2007.
• 33. S. Bull, and P. Gardner, “Highlighting Learning Across a Degree within an Independent Open Learner Model,” Proc. Conf. Artificial Intelligence in Education: Building Learning Systems That Care: From Knowledge Representation to Affective Modelling (AIED), vol. 200, pp. 275-282, 2009.
• 34. R. Raitman, N. Augar, and W. Zhou, “Employing Wikis for Online Collaboration in the E-Learning Environment: Case Study,” Proc. Third Int'l Conf. Information Technology and Applications (ICITA '05), vol. 2, pp. 142-146, 2005.
• 35. L. Lockyer, and J. Patterson, “Integrating Social Networking Technologies in Education: A Case Study of a Formal Learning Environment,” Proc. IEEE Eight Int'l Conf. Advanced Learning Technologies (ICALT '08), pp. 529-533, 2008.
• 36. T. Tan, and T. Liu, “The Mobile-Based Interactive Learning Environment (Mobile) and a Case Study for Assisting Elementary School English Learning,” Proc. IEEE Int'l Conf. Advanced Learning Technologies, pp. 530-534, 2004.
• 37. A. Ausserhofer, “Web Based Teaching and Learning: A Panacea?” IEEE Comm. Magazine, vol. 37, no. 3, pp. 92-96, Mar. 1999.
• 38. R. Gluga, J. Kay, and T. Lever, “Modeling Long Term Learning of Generic Skills,” Intelligent Tutoring Systems, V. Aleven, J. Kay, and J. Mostow, eds., vol. 6094, pp. 85-94, Springer, 2010.
• 39. B.S. Bloom, Handbook on Formative and Summative Evaluation of Student Learning. McGraw-Hill, 1971.
• 40. S. Morra, C. Gobbo, Z. Marini, and R. Sheese, Cognitive Development: Neo-Piagetian Perspectives. Psychology, 2007.
• 41. W. Robley, S. Whittle, and D. Murdoch-Eaton, “Mapping Generic Skills Curricula: A Recommended Methodology,” J. Further and Higher Education, vol. 29, no. 3, pp. 221-231, 2005.
• 42. J. Sumsion, and J. Goodfellow, “Identifying Generic Skills through Curriculum Mapping: A Critical Evaluation,” Higher Education Research & Development, vol. 23, no. 3, pp. 329-346, 2004.
• 43. J. Van den Akker, “Curriculum Perspectives: An Introduction,” Curriculum Landscapes and Trends, J. Van den Akker, W. Kuiper, and U. Hameyer, eds., pp. 1-10, Kluwer Academic, 2003.
• 44. J. Biggs, “Enhancing Teaching through Constructive Alignment,” Higher Education, vol. 32, no. 3, pp. 347-364, 1996.
• 45. J. Biggs, and C. Tang, Teaching for Quality Learning at University, third ed. McGraw-Hill/Soc. for Research into Higher Education & Open Univ. Press, 2007.
• 46. B. Oliver, S. Ferns, B. Whelan, and L. Lilly, “Mapping the Curriculum for Quality Enhancement: Refining a Tool and Processes for the Purpose of Curriculum Renewal,” Proc. Quality in Uncertain Times, p. 80, 2010.
• 47. D. Woodhouse, “When Will Australians Get a Proper Perspective and a Proper Standard?” Australian Financial Rev. Higher Education Conference - From Revolution to Higher Achievement - Driving Reform through Innovative Policy and Finance, http://www.auqa.edu.au/files/presentations/when_will_australians_get_a_proper_ perspective_and_a_proper_standard.pdf, 2010.
• 48. C. Kiu, and C. Lee, “Ontology Mapping and Merging through OntoDNA for Learning Object Reusability,” J. Educational Technology and Soc., vol. 9, no. 3, p. 27, 2006.
• 49. T. Pedersen, S. Patwardhan, and J. Michelizzi, “WordNet::Similarity: Measuring the Relatedness of Concepts,” Proc. Demonstration Papers at HLT-NAACL, pp. 38-41, 2004.
• 50. D. Gasevic, A. Zouaq, C. Torniai, J. Jovanovic, and M. Hatala, “An Approach to Folksonomy-Based Ontology Maintenance for Learning Environments,” IEEE Trans. Learning Technologies, vol. 4, no. 4, pp. 301-314, Oct.-Dec. 2011.
• 51. S. Barrie, “Understanding What We Mean by the Generic Attributes of Graduates,” Higher Education, vol. 51, no. 2, pp. 215-241, 2006.
• 52. S. Barrie, C. Hughes, and C. Smith, “The National Graduate Attributes Project: Integration and Assessment of Graduate Attributes in Curriculum,” The Univ. of Sydney, 2009.