The evolution of Information and Communication Technologies, coupled with their increased use in education, has led to a wide range of learning activities and the growing importance of Technology Enhanced Learning (TEL). Thus, many new issues have emerged and need to be explored.
In our research, we are particularly interested in the design of interoperable TEL systems across different Learning Management Systems (LMS).
We define a TEL system as a complex system formed by a set of interdependent and heterogeneous components (i.e., actors, tools, and learning objects) organized in space and time in order to satisfy a learning goal [ 36
An LMS is a software system that supports distance teaching and learning. An LMS provides much relevant functionality for collaborative learning, assessment, and communication using extremely powerful tools such as forums, chats, wikis, blogs, quizzes, etc. It also offers several administrative services such as the management and tracking of courses, students and teachers. In addition, an LMS offers technical services, such as backup, statistical reports, and language management [ 27
It is necessary to guarantee the interoperability of TEL systems across different Learning Management Systems to save development effort, time, and cost. The need for interoperability has led to many learning technology standards, such as SCORM [ 1
] and IMS-LD [ 19
However, these standards have limits as concerns personalization and contextual expressiveness [ 17
], [ 41
]. Adopting a particular standard also restricts the designer to choosing a compliant LMS.
In parallel, Model Driven Architecture (MDA) has emerged as a software engineering framework for dealing with the problem of system interoperability across different execution platforms. MDA does this by separating business and technical concerns and proposing techniques to integrate them. In addition, MDA code generation mechanisms allow generating code from models.
In this paper, we are interested in MDA as an instructional design process that ensures LMS interoperability while allowing modeling of TEL systems with specific languages.
Following MDA practices, a TEL system is designed in a three-step process. First, a model of the intended system with a specific metamodel is defined. This metamodel allows an accurate description of specific needs. Second, a model transformation engine with specific rules is used to transform the preceding model into an LMS-specific model. Third, the specific model can be deployed on the LMS using an automatic generator/deployer.
This type of design process involves a number of technical tasks that beginning instructional designers cannot do easily and efficiently.
In the early 1990s, several authors investigated the automation of some instructional design process sections for beginner instructional designers [ 23
], [ 26
], [ 34
]. Automated instructional design (AID) tools assist instructional designers in creating instructional products to improve learning [ 4
]. The strength of AID tools lies in their ability to guide novices and non-ID professionals through the process of creating effective instruction [ 6
], [ 34
In this paper, we focus on the second step of the MDA process related to the refinement of abstract learning scenarios (transformation in the terms of MDA) in order to define LMS-specific scenarios. We aim to study whether it is possible to help beginning designers perform technical tasks related to the transformation step.
Our proposal, which we call ACoMoD (Assistance for Contextualized Modeling of learning systems), addresses this issue, not only from a technical point of view, but also from an institutional point of view [ 35
] by helping designers to make proper decisions with regard to contextual best practices (CBP). A CBP provides best LMS tooling practices to designers in a specific context. It is very helpful to inform a designer of these practices for making the proper choices with respect to the institutional context [ 35
]. Thus, ACoMoD allows a contextualized refinement of learning scenarios.
This paper is organized as follows: In Section 2, we explain the benefits of MDA-based instructional design, especially when compared to design based on Technology Learning Standards. Section 3 discusses scientific work related to tools and contextualizing learning scenarios and describes how assistance may be provided to designers. In Section 4, we discuss our approach and how we have implemented it using Gen-COM. Section 5 presents a detailed case study showing how a designer can use Gen-COM, along with the different artifacts generated and used. We explain our evaluation methodology and discuss evaluation results in Section 6. We end with a conclusion and an overview of future possibilities.
2. Instructional Design Interoperability
Interoperability is a central concern in instructional design because Learning Management Systems are many and varied. Interoperability means being able to execute the same pedagogy on different systems. To ensure interoperability, many learning technology standards (LTS) have been proposed and are described in the literature.
2.1 Learning Technology Standards
Organizations with shared interests and approaches have developed specifications for TEL such as IMS [ 18
], AICC [ 3
], and ADL [ 2
]. Among the most prominent standards are those for IMS Learning Design (IMS-LD) [ 19
] and ADL SCORM (sharable content object reference model) [ 1
The advantages of using standards in learning design have already been pointed out: “Standards are generally developed for use in systems design and implementation for the purposes of ensuring interoperability, portability, and reusability.” However, the use of standards presents certain disadvantages [ 17
], [ 43
], [ 48
1. Complex semantics. To use an LTS, an instructional designer must learn its syntax and semantics and must adapt his or her modeling practices accordingly. For example, IMS Learning Design (IMS-LD) provides a language based on a theatrical metaphor, within which it is possible to describe the structure of activities and tasks, the assignment of roles and the workflow of a unit of learning as a “learning design” [ 19]. Each learning scenario modeled with IMS-LD must be defined according to the theatrical metaphor (play, act, role-part, etc.).
2. Generic concern. In order to cover a wide range of needs, standards such as IMS-LD and SCORM are generic. In consequence, they are shallower and allow only a superficial modeling of needs. They are characterized by a lack of pedagogic and contextual expressiveness:
- Pedagogic expressiveness is fundamental for instructional designers because it expresses how adequately the domain represents and how comfortably instructional designers can express their ideas.
- Contextual expressiveness is also fundamental because it allows outputting the adequate TEL system models with regard to context.
3. LTS-compliance required. A major inconvenience of standards is related to the fact that designers cannot choose freely since they must use an LMS compliant with the standard used for modeling. For example, SCORM content can be interpreted only by a SCORM-compliant LMS; i.e., an LMS which implements the SCORM Run Time Environment.
To address these drawbacks, LTS adaptation has inspired many papers, including one based on our previous work [ 38
]. However, standard adaptations are not interoperable because they are not supported by standard specifications.
Considerable attention has been focused recently on MDA (Model Driven Architecture) as an alternative solution to LTS that warrants personalization while ensuring interoperability based on software engineering standards.
2.2 Software Engineering Standards2.2.1 MDA Basis According to the Object Management Group (OMG) [ 28], Model Driven Engineering (MDE) [ 21], [ 22] is a specific approach to software engineering that defines a theoretical framework for generating a code from models using successive model transformations.
The main goal of this approach is to separate the business side of a system from its implementation. The business model of a system can therefore drive its implementations on different platforms. In this way, we can expect to obtain better coherence between implementation and interoperability.
The best-known MDE initiative is the Model Driven Architecture proposed by the OMG [ 22].
MDA states that it models the environment and the requirements for a system in a Computational Independent Model (CIM).
A CIM does not show the details of system structure. Thus, a CIM can be used to build a Platform Independent Model (PIM). A PIM focuses on the operation of the system while hiding details related to the use of a particular platform. A PIM maintains platform independence in order to be suitable for use with different platforms. The transformation of a PIM into a Platform Specific Model (PSM) is based on the associated Platform Model (PM).
A PSM is a system model for a specific platform. It combines PIM specifications with the details that specify how that system uses a particular platform.
Fig. 1 shows the main concepts and techniques used in MDA.
Fig. 1. Main concepts of the MDA approach.
2.2.2 MDA as a TEL Design Process Following MDA practices, a typical design scenario based on MDA is presented in Fig. 2 and can be described as follows [ 33].
Fig. 2. MDA approach to TEL system design.
First, the instructional designer informally defines the learning scenario to be created and the resources needed (CIM).
To formalize intent in a pedagogic model, a modeling language is used. This language allows defining the pedagogic method and contextual constraints. In MDA terms, the pedagogic model is a PIM.
Next, to obtain a system that can be executed by a LMS, the pedagogic model is transformed into a technical model (PSM). This corresponds to the LMS metamodel. The common way to transform one model into another (and the one we used) is to use a model transformation engine (like ATL [ 20]) and a set of model transformation rules dedicated to this type of mapping. Transformation rules express refinements from the pedagogic metamodel to the LMS metamodel.
We mention that the difference between a pedagogic concept (related to PIM) and a technical one (related to PSM) consists in the fact that a pedagogic concept is abstract and related to the pedagogic intents of the instructional designer, but a technical concept is concrete and represents an alternative tool allowing the execution of the corresponding pedagogic concept within an LMS. Therefore, pedagogic concepts remain abstract until they are contextualized by LMS concepts at the transformation step.
It is important to note that the semantic signification of each element in the pedagogic metamodel and in the LMS metamodel is as follows:
2.2.3 MDA Contribution to TEL Design In the literature, a number of recent papers show an interest in TEL system design based on MDA [ 9], [ 10], [ 13], [ 32], [ 33], [ 37].
- Each concept (of the LMS or pedagogic model) represents an activity, a resource, or an actor.
- Each attribute of a concept represents a property of that concept.
- Each association expresses a relationship between two concepts: two activities (e.g., complementary use), two actors (e.g., collaboration rules), two resources (e.g., simultaneous use), a resource and an activity (e.g., associating a resource to an activity), a resource and an actor (e.g., affecting an actor to a resource) and, finally, an activity and an actor (e.g., affecting an actor to an activity).
As explained in Section 2.1, MDA as an instructional design process addresses the limits of LTS by empowering instructional designers to:
1. Create specific modeling languages that ensure both pedagogic and contextual expressiveness. These languages are specific and are thus more targeted and focused. They allow accurate descriptions with a semantic precision not achievable with generic models.
2. Create specific design tools based on standards and generators proposed in the context of MDA such as Eclipse EMF [ 15] and GMF [ 16].
3. Reuse learning scenario models through standards such as MOF (Meta Object Facility [ 28]).
4. Address any LMS, which is an important benefit offered by MDA since the design based on an LTS requires adoption of a compliant LMS.
5. Automatically deploy TEL systems on the chosen LMS. Therefore, designers are spared the technical difficulties related to deployment.
Nevertheless, the main limit of MDA as an instructional design process is the high level of technical expertise it requires.
Our current research focuses on proposing automated instructional design tools to assist instructional designers in performing different tasks related to the MDA process. In this paper, we focus on the task of transforming pedagogy with regard to LMS tooling. In the next section, we present and analyze work related to this issue.
Two main solutions emerge from a review of the literature.
The first approach [ 13
], [ 14
] is typically based on MDA and proposes to transform the pedagogic scenario model into a technical model by applying transformation rules. These rules express mapping from PIM to PSM and must defined beforehand by a computer scientist using a transformation language. There are several transformation languages, such as ATL (ATLAS Transformation Language) [ 20
] and QVT (QueryView Transformation) [ 29
]. The advantage of using these languages rests on the possibility of automatically generating LMS-specific models from platform-independent models when transformation rules are defined between the two corresponding metamodels. Nevertheless, only experts can code transformation rules and apply them because it is necessary to be skilled in transformation languages.
The Warin et al. [ 7
] and Caron et al. [ 32
] show that this approach gives rise to serious limits for designers who find themselves confused because of the gap between their pedagogic models and the automatically generated technical models.
In order to address this limitation, a second approach proposes discarding the intermediate pedagogic model and directly expressing pedagogic intents in conformity with a modeling language which merges the pedagogic metamodel and the LMS metamodel [ 7
], [ 32
], [ 49
Experimentation with this approach proved to be more helpful to designers because there is no double modeling (pedagogic and technical). However, this solution does not ensure interoperability because both business and technical concerns are merged in the modeling language. Thus, every model produced is specific to a single LMS. Experimentation also highlights the problem of perceived affordance. For an LMS, the semantics of concepts and functionalities changes according to users. This assertion decreases the interest of having TLS compliance for each LMS. Mapping from the TLS to the LMS concepts would follow only one “affordance.”
In summary, these approaches provide a solution to the problem of expressing pedagogic intents according to the LMS metamodel, but they nonetheless present the following gaps.
First, refining pedagogic intent according to LMS capabilities requires the help of a computer scientist because instructional designers are usually novices in this task. Thus, there is a need to study whether it is possible to assist designers in carrying out transformations by themselves.
Second, all proposed approaches consider the issue of refining pedagogy from a technical point of view. As our study of contextualization of learning systems [ 35
] shows, the institutional point of view must be considered for a better contextualization of TEL systems. We propose studying the usefulness of helping designers take into account the institutional point of view, which we define as a set of contextual best practices in using LMS tools. We have named the refinement step “contextualized refinement” or “contextualized transformation.”
A contextual best practice is a successful practice with an impact that can be replicated in a specific context. A CBP offers ideas about what works best and what to avoid in a given situation; e.g., recommending or prohibiting the use of an LMS tool. We note that our concept of best practices differs from that of patterns [ 45
]. Patterns are structured sets of best practices with strong intentions that are generally pedagogic [ 46
], [ 47
]. In our case, we are interested in the directives given by the institution concerning the use of LMS tools. These directives do not have a very strong formal structure.
In the remainder of this paper, we will focus on presenting and evaluating our proposal and its contribution with respect to related work.
ACoMoD offers an approach for the “contextualized transformation” of pedagogy. Contextualized transformation consists of choosing the LMS tool(s) to be used to create each pedagogic element with respect to contextual best practices.
We propose Gen-COM, an assistant for contextualized transformation (see Fig. 3
). Gen-COM uses best practices that are modeled with our Gen-IC tool (see Fig. 3
). Gen-COM is intended for instructional designers, whereas Gen-IC is intended for institutional experts.
Fig. 3. Overview of the ACoMoD approach.
The process of transformation implemented by Gen-COM is based on parametered ULM models, called UML templates, that will be described in the next section.
4.2. UML Template Fundamentals
According to UML specifications [ 30
], a template is a parameterized element (class template, package template, etc.) that can be used to generate other model elements called “bound elements” using template binding relationships. To specify parameterization, the template owns a signature. A template signature corresponds to a list of formal parameters that will be substituted by actual parameters in a binding.
A template binding is a directed relationship labeled by the
stereotype from the “bound element” to the template and specifies a set of template parameter substitutions that associate actual elements to formal parameters. The semantics of a bind relationship are equivalent to the model elements that would result from copying the contents of the template into the bound element, replacing any element exposed as a template parameter with the corresponding element(s) specified as actual parameters in this binding. Several controls are indicated by the UML in order to guarantee that each bound element respects its template. For example, the parameters and their corresponding actual values must have the same type (attribute, class, etc).
A template has a specific graphical notation which consists in superimposing a small dashed rectangle containing the template signature on the upper right-hand corner of the standard symbol of the parameterized element (see the template tagged with the letter “a” in Fig. 5
4.3. Proposed Transformation Process
Our approach is based on the UML bind relationship. To our knowledge, this is the first time that UML templates have been used for the transformation from platform-independent to platform-dependent models.
The first step (see the arrow labeled “Transform to” in Fig. 4
) consists in transforming the pedagogic model into a package template. The transformation to platform concerns each element in this model, so each element (class, attribute, type, or association) is declared as a parameter in the signature of the package template. The process of transformation to platform (see the arrow labeled “bind” in Fig. 5
) is carried out by making decisions to substitute these parameters with elements of the platform metamodel (we verified that UML allows the substitution of template parameters by actual values taken from another model element). Each parameter substitution created in the “bind” relationship corresponds to a transformation rule; formal parameters are pedagogic concepts and actual parameters are platform concepts.
Fig. 4. Transformation to a platform based on bind relation.
Fig. 5. Example of transformation based on UML templates.
shows an example of transformation based on our approach. It concerns the transformation of a simple supervision strategy of a Java project by asynchronous meeting (see the template tagged with the letter “a” in Fig. 6
, where all the elements are declared as parameters in the template signature.) to Moodle LMS [ 25
]. The part of the Moodle metamodel that we use for transformation is presented in the model tagged with letter “c.”
Fig. 6. A screenshot of Gen-COM. Gen-COM allows matching a pedagogic model to an LMS metamodel and is guided by contextual best practices.
4.4. Gen-COM: Assisting Contextualized Transformation of Pedagogic Models According to LMS
To implement our proposal, using UML tools (e.g., Papyrus [ 31
]) is possible but requires significant experience with the UML metamodel.
We developed the Gen-COM tool in order to assist in carrying out the transformation process. There are two fundamentals behind the design of Gen-COM.
The first is to hide all technical difficulties related to the binding of UML templates (see the transformation component assistant in Fig. 3
). To do this, our technical solution relies on implementing Gen-COM as an Eclipse framework plug-in [ 1
]. This plug-in can use other plug-ins, namely UML [ 30
] and EMF [ 15
], and manipulates UML diagrams through programs.
The second is to provide designers with guidance and help for making the right decisions with regard to contextual best practices (see the contextualization component assistant in Fig. 3
A simplified example of how to use Gen-COM is described below:
First, when the instructional designer authenticates, Gen-COM identifies the context, which is composed of the LMS and the relevant best practices. When the designer selects his pedagogic model, the model will be transformed into a template by the transformer component of Gen-COM (see Fig. 3
Next, Gen-COM generates the interface of contextualized transformation (see Fig. 6
). The pedagogic toolkit (see Fig. 6
) is built on the concepts of the pedagogic model. The technical toolkit is built on the concepts of the LMS metamodel. Thus, it is possible to address any LMS because it has a metamodel that can be established once and used when needed.
To tailor the pedagogic model, the instructional designer chooses the LMS tool(s) to be used to concretize each pedagogic element. Within Gen-COM, this task can be done by simply dragging and dropping from the toolkits to the central match area (see Fig. 6
). When the matching of two elements is validated, the designer then matches the corresponding attributes. Gen-COM offers assistance in binding associations according to class binding and, for each association in the pedagogic model, tries to find its correspondence in the LMS metamodel according to the matching of concepts. In case of conflict (there is no association in the LMS metamodel for an association in the pedagogic model), Gen-COM guides the instructional designer to change certain choices through the incoherency resolving component (see Fig. 3
In parallel, the manager of the transformation component presented in Fig. 3
stores the instructional designer's decisions as “parameter substitutions” in the bind relationship. These substitutions will be used later to enerate the final LMS-specific model.
The principal interface of Gen-COM also contains two important components, “choice summary” and “contextualization sheet” (see Fig. 6
). The “choice summary” component allows the instructional designer to continuously consult validated choices. The “contextualization sheet” provides assistance to the instructional designer according to best practices for using LMS tools. This sheet is managed by rules manager assistance, as shown in Fig. 3
. The contextualization sheet is updated according to the configuration of best practices. Currently, two configurations are supported. These are “displayed constantly” and “displayed when activated.” If the configuration of a contextual rule is “displayed constantly,” the corresponding assistance message is added to the contextualization sheet from the beginning to the end of the process. However, if the configuration is “displayed when activated,” the corresponding assistance message is added to the contextualization sheet when the instructional designer selects the tab corresponding to the LMS tool. A mouse event is associated with the technical toolkit in order to detect chosen tools and verify if there is a best practice to activate in the best practices database.
When the designer finishes working, the generator of specific models implemented in Gen-COM (see Fig. 6
) automatically generates the LMS-specific model based on the pedagogic model chosen by the instructional designer and the list of validated associations. They replace each class, attribute, and association with the matching choice in a copy of the pedagogic model. The generated model conforms to the LMS metamodel so that it can be deployed.
4.5 Deployment of Contextualized Models
GenDep [ 32
], [ 44
], [ 49
], a tool developed and tested by our research group, is used for deployment.
GenDep is based on generating specific engines given a contextualized technical model and a LMS metamodel. GenDep interprets a technical model specific to a given LMS and communicates with this LMS through SOAP in order to automatically deploy the model elements. If GenDep is missing any information, it asks the user (for example, for the path to a list of students to create on the LMS). GenDep can also show the user the actual elements present in the LMS and allows the user to establish a correspondence between these elements and model elements.
A number of experiments were carried out on different learning management systems, such as Wikinimst, Postnuke, and Moodle [ 7
], [ 32
4.6. Best Practices Modeler Basics
In this section, we present the fundamentals of our proposal to model the best practices necessary for the contextualized refinement of pedagogy. Further details and results are the subject of another paper.
For each best practice, Gen-COM needs the following information:
1. Assistance message: a message explaining a best practice and where it is pertinent.
2. Configuration: timed display of an assistance message when a best practice is activated.
3. LMS element(s) concerned by a best practice.
In ACoMoD, the Gen-IC tool ensures visual modeling of contextual best practices and stores them in a database (see Fig. 3
In order to facilitate modeling, Gen-IC allows the modeling of generic and specific best practices (see components of Gen-IC in Fig. 3
). Using a generic best practice (GBP), it is possible to model as many specific best practices (SBP) as needed. Modeling a specific best practice is done by indicating a GBP, an LMS element or elements concerned by the SBP, values required by the GBP, configuration, and the assistance messages.
Examples of GBP and SBP are presented in Table 1
Table 1. Examples of Best Practices
5. A Gen-COM Use Case Study
To illustrate feasibility and show the different artifacts related to a real scenario, we selected the case where a learning scenario named Mepulco [ 24
] was transformed into a Moodle specific [ 25
Mepulco is an active learning [ 39
] scenario for supervising student projects. Mepulco was created at the Institute of Technology of the University of the Littoral Opale Coast [ 42
] in Calais (France).
Mepulco has two goals. The first is to help groups of students succeed in creating a common product with respect to a deadline and to develop different individual skills such as analysis, synthesis, and argumentation. The second goal is to help tutors supervise the different steps of a project and provide a justified evaluation of the final product.
Mepulco relies on regular meetings to track the project's progression and ensure the deadline is met.
The principles of Mepulco for both tutors and students are published in two documents: a student kit and a tutor kit. These kits first describe the project fulfillment and supervision steps. Second, they describe the organization of regular meetings and the topics they deal with. Third, they define the allocation of roles within a group. Fourth, they list guidelines for writing the final report and preparing the final presentation. In addition, the kits provide a set of template documents, such as those used for making a project request, specifying requirements or writing a progress report.
Each project carried out according to Mepulco is based on distant activities such as a website, blog or an LMS (for example, Moodle). Designing this type of TEL system with respect to Mepulco was our goal.
For this case study, we first present the Mepulco model to transform and the Moodle metamodel. Next, we present the modeling of some best practices used at Calais. Finally, we describe the Moodle-specific model of Mepulco which was obtained.
The Mepulco model presented in Fig. 7
describes that supervision (“Supervision” class) by a “team” of “students” is ensured (“TeamSupervision” class).
Fig. 7. Mepulco model.
A resource corresponding to the Mepulco kit (“KitMepulco” class) and another one corresponding to the project subject (“KitProject” class) are attached to “Supervision” and are common to all teams. Supervision is based on certain activities of “Meetings” and the document repository (“Document” class). We helped designers define this model. To help designers model the pedagogy task, we proposed a tool [ 37
] that allows the reuse of models and their adaptation. We do not give detailed information on this tool because it is beyond the scope of this paper (see [ 37
Several papers [ 5
], [ 8
] have focused on explaining how to create a platform metamodel: i.e., a model of the mechanisms and services offered by the platform. We used the recommendations of [ 5
] and [ 8
] as the basis for elaborating a Moodle metamodel. Since this metamodel is enormous, we present only the part pertaining to our case study in Fig. 8
Fig. 8. Moodle metamodel.
Best practices. Three best practices were used by Gen-COM to help the designer with the transformation: 1) maximum number of forums to use is two, 2) chat tool is not recommended, and 3) wiki tool is recommended for project supervision. These rules were modeled with Gen-IC and stored in the best practices database.
Mepulco specific model.
The designer used Gen-COM following the scenario described in Section 4.4 to refine the Mepulco model according to Moodle. Selection details are explained in Table 2
and the final Moodle-specific model generated is presented in Fig. 9
. This is an example of how the designer interacted with Gen-COM. According to a recommendation proposed by Gen-COM, the designer changed an initial choice of using a forum tool to carry out “supervision” and decided to use wikis instead. He explained that he hesitated between wiki and forum. The assistance message recommending wikis for supervision convinced him to prefer wiki, especially because group members would be able to contribute collectively to writing.
Table 2. List of Designer Choices
Fig. 9. Final Moodle specific model of Mepulco.
In this section, we describe tests to check the impact and usefulness of our proposal to help designers bridge the gap between pedagogy and technology while taking contextual best practices into account.
6.1 Evaluation Methodology
Several case studies were carried out with 44 designers in the context of both the MetaWep project [ 7
] and an international online Master's program (eServices) [ 12
The case studies offered feedback (see Table 3
) on the usefulness of proposed assistance to perform the transition from pedagogy to LMS; the usefulness of good practices recommendations for designers; the quality of the recommendations offered; the usability of GenCOM; the opinions of designers concerning these new ways to design and whether they would use it again.
Table 3. Evaluation Criteria Codes and Labels
In each case study, we started by asking participants questions about their level of expertise in UML modeling, UML templates, and model transformation (see Table 4
). This information was necessary in order to evaluate the usefulness and relevance of transformation assistance.
Table 4. Information on Participant Skills
At the end of each experiment, the participants were called on to answer a questionnaire containing 13 questions and were given the opportunity to make comments and suggestions and mention difficulties encountered.
In order to facilitate the interpretation of data collected from questionnaire answers, we classified the questions according to evaluation criteria, as shown in Table 5
, which also gives possible answers for each question.
Table 5. Evaluation Questionnaire
6.2 Results and Discussion
Designers gave us their feedback and opinions on using GenCOM by answering the questionnaire. The questions and the designers' answers are summarized below.
The first question was “ Has GenCOM helped you to tailor pedagogy with LMS tools?”; Designers were able to choose between three options: Yes, Maybe, and No.
In general, designers consider that GenCOM helps them to tailor pedagogy (82 percent). They explain this choice by saying that they usually have difficulty doing their expression of pedagogic intents in terms of LMS by themselves. This difficulty arises from the fact that they need to study LMS and the possible supported scenarios in order to execute their LMS-independent learning scenario before dealing with this task. With Gen-COM, there is easy comprehension and use of the LMS. Gen-COM presents the LMS as a toolkit with a description of each element. In addition, when the scenario built by a designer, it is incoherent with respect to the LMS specifications. Gen-COM automatically notifies the designer and proposes alternatives to resolve incoherencies. Therefore, tailoring and modifying are easier to do. Only 18 percent of the designers answered No
, as shown in Fig. 10
Fig. 10. Feedback about whether Gen-COM helps transformation.
The second question was “ Do you think that transformation is facilitated by GenCOM?”; Designers were able to choose between three options: Yes, Maybe, and No.
Thirty-two percent of designers decided that transformation is facilitated by GenCOM, which represents 100 percent of designers who had model transformation skills. Designers who selected Maybe (68 percent) had no model transformation skills. Thus, they were not able to compare the transformation solution proposed with existing transformation solutions.
Those who found that GenCOM facilitates transformation emphasized that this is the first time they have done transformation graphically without the need to write a program. In addition, there was no need to be able to use complex tools for executing the transformation rules. They also mentioned that the GenCOM technique, which is based on UML templates, can be reused in application fields other than instructional design. This gives us an idea of how our approach for transformation is accepted.
Answering the question “ What do you think about this new way of design based on model transformation?”; allowed designers to choose among five options: very interesting, interesting, makes no difference, not very interesting, and uninteresting.
Sixty-seven percent of the designers selected very interesting
, and 14 percent selected interesting
(see Fig. 11
). They felt that GenCOM allows them to approach the transformation from pedagogy to LMS tools in a new and interactive way. Some designers said that the strength of this way of designing, which is based on modeling and transformation, is related to: 1) having a model representing pedagogy that can be reused, 2) conserving traces of the passage from pedagogy to technology that are reusable and adaptable to new situations, 3) automatic deployment in the LMS replaces manual deployment for which significant expertise is required.
Fig. 11. Feedback of opinions about design based on a model-driven approach.
A few designers (14 percent) found this way of design not very interesting and 5 percent thought it made no difference. They argued that design is easier when it can be performed without an intermediate pedagogic modeling step. They prefer modeling directly, based on LMS. This means it would be better for them if GenCOM helped them to model directly with the technical toolkit to create a model meeting their needs. In this way, they could benefit from automatic deployment.
The next two questions concerned the usefulness of two functionalities of Gen COM: “ Do you find the automation of some transformation tasks like generation of the final model useful?”; and “Do you find that help in resolving association binding incoherencies is useful?”; For both questions, designers were able to choose among five options: very useful, useful, makes no difference, not very useful, and useless.
No designer chose the useless
or not very useful
option for either question. Ninety-one percent consider the automation of tasks to be very useful, and 9 percent consider it useful (see Fig. 12
). Designers explain that they request automation of tasks when it does not make decisions in their place. In GenCOM, task automation is based on decisions already made by the designer.
Fig. 12. Feedback on the usefulness of automating tasks.
Regarding assistance for resolving incoherencies between choices made and the LMS capabilities, 77 percent consider it very useful and 23 percent considered it useful.
Two questions were asked concerning the appropriateness of assistance messages, their timing, and the duration of display: “ Do you find assistance messages appropriate?”; and “ Do you find the timing and duration of the recommendation display to be appropriate?”; For both questions, designers were able to choose between three options: Yes, No, and Maybe.
Designers feel that recommendations are clearly presented through concise and precise messages. In addition, detailed explanations are offered for each recommendation. In this case, the percentage of Yes
answers is 73 percent (see Fig. 13
) while the No
options were selected by 27 percent. Some designers say that these messages can be ambiguous. Other designers commented that in some cases messages are contradictory; for example, whereas one message does not recommend the use of forums, another recommends no more than two forums. We explain this issue by the fact that Gen-COM presents many practices which are useful in different situations, so there is no need for them to be coherent.
Fig. 13. Feedback on appropriateness of assistance messages.
Regarding the time and duration of the display of recommendations, only two designers selected the No
option, (see Fig. 14
Fig. 14. Feedback on appropriateness of timing and display duration of assistance messages.
Another question was: “ How often did recommendations influence your decisions?”; The possible responses were: always, sometimes, and never.
Sixty-three percent of designers always followed recommendations, 23 percent sometimes followed recommendations, and 14 percent percent never followed recommendations (see Fig. 15
). When we analyzed the designers' comments and explanations, we noted that their responses depended directly on their degree of experience in the context where GenCOM was used. The majority of designers who selected “always” or “sometimes” were recently recruited; therefore, they expressed a need for help in taking into account previous experience. The designers who selected the “never” option are experts in the context, where they have worked for many years, and thus do not need the proposed recommendations. However, they appreciated the fact that GenCOM did not oblige them to follow the proposed recommendations and that the Gen-COM interface clearly separates the main work area and the recommendation area.
Fig. 15. Feedback on how the recommendations influenced designers' choices.
The next question was: “ Did you like that GenCOM recommends best practices or would you have preferred no recommendations?”; The possible choices were: better with, makes no difference, or better without.
The answers to this question are similar to those of the previous question. Most designers said that they prefer recommendations (77 percent). These are the inexperienced designers. Fourteen percent preferred no recommendations, since they are all experienced designers (see Fig. 16
). Only four designers chose makes no difference
Fig. 16. Feedback on the pertinence of recommendations.
Regarding ease of use, 73 percent of designers selected the “yes” option and felt that GenCOM is easy to use, while 27 percent selected the “no” option. They had some difficulty in following the chain interfaces and proposed adding a summary of the different transformation process steps on the starting interface. This suggestion is being studied and implemented. Another remark concerned the relevance of allowing users to hide certain areas of the interface.
Finally, 86 percent plan to continue using GenCOM, while 5 percent are not sure if they will continue to use it. The remaining 14 percent will not continue to use it.
This paper focuses on two main points. On the one hand, we showed that it is possible to design interoperable TEL systems while using an instructional design process that can be adapted to specific needs. On the other, we studied how to support instructional designers in bridging the gap between their pedagogies and LMS specifications. Our proposal enables designers to easily match the particularities of a given pedagogic model with the available LMS tools while taking into account the best practices of the institutional or educational context.
We showed that UML template binding is a powerful mechanism for performing the transformation. Compared to related work in the literature, our approach to using transformation languages has the advantage of not requiring expertise in tranformation or programming language to code and apply the transformation rules.
Providing tools that facilitate the designer's task and include LMS best practices is crucial. In our work, we showed how the GenCOM tool we have developed allows a contextualized transformation to be done graphically.
As described above, feedback from designers using GenCOM shows their confidence in and acceptance of the usefulness of our proposal. They also suggested improvements, which are currently being studied.
Feedback concerned the usefulness of help displays in transitioning from pedagogy to LMS; the usefulness of good practice recommendations for designers; recommendation quality; GenCOM usability; how this new way of working impacts designers; and whether designers will continue to use the tool.
The main results can be summarized as follows: first, designers found that GenCOM was useful in tailoring pedagogy with LMS tools. Designers who are skilled in model transformation emphasized that GenCOM offers a powerful transformation mechanism. The originality of our proposal stems from a new way (graphical and interactive) to approach model transformation. Second, although the integration of best practices in the design process is useful for novices, it is less so for designers who are very familiar with their institutional context. Designers who have been recently recruited express more interest in the tool because they can use it to profit from accumulated experience. Third, Gen-COM clearly separates the work space for matching pedagogy and technology from the best practice reminders. In addition, these reminders do not disturb experienced designers when they carry out the matching operation.
Finally, most designers state that they are more likely to use a model-driven approach with tools like Gen-COM, which hide technical difficulties while allowing them to benefit from many advantages. These advantages include interoperability, reuse, and personalization (the process adapts to the designers and not the other way around, as is the case with LTS).
Gen-COM is therefore adapted to beginner and expert instructional designers. The Gen-COM module recommending best practices is useful to novice designers but can be ignored by experienced designers.
In light of related work, we can summarize our contribution as follows: first, we explain the interest of using MDA as an instructional design process. We demonstrate that it is a very encouraging approach. Its success depends on proposing assistance tools that mask technical difficulties, since this is the main barrier to acceptance. Second, we propose a novel approach for helping designers to bridge the gap between their pedagogy and LMS. This approach is based on graphic, interactive model transformations. Our approach lets designers incrementally construct LMS-specific models, thus enhancing their LMS proficiency. Third, integrating LMS tool best practices in the design process has not been attempted in related work. These best practices help designers use LMS more effectively.
In the future, we plan to automate transformation based on the designer's profile. This profile would include technical skills, personal preferences as concerns tools, a history of previous designer actions, and models and their context. The idea is to offer them possible models according to their profile, a given learning scenario and a given LMS.
We are also working on extending the assistant to contextualization to propose recommendations to designers based on their personal characteristics.
Another interesting issue we are looking at concerns tracing TEL systems at runtime to extract conclusions on the appropriateness of designer choices and to recommend adjustments. At runtime, design continues as the TEL system adjusts to a changing context (for example, when a resource is not available, it must be replaced).
This work was supported in part by the Franco-Tunisian SAME2 project (e-Services, MultiScale approaches, and TEL). The authors wish to thank all project participants from RIADI, NOCE, and CRI.
R. Drira and M. Laroussi are with the RIADI Laboratory, ENSI Manouba Campus 2010, Tunisia.
E-mail: email@example.com, firstname.lastname@example.org.
X. Le Pallec is with the LIFL Laboratory, USTL, F-59650 Villeneuve d'Ascq, France. E-mail: email@example.com.
B. Warin is with ULCO, LISIC, Calais F-62100, France.
Manuscript received 16 Mar. 2011; revised 27 Oct. 2011; accepted 18 Nov. 2011; published online 6 Feb. 2012.
For information on obtaining reprints of this article, please send e-mail to: firstname.lastname@example.org, and reference IEEECS Log Number TLT-2011-03-0030.
Digital Object Identifier no. 10.1109/TLT.2011.35.
received the PhD degree in computer science in 2010 from the University of Manouba (Tunisia) and the University of Sciences and Technologies of Lille I (France). She is currently associated with the Riadi Laboratory ( http://www.riadi.rnu.tn) and the NOCE Team of the LIFL Laboratory ( http://www.lifl.fr) and is a teaching assistant in the University of Tunis. Since 2005, she has been interested in TEL system design, in particular, based on technology learning standards and a model-driven approach. She has coauthored around 10 publications.
received the PhD degree in computer science from the University of Tunis in Tunisia in 2001. She is currently an assistant professor in the Department of Computer Science at the National Institute of Applied Sciences and Technology, Tunisia, and an associated researcher in the NOCE group of the LIFL Laboratory, Lille, France ( http://noce. univ-lille1.fr/cms). Her current research interests include mobile learning, user models, adaptability, and context-awareness. She is a coauthor of more than 40 publications related to her research areas.
Xavier Le Pallec
received the PhD degree in computer science in 2002. He is an associate professor in the Computer Science Research Lab of the University of Lille 1 (LIFL). His research interests include model-driven engineering, visual notations, and model components. After studies on e-learning as an application domain, he has worked on multimodal interactions, and in particular on intelligent environments, since 2007.
received the PhD degree in 1986 with a thesis in theoretical computer science on “rationality and recognizability on graphs.” From 1992 to 1998, while holding a position as lecturer at the Université du Littoral Côte d'Opale, he created a company to develop an expert system in insurance pricing. This system is the French market leader. Since 1999, his research interests have centered on the construction of technology-enhanced learning. In parallel to his research, he works in higher education, in particular on project-based learning and software engineering applied to learning management systems.