The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.02 - April-June (2010 vol.3)
pp: 96-109
Published by the IEEE Computer Society
Leslie Carr , University of Southampton, Southampton
Jessie M.N. Hey , University of Southampton, Southampton
Yvonne Howard , University of Southampton, Southampton
David Millard , University of Southampton, Southampton
Debra Morris , University of Southampton, Southampton
Su White , University of Southampton, Southampton
ABSTRACT
It seems self-evident that life for teachers would be simplified if there existed a large corpus of relevant resources that was available for them to reuse and for inquisitive students to download. The learning object community has worked for the past decade and more to provide the necessary infrastructure, standards, and specifications to facilitate such beneficial activity, but the take-up has been disappointingly small, particularly in University and Higher Education, which is the subject of this research. The problem has been that practitioners have not deposited their teaching resources, or have not made them openly available, in the quantity that would achieve critical mass for uptake. EdShare and the Language Box are two initiatives that have concentrated on the issue of facilitating and improving the practice of sharing, the former in an institutional setting and the latter in a subject community of practice. This paper describes and analyzes the motivations for these projects, the design decisions they took in implementing their repositories, the approaches they took to change agency and practice within their communities, and the changes, in practice, that have so far been observed. The contribution of this paper is an improved understanding of how to encourage educational communities to share.
Introduction
The idea of creating digital learning resources with the express purpose of reuse has been around since the early 1990s. Wiley looks back to the flourishing of open-source software as the roots of the open education movement [ 1]. Even before the Web, systems such as Microcosm were promoting the idea of creating learning paths through repositories made from appropriately sized chunks of learning resources [ 2], [ 3]. The strength of the movement can be gauged from initiatives such as OpenContent at Utah State University in 1998—now the Center for Open and Sustainable Learning (COSL). 1 It seemed self-evident at that time that as soon as teachers were able to search the Web for teaching resources a culture of publication and reuse of such materials would develop.
We have witnessed a number of initiatives that have attempted to provide the technology to support the practice of publication and reuse: metadata standards and specifications such as IEEE Learning Object Metadata (LOM) [ 4], [ 5] have been developed to enable us to describe the content and purpose of resources so that they can be accurately located; learning object repositories (LORs) have been developed in which to store the resources and their metadata; content packaging specifications such as IMS-CP have been developed in order to enable interoperability and the transfer of sets of interlinked files in a manner that will allow them to be unpacked on another server; and runtime engines such as SCORM RTE and IMS-LD Coppercore [ 6] have been developed that allow quite complex sequencing of users through sets of resources.
In addition to the development of the technical infrastructure, various funding bodies and charitable institutions have supported programs and projects with the objective of creating enough open content to bootstrap the practice of reuse. There are now many repositories of learning objects. In the UK, significant effort and funding has been directed to the JORUM 2 national repository established by funding from the Joint Information Systems Committee (JISC). 3 In recent years, nationally funded projects have been required to deposit any learning material outputs into this repository [ 7], [ 8]. Indeed, at the time of writing, the funding bodies are starting a major Open Educational Resources (OERs) program with the intent of further promoting the practice of exposing content that has appropriate licensing to allow reuse.
However, in spite of all these initiatives, it would be fair to say that the concept of reusable learning objects has hardly registered with the average "chalkface" academic in Higher Education. In recent consultations with academic staff, we have observed that few of our colleagues look for learning objects to reuse in their teaching. In contrast, we have significant evidence that teachers frequently use search engines to identify suitable resources on the Web, which they then cut and paste and mash-up in numerous ways to create new, tailor-made resources. Such observations tend to indicate that there is no lack of interest in reuse, but rather that the infrastructure that the community had built to support reuse may not be appropriate to the needs of the work-a-day academic.
We have noted some interesting behaviors among teachers sharing resources they have created. Often institutional policy or preferred practice is that these resources are locked into the institution Learning Management System (LMS), Virtual Learning Environment (VLE) so that just the teacher(s) and their current class can access them. Teaching resource Web sites are often hosted on a firewalled intranet remaining invisible to the wider world. A few public-spirited academics make their teaching materials open, by putting them on an unrestricted Web site, and these materials are often much downloaded from outside the institution. However, many teachers are hesitant or reluctant to make their materials open: Such teachers are concerned with losing rights and control of their materials, and thus, forgoing possible financial profit; they are concerned about quality judgments of their materials; and possible copyright claims against embedded content that they have downloaded and reused in their resources without the specific permission of the owner or publisher.
In this paper, we describe the work of two on-going initiatives, EdShare, an institutional educational repository for the University of Southampton, UK (See Fig. 1), and the Language Box [ 9], a repository for the UK national language teaching community, both of which set out to challenge existing approaches to learning objects by providing the sort of environment characterized by some of the Web 2.0 sharing sites, such as YouTube and Flickr, that encourage people to share and reuse their resources. Thus, we are able to compare technology adoption between an institutional and a discipline-based culture. In each case, we are interested in how the behaviors of early adopters can be migrated to the majority. In both instances, the project teams worked with academics to address their concerns and change practice as well as to develop further understanding of policy work that would both support and encourage further development.


Fig. 1. The EdShare institutional repository. In the Language Box community repository, all registered users can deposit.




2. Background
The important tasks in setting up Southampton's now well-established institutional research repository (EPrints Soton) also included working with academics in addressing concerns, articulating benefits, and working to change practice. Such feedback strongly influenced the model of repository setup at Southampton [ 10], [ 11]. EPrints Soton was an early institutional exemplar, supported by JISC's Focus on Access to Institutional Resources Programme (FAIR). It is useful, at this time, to look at the external environment influencing repositories for research and for teaching and learning.
The Open Access movement has a long history. A Physics community archive called arXiv 4 was a prominent pioneer. The Internet provided an opportunity for speeding up the established practice of circulating "preprints" to aid awareness and collaboration. The archive expanded its remit into related subjects and by 2008 contained more than half a million research articles. Stevan Harnad, a professor at the University of Southampton, has remained one of the staunchest advocates of Open Access, working tirelessly through mailing lists and global activities such as the 2002 Budapest Open Access Initiative to promote awareness and action. The EPrints 5 software was produced at Southampton to allow repositories to be implemented and the FAIR Programme enabled institutional research repositories to be set up and researched as complements to subject-based archives such as arXiv.
Institutional repositories were envisaged as a set of services supporting all the intellectual assets of the institution [ 13]. Southampton has now established a steering committee looking at a range of its significant assets such as research, data, and educational resources. The Southampton research repository model enabled researchers to deposit metadata on their research papers and add full text as they felt more able to share them—often influenced by the external initiatives. However, there is often less confidence in sharing other materials such as data [ 14] and now teaching and learning—the subject of our paper.
Teaching and learning has been supported by global initiatives more recently than research. On 22 January 2008, The Cape Town Open Education Declaration, called for free, adaptable learning materials [ 15]. Institutions such as MIT, with its OpenCourseWare [ 16], have been pioneers in showcasing their teaching in the form of courses. MIT started 2009 with 1,890 courses on view. The OpenCourseWare Consortium is now a collaboration of more than 200 higher education institutions and associated organizations from around the world creating a body of open educational content using a shared model.
The Open University, with its OpenLearn service, is an interesting case study in the UK [ 17]. In April 2008, OpenLearn reached its target to have 5,400 learning hours of content in the LearningSpace and 8,100 hours in the LabSpace, which allows students to reuse and remix materials. Communities are also represented by established resources such as SMETE 6 in the US. The SMETE Open Federation was formed to promote the teaching and learning of science, mathematics, engineering, and technology at all levels and is aimed at both students and teachers. This shares common aspects with our endeavors, since its variety of resources is not focused on whole courses.
The National Science Digital Library (NSDL) is a similar effort to provide organized access to materials at all levels of science, technology, engineering, and mathematics. NSDL also uses a number of Pathways to provide access to partners with more specific subject domains (for example, AMSER—Applied Math and Science, and BEN—the Biological Science Pathway).
Connexions too is a powerful example which has more of a grassroots organization, like our repositories, but in this case, it encourages contributions from all comers [ 18], [ 19]. Finally, OER Commons 7 based in California harvests material from many sources including those we have mentioned but it also makes efforts to find and support more modestly presented materials. It has a global reach and broad range of educational levels although more are at postsecondary level.
We have evidence of a variety of repository types: institutional, funder, regional, and subject community side-by-side in the research space. From the examples above, it is evident that we will now find a wide range of repositories in the education space. As in the research environment, global Open Access initiatives are having a significant effect [ 20], [ 21], [ 22], [ 23], [ 24]. However, there remains room for practical efforts to encourage individual teachers to collaborate as part of their natural workflow whether within an institution with its specific ethos or within a subject community of specific characteristics.
3. Context
It is clear that there are benefits to an institution or a subject community in having a repository of educational resources. As well as providing a source for teachers to reuse, leading to improved efficiency, the repository can be used for: curriculum planning, student-centered learning, administration, evidence for quality assurance, educational development, student information, public information, marketing, and outreach.
The University of Southampton is the home to EPrints, one the most internationally well-established packages for institutional repositories. The University has a well-developed culture of depositing research outputs in the institutional repository, which was reinforced by a recent national research audit; the use of data direct from the repository significantly reduced the effort in responding to this audit. Furthermore, publications that are stored in the repository gain high hit positions from search engines, especially Google Scholar, so they are much downloaded and consequently frequently cited.
A separate educational repository based on the EPrints engine would generate little additional maintenance load in the long term, but provide an identifiable focus for a slightly different data collection. The challenges for an educational repository are to identify a set of benefits for teachers that will motivate them to deposit materials and identify perceived barriers, and then, find ways of removing or overcoming such barriers.
In the next section, we examine the approaches we took, from both a technical point of view and a sociopolitical standpoint. An essential aspect of our approach was to convince teachers that what we were asking them to share was not some perfectly completed learning object, but rather the artifacts that make up their everyday teaching: the PowerPoint presentations, the reading lists, the work sheets, the assignments, the diagrams they have drawn, the data sets they have collected, and the photographs, slides, and videos they have shot.
The repositories we developed were implemented in EPrints, and we made the following basic design requirements in order to provide the framework to encourage colleagues to use them and realize fast benefits from their efforts:

    Users can simply and speedily deposit any kind of file, or collection of files, and describe them to a chosen level of detail using metadata or free-form text.

    Users can control the access levels to their files (typically, "institution-only" or "open access").

    Items are allocated a unique and permanent URL which can be referenced by other systems, such as the institutional VLE.

    The free-form text and metadata descriptions of all files lodged in the repository can be found and indexed by search engines.

    Search engines can only find the content of files if they are lodged for open access.

    Users can browse all items in the repository and download those lodged for open access without the need to be a registered user of the repository.

    Users browsing the repository can see Web 2.0-like tagging and annotation features.

4. Implementation
4.1 Technical
For a number of years, Learning Objects (LOs) have been promoted as an appropriate way for teachers and lecturers to package and share their materials. These LOs could be formally defined (for example, using IEEE LOM) or might be more informal (for example, a collection of Web pages hosted on an institutional machine). In both cases, the LO describes a set of materials that have been carefully assembled and described with the intention that they could be picked up and reused by others.
In contrast, many of the everyday materials that teachers and lecturers use are tied more tightly to a particular course or cohort, and they are not typically packaged, either in the sense of being wrapped in explanatory material, or of being described using a metadata schema. These materials are more akin to working documents, and although they are not as polished as LOs they represent the vast majority of teaching and learning materials.
Existing repositories are not designed for managing these everyday materials, and teachers and lecturers have shown reluctance in using them. The challenge for the EdShare and Language Box development teams was to understand why there was a mismatch and to reimagine teaching and learning repositories for everyday materials.

4.1.1 Issues with the Traditional LO Approach for Sharing Everyday Materials In previous projects, CLARE and CLARET, we have designed and developed more traditional repositories for Learning Objects [ 25], [ 26] and have explored, in a number of evaluation workshops, whether they could be used for everyday materials. Although the response to both systems was positive (in terms of aims and user interface), our workshops highlighted a number of difficulties that teachers and lecturers have with the LO approach:

    1. Complex metadata. Although it has been argued that LORs need careful construction of metadata [ 27] it was clear that a significant barrier to teachers using a repository was the complexity of the deposit process, and in particular, the need to specify a large number of metadata fields before a deposit could be made. CLARE used a variation of the UK LOM Core, a schema which includes 25 required fields, plus a further 27 recommended fields [ 4]. It was clear that while professional LO developers were prepared to take the time to understand and complete the schema, everyday sharers would not be.

    2. Unfamiliar terms. Another issue with the schema was the use of pedagogical terms that were sometimes unfamiliar to practitioners. For example, the schema would talk about scaffolding, but teachers would talk about supporting materials. An everyday repository needs to use simple, clear terms that relate to practice.

    3. Content packaging. In CLARE, LOs were downloaded as compressed zip files with an xml manifest that included the metadata and deployment information. Most teachers in our workshops had encountered compressed zip files before, but many did not really understand what they were, or how to open them. Those that did were confused by the internal structure of the LO and baffled by the xml manifest. Teachers expected the materials downloaded from a repository to come in a familiar format, matching the digital resources they created themselves.

    4. Lack of contextual information. Although the LOs contain a lot of metadata about how they are supposed to be used, they do not contain simple context information—such as comments as to the quality of the LO, or reports of how well the LO functioned with a particular group of students. One of the biggest realizations for us was that this sort of unstructured feedback from other users is far more important to teachers and lecturers, in terms of helping them decide if a resource will be useful, than the formal descriptions created by the LO author.

These observations are effectively a requirement for simplicity; particularly, removing the barriers to making deposits, so that it is as straightforward as possible to add materials into the repository. This may introduce problems in terms of ensuring adequate quality of content and sufficient descriptions of materials so that they can be found. However, value of attracting deposits is paramount, and extant solutions such as automatic metadata generation can address the major problems identified.


4.1.2 Analysis of Web 2.0 Sharing Sites Web 2.0 sharing sites such as YouTube, Flickr, Metacafe, and bookmarking services like Digg and Del.icio.us allow users to deposit content with the minimum of overhead. The sites are very popular, attracting hundreds of thousands, even millions, of active users. At the beginning of EdShare and Language Box development, we undertook a service-driven analysis of these sites in order to identify what services they offer to their users, and compared these services to existing repository systems.

Research repository software such as EPrints and DSpace offers a number of advantages to depositors. Papers in these repositories are highly visible to search engines, they include standard metadata making it easier to cite the papers, and they offer a permanent record of the depositor's work, which will even remain online if the author leaves to join a different institution. We might summarize this by saying that research repositories offer an Archiving service.

More formal online collections of teaching resources, such as OER Commons, act as a publicly available index of material. They offer a single point of search and provide educational oriented filters (such as filtering by level). Some, such as Connexions [ 18], may also provide packaging services, where it is possible to order physical books based on the open material in the repository; effectively offering a publishing service.

This service-driven perspective makes it easier to understand why everyday teaching and learning repositories might fail. Teachers and lecturers rarely want to archive their teaching materials, which are often dynamic documents, changed and updated regularly, that have their own life cycle. They may also be reluctant to publish such working documents, which may not have been subjected to the same quality control as research publications.

Looking at the Web 2.0 sharing sites revealed a number of alternative services that may be appropriate for teaching materials. We identified three common services.

    1. Hosting. The key benefit with these sharing sites is the ability for nontechnical users to put complex content (for example, video) online and inside the browser window. They also provide a third-party storage service and allocate all uploads their own URL, allowing users to access their materials from many places.

    2. Organization. Most sharing sites have a number of organizational mechanisms enabling users to make sense of the materials that they have uploaded. The most common of these are tagging and explicit collections (such as sets in Flickr, or channels in YouTube). These organizational elements also usually have a URL of their own.

    3. Community. The sharing sites encourage users to see themselves as part of a larger site community. Users are often given their own profile page, which not only provides some information about that person, but also acts as a place where all the information about their activity can be aggregated together. Other mechanisms such as recommenders also encourage people to see their materials in the context of other people's uploads.

Sharing does not appear to be an explicit service. This may seem counter intuitive, but people are using the hosting service to help them communicate, often with an explicit audience in mind. For example, someone uploading a video to YouTube intends for others to see it, but not necessarily to repurpose it. Sharing in the greater, altruistic sense seems to be a side effect of more pragmatic selfish motives, and it is well understood [ 28] that the majority of users who take part in a community will not necessarily contribute.


4.1.3 Design of EdShare and the Language Box Our experiences of working with CLARE and our analysis of Web 2.0 sharing sites brought us to the conclusion that a repository for everyday materials needed to be focused around hosting, organization, and community, with a core philosophy of simplicity in order to lower any barriers to making deposits.

The service-driven approach gives us a framework for designing the repository system. Web 2.0 features such as tagging and commentary may be part of the solution, but will only work if they are used to support the specific services that the repository is offering.

Both repositories have simplicity at their heart. We used a minimum manual set of metadata, requiring only that users name their deposits and provide a minimum set of automatic metadata such as time and date of deposit, attribution, etc. The few optional metadata fields are based on well-understood terms (such as language) or are nonrestrictive (such as a description and tag fields).

We do not use explicit content packaging, instead users are encouraged to think about the resulting upload page as the package. So, the act of aggregation happens during upload rather than before. Users downloading a deposit will be given the same object that was uploaded, free of any site-added metadata.

The philosophy of simplicity extended to navigation. We implemented the most exhaustive search functionality we could, so a search term will be matched against titles and descriptions, against all other metadata fields, and against the textual content of the upload itself ( Fig. 2 shows the EdShare search results page). In addition, when metadata are displayed, it is turned into a link that acts as a search for that metadata value (for example, clicking on the attribution value will show all other materials with that same attribution). This gives the two repositories a sense of depth and interconnection. The main service we concentrated on was one of hosting.



Fig. 2. EdShare search results page.




We developed a common flash-based preview tool that shows a variety of media inside the Web page using a "coverflow" style interface. Videos and sound files can be played in place, and users can look through the documents and slide shows page by page ( Fig. 3 shows a resource page from the Language Box).



Fig. 3. Language Box resource page.




Double clicking on the preview will load the original file—this is useful for some documents that cannot be effectively read within the preview tool. Users uploading their materials can choose to make them public, private, or can restrict them to selected groups of users (in EdShare, this might be members of the University, while in Language Box, it is registered users).

We also support organization by allowing users to write tags, make notes, and author collections. Tags are public, both in the sense that they can be reused by other authors, but also that anyone can tag a resource, even if they are not the author. Notes are private comments that can be attached to resources by the author. Collections are sets of resources that become deposits in their own right, with their own name, description, tags, and comments. Users can include other people's materials in their collections, and we are not prescriptive about what collections are used for (for example, collections by class or by topic); instead, we allow users to appropriate the collections mechanism in whatever way they see fit ( Fig. 4 shows a collections page from EdShare).



Fig. 4. EdShare collections page.




We have created a file-browser style collection management system using dynamic html and javascript. This acts as a management page for users. It provides a quick and easy way for them to create new collections, and drag and drop materials from one collection to another. It also gives us a good metaphor for other features, for example, bookmarks are implemented as a private collection.

Lastly, we support community through profiles, comments, and activities. Profile pages are particularly important in the Language Box as it is a community repository, and therefore, perceptions of quality are tightly bound to the identity of authors. Profile pages also act as a home for users within the repository and show their most recent uploads and other activities ( Fig. 5 shows a profile page from the Language Box). The commentary mechanism is based on the EPrints SNEEP 8 plugin. Initially, we were unsure how users would react to other people being allowed to comment on their materials. However, we have found that comments provide a good basis for the contextual information (about usage and experience) that was missing in CLARE. Like collections users can appropriate the comments mechanism as they wish, for example, to report on experience, make statements about quality or attribution, suggestions for use, or to indicate alternative resources.



Fig. 5. Language Box profile page.




In the Language Box, we have also explored the idea of explicit Activities. These are new deposits that build on existing resources, by adding activity or task information (which may involve uploading new files). For example, an original video resource could have an attached classroom activity that included a worksheet in PDF format to be given to students. Activity pages show the original files alongside the files required for the activity, and have their own URL.

These mechanisms enable users to engage with their repository at different levels, participating in the ways in which they are most comfortable—uploading, tagging, collecting, and remixing (in the form of activities). Not all users will participate in all of the ways offered, but by making it as simple as possible to use the two repositories, and by focusing on explicit services that give real benefits to the people uploading, we are minimizing the effort required to use the repository, and maximizing the perceived benefit. It is from these seemingly selfish services that a sharing community will emerge.

4.2 Policy and Practice and Change Agency
The previous section described the motivation and implementation of a technical infrastructure that diverges significantly from earlier systems that we (the learning object repository community) have offered. However, while a suitable, appropriate, and usable technical infrastructure is an important component of success, it is widely acknowledged, e.g., David Wiley's keynote to ICALT 2006, that major barriers to further progress are human rather than technical as factors; the culture and practice of preparing learning materials needs to change.
As part of the EdShare and Language Box initiatives, we invested significant time and effort working with the user communities to identify the particular benefits to individuals that will encourage them to participate in a sharing community and to identify, and hopefully reduce, the barriers to change. There have been some important differences in approach between the two initiatives.
Language Box is a community of practice repository. Its objective has been to persuade language teachers to participate in and feel membership of this community.
In contrast, EdShare is an institutional repository, and the focus has been on persuading teachers to use the repository for their everyday teaching materials in very much the same way that they often use the VLE—as a place to store their resources so that their students may find them. However, EdShare allows the teacher to share their resources with a much wider audience (the whole university? open access?). Debate has arisen around the value and benefits of wider distribution, and the problems that arise related to ownership, IPR, copyright, and quality control. The majority of the discussion that follows relates to our experiences of engagement with academics at the institutional level.
The context of this work, we should explain, is that the University of Southampton has a management structure in which much of the academic decision-making is devolved to the 20 schools. We have engaged with each school by attending their school academic boards, arranging developmental events in the schools, and by use of university teaching and learning discussion forums. But most importantly, we have identified champions in the schools who are acknowledged as exemplary teachers, and we have engaged them as change agents. In some cases, we have used small amounts of project funding to assist them in transferring their resources into EdShare.
We have identified motivations of use built from established practice. We observe that most teaching colleagues have collections of accumulated materials, already organized on their local hard disks or in the learning environment. The chance to tag, index, and store resource centrally and securely in an accessible manner has been welcomed. One colleague responded to a description of the service "I'm probably going to use it as a cloud service for all my teaching material instead of my hard drive. At least I know its safe and I can access it from anywhere." In addition, it has been possible to persuade some people of the "moral high ground"—that learning resources produced by a publicly funded teacher, or as part of a publicly funded project, should be placed in the public domain. In moving from early adopters to the majority, we aim to harness the drivers and overcome the identified barriers to use. Below, we summarize key points identified thus far.

4.2.1 Ownership of Teaching Resources In the UK, as in some other places, the teachers' assumption is that they own the resources that they create. On the other hand, universities often have a policy that states that resources that are used for teaching and assessment are copyright of the university, giving the university the right to continue to use and develop the resource. While there are few cases of universities aggressively pursuing these rights, there have been cases where teachers, on leaving a university, have deleted all the resources they had created in the VLE—or have even refused to put teaching resources in the university VLE in order to prevent the university from asserting its rights. Teachers rarely understand the distinction between ownership of IPR and copyright, and suitable education and understanding is necessary to diminish their concerns, which will, otherwise, be a barrier to sharing.
4.2.2 Copyright Issues Copyright of resources developed for teaching is also an area of major concern for teachers, and the law varies from country to country. Examination of a typical PowerPoint presentation, for example, often reveals multiple pictures and other information that have been reused from elsewhere—most often located by simple open Web browsing. These resources rarely carry a license for reuse, such as Creative Commons. 9 Universities often require that a teacher who puts a resource into the repository confirms that all material was either created by them, or that they have an explicit license to reuse. This has many implications, since few universities have the workflows in place to clear copyright on all such everyday teaching materials, or the infrastructure to store the evidence.

On the other hand, in research publication, it is quite normal to quote the work of others, and cite it accordingly. Would not most authors be quite happy for teachers to do the same? Clearly, there are some limits: Commercial companies are often sensitive about their materials being used out of context, and organizations that make a living out of collecting and adding value to data are understandably protective of their copyright. But a requirement to avoid such sensitive areas and carefully cite sources of materials, coupled with a clearly stated "take-down" policy would put most teachers at ease. This requires that universities develop a low-risk policy as opposed to a no-risk policy. We continue to work with our legal department on this issue.


4.2.3 The Benefits of Wider Sharing Teachers have been quick to identify some of the benefits of sharing their resources across the university.

    There has been much investment recently in preparing generic resources (academic skills, English as a foreign language, and research skills). The VLE has not been the ideal vehicle for these resourses, as students need to locate the course and enroll and teachers may need to duplicate resources in multiple locations. With the resources in EdShare, the task is simpler for the teacher and easily found by the student.

    Senior staff concerned with overview and quality assurance of our courses benefit from the ability to see the teaching materials being used by courses.

    The university has promotion based on teaching excellence, but it is difficult to identify many metrics to provide evidence; in the past, the publication of a textbook might have provided such evidence. We are looking now at how deposit statistics and downloads of resources might be used as metrics, in much the same way as, for example, Google Scholar citation statistics are used as evidence of research excellence.

    Using EdShare has enabled colleagues to identify common areas of the curriculum across disciplinary boundaries. It has facilitated sharing of educational resources inside the university.

Perhaps, more interesting is the extent to which the design of the system has acted as a mechanism for encouraging open access. This happens as sharing within the school or university has a low threshold initially; people who are not sufficiently confident in aspects of their resources are generally happy to share within their closer community.

However, once they start sharing, the search engines will index the description of the resource they have shared. Our experience is that we now receive a regular stream of e-mails from outside the university asking for access to the files associated with resources that they have discovered via search engines but have been shared only within the university. In all cases that we are aware of, the owner has been happy to share with the person who has requested access, and in many cases, such requests have persuaded the owner to make the resource open access. When we have asked our teachers why they had not made the resource open access in the first place, the answer has always concerned either lack of confidence in the applicability of the resource in a wider context, or else licensing concerns.

Teachers have also identified a number of benefits from sharing their resources as open access. An important point here is the esteem that is gained by the individual, their School, or the wider university from providing such resources to the wider world.

5. Results and Evaluation
We approached the evaluation of our work from a number of angles. We have examined the usage metrics to understand how EdShare is being used. We have compared these metrics with the UK national education repository, looking for evidence of different sorts of depositing and download behavior. We have also consulted our users extensively. In the case of the institutional repository, EdShare, we have polled our frequent users electronically to understand their motivations and expectations. In the case of the community repository, the Language Box, we have entered into a codesign approach by organizing a number of workshops, where we have presented our ideas and sought the users' reactions and developing requirements. In this section, we summarize some of the results we have acquired.
5.1 Metrics
Between its official launch in October 2008 and the time of writing (March 2009), our institutional repository has gained 622 resources consisting of 2,477 individual items. Fig. 6 shows the range of file types being shared. We observe that there is a wide range of data types that might well be representative of the file types used in day-to-day teaching, with PowerPoint presentations as the most used, followed by PDF. There is a much smaller number of video and audio "podcasts," which are more time-intensive to produce. This data would appear to support our claim that our approach is giving users the confidence to share their everyday teaching materials.


Fig. 6. The proportion of file types in EdShare.




Fig. 7 shows the different ways in which the resources (the actual files, rather than the descriptions) have been shared with others. At deposit in EdShare, the default permission level for visibility is "University." It can be seen from these figures that a small number of people have chosen to confine the visibility to their school—indicating possibly a licensing or privacy restriction (e.g., medical images), but probably often indicating a lack of confidence in sharing more widely. Three times as many people have chosen to share with the whole world, and this has produced real benefits when it comes to the download metrics. The "selected users" option occurs when a user chooses to share with only a named set of users (possibly external). This option is to allow community approaches to developing and sharing.


Fig. 7. The visibility of files in EdShare.




We have analyzed the distribution of the deposits to see if there is any subject bias, by identifying the academic discipline all of the people who have deposited more than 10 resources. There was no particular pattern except that the Library and the School of Electronics and Computer Science were high up the list, probably as they were the hosts for the project. The School of Humanities had the largest number of deposits.
Fig. 8 shows the EdShare download statistics particularly since the official launch in October 2008. In these figures, we distinguish between downloads from within the university and external downloads, and also distinguish between downloads of the description of the document (which is always visible to the whole world) and downloads of the files themselves. Since the launch in October 2008, there has been a total of 87,000 downloads (we have attempted to exclude bots from these figures), most of which have been of the descriptions of the documents—roughly half of these coming from off campus.


Fig. 8. EdShare access metrics. Showing access/downloads of actual resources and descriptions from within the campus and from outside the campus. The repository was released in October 2008.




The number of downloads of the actual files is relatively small—11,500 downloads of the files as opposed to 77,500 views of the resource descriptors. This might indicate that teachers are tending to download a file and make a copy of it in their VLE site, rather than linking to the resource in EdShare. It is an important objective of our future work to make linking to resources in EdShare simpler for people who are less IT literate.
5.2 Benchmarking against National Repository
JORUM is a free online repository service for teaching in UK HE and FE institutions, to build a community for the sharing, reuse, and repurposing of learning and teaching materials. Thus far, it has not been an open access community although it is now being remodeled to engage with the OER agenda. It is a useful benchmark because it is a nationally visible project that is backed by the JISC funding agency; however, it is not an OER benchmark.
Table 1 compares EdShare metrics with JORUM quick statistics published on their Web site. We should make it quite clear that this is not a comparison of like with like. Access to JORUM is only available to registered users from UK Higher Education, whereas EdShare is open for anyone to browse and download the documents marked as open. The importance of this comparison is to demonstrate the different behaviors that are developing in the EdShare community when compared to more tradition LORs; users are depositing resources at a far higher rate and being rewarded with many more downloads. We believe that this again demonstrates the extent to which users feel empowered to share their everyday teaching resources as they have control of the audience and can progressively disclose the documents as they gain in confidence, whereas in JORUM, there is an assumption that only exemplary resources that are complete and finished are deposited.

Table 1. Metrics: EdShare Compared with JORUM


5.3 EdShare User Poll
A survey of our most active early adopters provided some insights which suggest some of the comparative strengths of the EdShare approach over more traditional approaches to resource repository. Some of those reasons appear to be the strength of the local system over the national, while others point to particular strengths of the user interface and system functionality. Responses and specific analysis are summarized below.
Q1 Have you found it easy to add content to EdShare and describe the content in a way that satisfies you?
Considering adding content, the majority replied that adding and describing content was "mostly" very easy and intuitive, although one highly experienced learning technologist identified problems adding complex learning objects and video podcasts.
When describing content, responses were largely positive; and it would appear that the uploading form which only requires " core metadata data" is "about the right length," although a librarian commented that "only limited options are available to describe URLs." There was also some evidence of learning about the system through familiarity, for example, one academic observed: "My descriptions became more elaborate over time."
Q2 Once you have added content to EdShare, have you found that you have returned to the EdShare record to "tweak" the description or the tags you have added?
Echoing the response in Q1 which suggested learning through increasing familiarity, nine respondents confirmed that they—either returned to tweak, or revise as they added more content and their understanding increased about how EdShare works, behaves, and presents.
Q3 Have you recommended use of EdShare to colleagues/students—either as a way for them to present content or as a way to find content (please describe which)?
Nine respondents acknowledged recommending use to colleagues, including building it into Staff Development courses and making decisions to use the service in their own Project work.
Three people differentiated between their behaviors with staff and students. For students, they found it less obvious that they would recommend EdShare. A student said: "If there is stuff that would help us with our modules, we'd love it—market that to us."
Q4 What advantages does EdShare have over other similar services you may use (please say which others you do use)?
There were a number of comments along the lines of "I'd say it was on a par with SlideShare," "Simplicity," "It's good for storing resources," and "It looks very current, whereas JORUM looks quite dated."
The lightweight metadata were the subject of five of the comments, typified by the comments of one experienced educational developer "It's a local repository for local people. The metadata are lightweight. I have used JORUM, but it's pretty user-hostile." A learning technologist pointed out: "the tagging and search facility on EdShare works REALLY well. Blackboard [the institutional VLE] has no search facility."
Q5 What would you like to change about EdShare?
Here, we received a number of constructive comments, many of which related to improving the handling of complex resources and improving the description of externally linked items. (URLs outside the University.) Comments also related to useful changes to the interface, and suggestions for extra functionality—such as conversion between file types and interfaces to Del.ici.ous.
An educational developer, keen to encourage wider use across the university suggested: "better integration with Blackboard—an easy method to include links to EShare resources." Some requests implicitly referred to with social software use, for example, an academic said: "Recommendations of similar material based on my shares when tagging, to see if similar tags should be reused by me." Similarly, a student commented: "There should be a way of saying 'This topic is related to this one,' or 'This major concept is also used by this doc.'"
Q6 What do you most like about EdShare?
Many responses referred to the attractive interface, the ease of use, and the similarity to existing Web 2.0 applications. Around half of the respondents commented on the value, they placed in sharing, either contributing or being able to access and use other people's materials "Easy to use and the Philosophy of sharing" said one Educational Developer. One respondent highlighted the importance of openness—being able to browse the material and download without having to register on the system: A Web developer said: "There are so many nice things about EdShare. I really think that this will help breaking down the walls among resources that were created when staff were asked to place and develop their teaching resources in Blackboard (the institutional VLE). EdShare will also provide a home for important content that may not be immediately related to the teaching of a specific module, e.g., academic skills, student evaluation, revision strategy, etc., for academics to collaborate on and develop further."
Academics' comments included "Opportunity to share the teaching resources that I am proud of with the world!" and "I like seeing the latest items from other people."
5.4 Community Reaction
A key part of our participatory design approach was to work with the target community right from the start of both projects. With EdShare, this involved visiting Schools and introducing the repository to staff members, and with the Language Box, we held community workshops in which we invited Language teachers from across the country to explore new features.
These community sessions were designed as practical, hands-on explorations so that users could feel that the software was designed around the teaching and learning services they needed. The progression of the workshop content followed our "perpetual beta" release cycle. At our first workshop, we showed storyboards and technology mock-ups to focus discussions on potential features, as well as the look and feel. At the final workshops, we were working with the completed Language Box and were able to explore attitudes to abstract concepts such as the relationships between resources and activities because there were concrete instances as examples. Working in this way gave us a clear direction for designing services that met the needs of the teaching and learning community. It also allowed us to evaluate changes in attitudes and practices in our emerging user community.
Our user engagement feedback confirmed that a focus on simplicity was of the highest importance. In the earlier workshops, we undertook paper-based exercises to explore the amount of metadata that users were prepared to fill in, compared to how much they would require from other people. Our discovery was that users typically found around seven fields to be an ideal compromise (name, description, tags, type, owner, permissions, and language)—and expected the system to fill in three of these automatically (type, owner, and permissions) at least with default values. Teaching practitioners are astute in their use of time: if too many decisions have to be made, it deters them from uploading resources into repositories. They felt that many typical metadata fields (such as required resources, level, scaffolding, etc.) seemed to address other people's needs rather than their own.
At a later workshop, coinciding with the first beta release of the Language Box, users needed no encouragement and little support to start uploading the digital resources they had brought with them. The simple and direct workflow received a very positive reaction: "Is that all I have to do? I never realized that it could be so easy and so quick to put things in. I can see my resources in the preview already."
It was clear from this exercise that the in-line preview tool was extremely popular and comments from users confirmed that they were able to judge whether a resource may be useful much more quickly by seeing it rather than by interpreting complex metadata. It also helped them to see that they were not so much depositing their resources in a repository, as creating an online representation of the resource. "Because I can see the resource, I don't see the point of lots of metadata... But why is it called a preview? Isn't it the resource itself?"
As expected, attitudes to sharing were mixed; however, this reticence was even deeper than we had anticipated. We had assumed that sharing would not be a sufficient motivator for users; however, in some cases, it acted as a negative deterrent, with users unwilling to use the system if their resources were necessarily made public. This highlighted the value of the permissions system built into the EdShare repository. We may need to address perceptions about the quality of everyday teaching materials in the future, a common problem that is exemplified by the comment: "I have some materials that are nearly ready, they need some more work, then I'll put them in."
Some people were also worried about profiles. Partly, this seemed to be about a natural shyness and inhibition about personal exposure, so, for example, some participants didn't want to make the profile about them as an individual and preferred not to fill in their interests. "I am not sure that I like other people to see my personal interests, or my picture. I don't mind others being able to see my professional data." This may mark their hesitancy about their transition from being consumers of other people's resources to becoming active participants and providers themselves. Nielsen believes that 90 percent of Web 2.0 users are lurkers, 9 percent contribute occasionally, and 1 percent contribute heavily [ 29]. This implies that there needs to be substantial cultural change before such concerns disappear. Perhaps, the personal benefits of profile data will become more obvious as community features are released (esteem factors, building links with other community members, etc.) and reticence among our own users will diminish.
One change we have observed in recent repository projects is in attitudes to copyright, probably because more people are now familiar with Web 2.0 sharing sites and so the concept of sharing is more routine. Users were still concerned about their professional ability to confidently deal with copyright and licensing questions in relation to: 1) using existing digital resources created by others and 2) their own resources in relation to their institution. Both repositories developed guides to usage, attribution, and copyright in an effort to make users more comfortable.
In the workshops, it also became clear that users were enthusiastic about the potential to remix other people's material—even if this was only at a high level (such as the Activities in the Language Box). "Another option I really liked was the ability to add your own items to an existing resource, i.e., if you really like the video content, someone has posted up but create your own accompanying worksheet you can add that for others to use if they wish."
However, through our interactions with teachers, we have come to understand that we must be cautious about introducing complex remixing of materials. As computer scientists, we have learned to understand and use abstract models, and have become very good at automatically deconstructing and classifying information. This is not a universal skill, and many users find it difficult to create mental models of exactly how things like collections, revisions, and activities actually work.
For example, at one of our later workshops, we asked workshop members to try to classify typical language materials as either Resources or Activities. Both terms are well understood by practitioners and are part of their everyday vocabulary. They had difficulty with the classification because they could not easily separate a resource, for example, a video of a conversation, from its use, a comprehension activity which uses that video.
However, classification for practitioners became easy when there were explicit files involved. For example, when uploading the video, they were not clear as to whether it was a resource or an activity, if the comprehension exercise was written and uploaded as a separate PDF document, then they clearly understood that the video was the resource and the PDF was the activity. As a result, we developed the Activity Pages in the Language Box as a mechanism for users to extend existing resources with new files, thus creating a new kind of "doing" page. This seems especially popular with URL resources, as it means that teachers can create an activity around an existing Web resource very easily. However, the inclusion of activities changes the way the repository is perceived, making it look more like a learning tool than an archive.
There is some concern that users will become confused about how the repository relates to their VLE. When asked we discovered that 89 percent of teachers in our community already use a VLE to deliver content. Integration with VLE systems is therefore essential if we are to fit repositories into users existing working patterns.
Our experience is that teachers understand the relationship, and see the repository as a device to manage resources that can be referenced by the VLE: "But the real difference and one that perhaps gives Language Box more potential is that the material can be visible, accessible, and usable by students without needing to transfer it to the VLE (although you can also do that if you want to, but why bother when you can link)."
6. Discussion and Conclusion
This paper has identified some of the core challenges to successful deployment of teaching and learning repositories, using our work with two systems, EdShare and Language Box, to show how these challenges might be overcome in the institutional and community contexts.
In this final section, we will draw out some of the key lessons that we have learned that we hope will be of benefit to others deploying similar repositories, and to the OER movement in general.
6.1 EdShare and the Language Box
Our conclusions are driven by our experiences with developing the EdShare and Language Box software, and developing close working relationships with our early adopter user colleagues within the institution and the language-based community of practice. Through our workshops, questionnaires, and activity logs, we have evidence that users engage with their local or community repository to a greater degree than with remote systems. In addition to the activity around the deployments, our activities have generated interest from other institutions surrounding our technical approach and the software that we have developed. Several institutions are planning to work with us to deploy their own versions within the next year, further community of practice implementations are planned in the humanities and social science.
There are some remaining challenges. In our community repository, it is clear that there are tensions in the way that users see and understand the system. It has been designed as a home for their online materials, but many users assume that it is a public bank of resources. In reality, the system functions in both of these ways, but viewing it as a bank raises concerns in users minds about the quality of the resources inside, when, in fact, quality mechanisms (such as comments and user identity through profiles) help to manage quality in an open system.
Working with the institutional repository, it has become clear that we need to clarify the relationship between EdShare and the institutional VLE, perhaps by developing more explicit connections between the two systems (e.g., enabling users in the VLE to directly deposit into EdShare). Our users have established workflows, which it is important that we respect, at the same time that we encourage and support them to routinely regard their materials as open educational resources.
6.2 Learning Objects in the Wild
Key to the success of both repositories is the underlying philosophy of simplicity. The most important way that this is manifest is in our deconstruction of learning objects. Both repositories are concentrated around everyday learning resources, and that means a reduced set of metadata and no content packaging. We believe that this simplicity contributes to people's understanding and positively affects their willingness to use the system.
However, this only works because the systems themselves compensate by providing a new context to uploaded resources. The resource pages in the repository act as readily understood and readable packages, and the attention metadata generated by the system (download statistics, collections membership, comments, etc.) make up for much of the missing quality data. It is the pages generated by uploading resources that are analogous to Learning Objects, not the resources themselves.
This has enabled us to have more meaningful conversations with users about what reusability means, and has also enabled us to begin to explore remixing within the sites—effectively a form of Learning Object authoring— without any technological complexity getting in the way. Users simply see adding new files to resource pages, or adding comments on potential use, as a natural extension to their normal uploading and browsing behavior.
6.3 Working with Institutions and Communities
Both of our repository projects have involved substantial engagement activities alongside the technical design and development. For EdShare, we have worked closely with a variety of potential user groups within the University and have also engaged with management teams in key areas (such as e-learning). For the Language Box, we have made extensive efforts to meet with the community, disseminate our work, and gather feedback through a number of workshops around the country and also through direct site visits in an effort to create local champions for the repository. We have been careful to build upon the existing strength of the communities of practice (for example, by working with existing community groups such as the University of Southampton e-Learning Implementation Group and the HEA Languages, Linguistics and Area Studies Subject Centre).
Both forms of engagement contain challenges. Community repositories do not have institutional incentives to engage users, but utilize content and community to encourage and explore a wide range of behavior. By contrast, an institutional repository has to be more carefully placed within the policy framework of the institution, but can seem more immediately relevant to users, who see the institutional backing as a green light to engage.
In both cases, the engagement activity builds trust in the effectiveness of the system itself and also in the ability of the development team to deliver a robust, sustainable online resource that users feel comfortable to invest time in. The use of a well-known repository framework such as EPrints also gives users confidence in the system.
It has also allowed us to develop the repositories in an agile manner. Well defined but tight development and deployment cycles have enabled us to respond quickly to users feedback and suggestions. This was especially important in the early iterations, where supporting key use cases was valuable in attracting users. Demonstrating that we are reacting to feedback also reinforces the users relationship with the system team and the development team.
6.4 Changing Culture
Perhaps, the most important finding from our experiences is that for a teaching and learning repository to succeed, there must be a change in culture, and that this change requires alignment of technical, community, and institutional factors.
At the technical level, the software deployed must not only be straightforward and useable, but must also offer a valuable service to its users. Users see the value of sharing, but altruistic sharing is not a good enough incentive on its own. In our projects, we have learned from emerging behaviors from Web 2.0 sharing sites, and encouraged users to see the repository not as an archive, but as a living online home for their materials. Users liked lightweight metadata, but perhaps integration of automated metadata generation tools such as those developed at KU Leuven would be a powerful addition [ 30].
Within the user community, it is important to build up the culture of sharing by removing barriers and demonstrating real benefits. Often what can seem to potential users to be insurmountable problems (such as worries over quality) fade away in the light of a real system. In our own projects, we have found that engaging with user communities not only helps to refine the software, but also establishes and builds trust.
Finally, there is the institutional perspective. This can be critically important, even for a community repository, as institutional policies guide the behavior of all users. For example, users may feel comfortable dealing with copyright on an individual basis, but not in a professional context under the auspices of an institution. Clear institutional policies give users confidence that they are able to engage, and that they will be supported by their institution if problems arise. The attitudes of the institution toward open content will inform the views of its staff, further incentives (such as including OER contributions into personal development processes) will help users to see engagement with the repository as part of the usual behavior, in the same way as they view the use of a VLE.
Technical, Community, and Institutional factors must be addressed together if teaching and learning repositories are to become accepted by staff and embedded into practice. Our experiences show that sharing our teaching resources, and learning to open up our content, is something that must be carefully fostered with practical software, consistent community engagement, and supportive institutional policies.

Acknowledgments

The EdSpace Project, which developed EdShare, was funded under the JISC Institutional Exemplars Programme, and the Faroes project, which produced LanguageBox, was funded by the JISC Repositories and Preservation Programme. Other Southampton projects mentioned here were funded by JISC and Eduserv. The authors would like to acknowledge the development team, Dr. Tim Miles-Board, Sebastien François, and Patrick McSweeney, and the community support, particularly from Miguel Arrebola of the University of Portsmouth and Kate Borthwick of the HEA Languages, Linguistics, and Area Studies Subject Centre at the University of Southampton. One of the authors (Debra Morris) is on secondment from the University Library.

    The authors are with the Learning Societies Lab, School of Electronics and Computer Science, University of Southampton, Southampton SO17 1BJ, United Kingdom.

    E-mail: {hcd, lac, ymh, dem, saw}@ecs.soton.ac.uk, jessie.hey@gmail.com, d.morris@soton.ac.uk.

Manuscript received 7 Mar. 2009; revised 20 May 2009; accepted 29 July 2009; published online 13 Aug. 2009.

For information on obtaining reprints of this article, please send e-mail to: lt@computer.org, and reference IEEECS Log Number TLTSI-2009-03-0036.

Digital Object Identifier no. 10.1109/TLTSI.2009-03-0036.

1. http://cosl.usu.edu/.

2. http://www.jorum.ac.uk/.

3. http://www.jisc.ac.uk/.

4. http://arxiv.org/.

5. http://www.eprints.org.

6. http://www.smete.org/smete/.

7. http://www.oercommons.org/.

8. http://sneep.ulcc.ac.uk/.

9. http://creativecommons.org/.

References



Hugh C. Davis received the BSc degree in ship science form the University of Southampton in 1981, the MSc degree from City University in 1987, and the PhD degree in computer science from Southampton in 1995. He worked as a social worker and a teacher before starting an academic career at Southampton in 1987. He has a long history of research in hypertext systems and technology-enhanced learning with more than 200 published papers, three Best Paper Awards, and 30 grants. He is currently the university director of education responsible for e-learning at Southampton, and leads the Learning Societies Lab. His current interests focus on the use of technology as an agent for educational change. He is a member of the BCS and a professional member of the ACM.



Leslie Carr received the BSc degree in computer science from the University of Southampton in 1986 and the PhD degree from the University of Southampton in 2005. He is now a senior lecturer of computer science at the university and has published more than 100 papers on hypertext, Web, and repositories. He is the director of the EPrints project and blogs about his experiences as RepositoryMan.



Jessie M.N. Hey received the MA Oxon physics, CertEd, and DipLibInf St degrees, and the PhD degree in resource discovery in digital libraries from the University of Southampton, UK, following her more recent work in providing and researching hybrid libraries. She is a member of the Chartered Institute of Library and Information Professionals and the ACM. She has been providing knowledge management services in industry, research laboratories, and academia since 10 years. She has been active in the worldwide community of Open Access to Research and Scholarship over the past decade—researching practical and innovative ways of developing archives, including the now established institutional repositories.



Yvonne Howard received the BSc degree in computer science from the University of Southampton in 1997 and the PhD degree in evolutionary software process modeling from the University of Southampton in 2004. She is a senior research fellow in computer science at the university. She is the project manager for the Faroes project. Her research interests include in the use of Agile Software Engineering Techniques, particularly in placing users at the heart of the design process and using rapid feedback to build the capacity for innovation in the user community. She has published more than 30 papers. She is a member of a UK JISC Special Expert Group that aims to provide a repository of contextualized models of the higher education domain to support innovation in the sector.



David Millard received the BSc degree in computer science from the University of Southampton in 1997 and the PhD degree in contextual hypermedia systems from the University of Southampton in 2001. He is now a senior lecturer of computer science at the university, a member of the ACM Special Interest Group on the Web, and has published more than 100 papers on hypertext, Web, and e-learning.



Debra Morris received the BA (Hons) Oxon degree in 1983, the MA degree in 1987, and the MLib degree in 1988. She works as the project manager for EdSpace at the University of Southampton, and an e-learning lead for the University Library as well as Liaison Librarian within the University Library. She is a member of the University E-Learning Strategy Group, University E-Learning Implementation Group, and the University Learning Spaces Strategy Group. In 2005, she was awarded a University of Southampton Vice Chancellor's Award for Learning and Teaching. She is a chartered member of the Chartered Institute of Library and Information Professionals.



Su White received the BSc (Econ) from the London School of Economics in 1983, the postgraduate degrees in computer science and in education from the University of London in 1985 and 1989, respectively, and the PhD degree, looking at organizational culture and learning technologies in higher education, from the University of Southampton in 2006. She is a senior lecturer in the Learning Societies Lab at the University of Southampton. She has published more than 80 papers in the areas of e-learning, science and technology education, and educational development. She is a member of the ACM Special Interest Group in Computer Science Education. In 2005, she won the Vice Chancellor's Teaching Award for innovations in computer science teaching.
9 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool