The problem is that ontologies are very hard to engineer, despite the availability of editing tools like Protégé. Just like MS Word does not help much in writing a meaningful essay, creating a consistent map of meaning in a given domain depends on the skill and art of the knowledge engineer and always reflects his or her individual viewpoint and understanding of the domain. As soon as the domain gets more realistic, complex, and interesting, people's viewpoints no longer agree and they start interpreting things in different ways. Different communities use different naming agreements. It is hard or impossible to agree on the semantics of relationships—people interpret them in different ways, even when they agree that there is a relationship between two entities.

In reality, most of the systems that claim to use ontologies are based on taxonomies or topic maps (categories connected with certain types of relationships). These simpler semantic representations allow straightforward mapping of terms using applications like WordNet. Just like translating, word by word, a text in a foreign language using a dictionary, this kind of mapping excludes the semantics of the relationships among objects (the relations can be mapped in name only). Research is currently going on to allow for more advanced structural ontology mapping, but the practical application of such mapping is still limited.

Even if there is agreement among designers about using a particular simple ontology, from a user's point of view, it imposes a cumbersome and inconvenient way of organizing or finding content. I will illustrate this with an experience with an early version of our Comtella system.

Yet, finding the required category in a subject category index is challenging, as any author who ever published a paper with the ACM or the IEEE knows. So, we simplified the ontology by selecting only the top categories corresponding to areas in which our department has active research projects and limiting the depth of hierarchy of the topics to three (instead of six—the depth that the ACM category index reached at that time). We then organized the ontology as a set of three hierarchically nested menus (see Fig. 2 ) which were used to annotate a new contribution to the system and to search for articles that have been shared. The system had very little use—the users indicated that it took too much effort to both categorize a contribution and to find a contribution using the menus. The biggest difficulty for users was not knowing the "map" of categories: To access the right level-3 menu, one had to make the right choices in the level-1 and level-2 menus. Some of the top categories were vague and the users had no clue where the particular level-3 category they were looking for was hidden. This confusion resulted from the absence of a common agreement about how categories in computer science should be structured. The ACM category index, like any other category list, even though likely resulting from the dedicated work of a committee of experts, reflects the particular viewpoint of the committee that designed it. Luckily, in the design of the top-level menu, we had included a category "All" (Other) in case the user was not able find the category they were looking for. Interestingly, most of the user contributions were labeled under this category. The users explained later that this was the easiest way to share an article. This also emerged as the easiest way to search for articles, since all the articles were in this category. Yet, it worked well only because there weren't too many articles shared and the list of results was tractable. However, categorizing everything in the category "All" is the same as not having categories at all. This is an anecdotal confirmation of the "Everything is miscellaneous" postulate by David Weinberger in his recently published book [ ⁴⁷ ], which discusses if it is possible to impose one ontology or unified classification schema on diverse and autonomous users. Weinberger answers this question negatively, with many convincing examples, and argues for collaborative tagging, a simpler, non-AI-based approach supporting "findability" rather than "interoperability."

However, the tags in a folksonomy are not machine-understandable. From the machine's point of view, the tags are discrete labels with no relationship among them. The tags can be used for retrieval but not for machine reasoning and decision making. A machine cannot say how two documents, tagged with the same tag(s), are semantically related to each other or why they are similar. So, it would be, for example, very difficult to create a sequence of content from tagged learning objects. Tags are good for a "one-shot" retrieval by a user but are insufficient for inference or reasoning. There has been some interesting work comparing whether folksonomies capture the semantics of a document as well as automatic term extraction. Brooks and Montanez [ ³ ] did an experiment with the 250 most popular tags in Technorati. They grouped documents that share tags into clusters and then compared the similarity of all documents within a cluster. Their hypothesis was that documents in a cluster that shared a tag should be more similar to each other than a randomly constructed set of documents. As a benchmark, they also compared clusters of documents known to be similar (based on Google search results for the tag). Finally, they constructed tags automatically by extracting relevant keywords from documents and used these tags to construct clusters of documents. This was intended to determine whether humans were better at categorizing articles by tagging than automated techniques (semantic lexical analyis). The results showed that articles sharing a tag had a 0.3 pair-wise similarity; the articles considered similar by Google had 0.4. Automated tagging performed the best—the articles that shared three words with the highest TFDIF yielded a result between 0.5-0.7 (mean 0.6). The authors then applied agglomerative clustering over the tags and obtained a hierarchy of tags very similar to a hand-made taxonomy of tags which can be browsed. This is an interesting and potentially useful result. Automatic ontology generation based on tagged documents may be a promising direction of research avoiding some of the pitfalls in ontology research so far (the need for agreement among experts and the development of standards that are then imposed on others to follow). Yet, there is no guarantee that the automatically generated ontology will be understandable for humans and that humans will agree with it. On the other hand, agreement may not be necessary, since for practical reasons, the ontology is better used by the machine and hidden away from human eyes. The humans can deal with tags, which are user-friendly; the ontology should stay in the background to support machine reasoning and more complex inference and adaptation.

It uses three dimensions (semantic categories): time (horizontal axis), content (posts, vertical axis, upward), and people (comments of readers, vertical axis, downward). In the space defined by these three dimensions, each post is represented as a dot on the horizontal time axis. A line stretching up from each post indicates the length of the post. A line stretching down from each post indicates the average length of the comments made on the article. The downward line ends with a bubble whose size represents the number of comments received. In the upper (content) part of the space, tags indicate the semantics of posts written in the corresponding period of time, while, in the lower (people) part of the space, user names indicate the commentators of the posts during the period of time. In a case study, evaluation of the interactive visualization uses found it easy, intuitive, and efficient to find blog posts of interest in a large archive [ ¹⁹ ]. What makes this approach interesting is that it allows the combination of several principally different ways of searching information (corresponding to different purposes) that are not possible in the current state of the art tag-based systems: by time, content, and social interaction history. Providing a map-like overview of a blog archive allows the user to filter a large amount of information and discover interesting posts. It adds the power of query-based search to the freedom of browsing and the usability of a tag-oriented user interface.

The points accumulated by each student through her participative actions allow students to be classified into different status levels, for example, gold, silver, or bronze. Explicit status categorization is used in many customer loyalty programs, such as the Star Alliance group of airlines that award certain status and related privileges to frequent flyers. The success of this marketing approach (part of a whole range of approaches in customer relationship management) can be explained by social psychology with the social comparison theory [ ¹¹ ] and the theory of discrete emotions (fear). According to the social comparison theory, people strive to achieve higher status not just to gain the privileges (utility) associated with it. Belonging to an exclusive or elite club increases the individual's self-esteem because of downward comparison with people who have not achieved the same status.

It is important that status can be lost as a result of inactivity. Airlines usually award a certain level of status for one year, which is based on the accumulated miles during the previous year. According to the theory of discrete emotions (fear), people are generally more motivated by the fear of losing something they have than acquiring the same thing if they don't already have it. Therefore, people who have achieved a higher status, even one time, will try to avoid losing it.

In the first version of Comtella that was used to support a class of students taking an Ethics and IT course, we defined three status levels: Gold (the top 10 percent of the students based on their participation points), Silver (the next 60 percent of the students), and Bronze (the remaining 30 percent of the students). The status was valid for one week and it was based on the points collected from the participation actions of the student during the previous week. The status was displayed in the interface of the application as a shiny metallic card in the top left corner. By clicking on it, the student saw their level of participation in each of the rewarded activities compared to the top student in this action. A social visualization accessible from the main interface of the application showed all of the students as different sizes of stars in a night sky, which would encourage students to engage in social comparison. The students could choose to view the stars sorted by status, by number of new contributed papers, by number of downloaded papers, and also by login frequency (see Fig. 5 ). The incentive mechanism was introduced in Comtella in the middle of the term. The students used Comtella for 6 weeks without the incentive mechanism and for 4 weeks with the incentive mechanism. We saw a dramatic increase in the overall number of contributions during the first two weeks following the introduction of the mechanism and a decline in the next two weeks, but the contributions nevertheless remained at a level higher than most of the weeks before introducing the mechanism. We also observed a correlation (0.66) between the number of new contributions shared by individual students and their accesses to the social visualization, which displayed the students compared by number of new contributions as a default view [ ³⁷ ]. This shows that the students engaged in social comparison.

In fact, the mechanism was too successful in encouraging participation; it encouraged "gaming," or students submitting low quality papers to achieve a higher status. This resulted in an excessive number of contributions during the second week after introduction of the mechanism, which lead to cognitive overload and withdrawal of some students. We learned several lessons from this experience. First, that in the next version of the mechanism, we need to encourage students to submit papers with high quality. Second, to enable a metric for paper quality, we had to encourage students to rate papers more often. Third, we had to stimulate contributions early in the week. We found that most of the contributions came late in the week when there was little time left for the students to read and rate them [ ⁸ ].

Gaming the system is a phenomenon that happens almost always when there is an incentive mechanism in place. According to Lewitt and Dubner [ ²⁵ ], people are economic creatures who always try to minimize their efforts and maximize their rewards. In education, gaming the system is frequently found; for example, in course work as plagiarism or in exams as cheating. There are more sophisticated forms of gaming the system that aren't easily caught or punished, e.g., finding a critical path toward receiving a degree by selecting the easiest classes, persuading instructors to waive prerequisites for them, and so on. Generally, gaming the system means finding and exploiting loopholes in the rules of the incentive mechanism to gain advantage. This happens when students are under high pressure or when there aren't strong deterrents in place, such as strong penalties and successful policing. In the online world, gaming the system has been found in all online communities that make use of incentive mechanisms, i.e., Slashdot [ ²⁴ ], in multiplayer online games, and even in students interacting with intelligent tutoring systems [ ² ].

In game theory, a good mechanism is one that can be proven to be not gameable. Yet, in practical mechanism design, it is very hard to find such mechanisms apart from very constrained domains, like markets and auctions, since the rules and their possible interactions are very complex. In practice, designers often try to obscure the rules so that it is more expensive for students to find ways to game the system than to put in the required effort. Slashdot, for example, does not publish the algorithms of awarding "karma." We, similarly, did not tell the students how many points they were awarded for different participation actions and they had to explore this themselves. The exploration was made more challenging by displaying students' participation statistics only for the previous week. The limited data meant that the students needed to keep track of all their actions for a full week in order to discover how many points each action earned.

As a result of the lessons learned from the first version, we developed in 2004/2005 a new version of the Comtella incentive mechanism that differed from the first in the following aspects:

These aspects are explained below in more detail. The needs of the community change and evolve in time. It was important that students submit new contributions early in the week, so that their colleagues had time to read and rate them before the topic changed in the week after. Therefore, more points were awarded for new papers shared early in the week than to late papers. As the number of shared papers increased over the week, it became increasingly important to rate the papers so that they could be sorted by quality and thus facilitate students in finding better papers to read. Therefore, the points awarded for rating papers increased with time. We defined a time-dependent reward function which reflected the community needs over time and served as a communuty model.

Students are different. Some students tend to contribute a lot of lower quality articles, while others are more selective about their submissions. Therefore, an individual user model was created to compute the average quality of contributions submitted by each student. The average quality of each contributed paper by the student was computed based on the ratings received from all students. The average quality of the ratings the student has given was computed by comparing his or her ratings to the average rating that the papers have received from other students. These two values comprised the individual model. Based on the individual and community models, individual weights were computed for the actions of each student (the payoff matrix). The total number of points determined the students' status for the new week, which brought different feedback and privileges: a different color scheme in the interface, a higher number of ratings a student could give out, and personal complimentary messages. The status is reflected in the community visualization ( Fig. 6 ). The visualization differed from the earlier one by showing only one view, the same for all students [ ³⁶ ]. The stars representing the individual students looked more like real stars and differed by size (representing the number of papers contributed), color (representing the status of the student), brightness (showing the reputation of the student based on the ratings of his or her contributions), and whether the student was online at the moment.

Students were encouraged to rate the papers submitted by their colleagues through the introduction of a new extrinsic reward (Cpoints) that the students could use like currency. The students were able to utlize the Cpoints to make their own submissions appear at the top of the search results for each week (similar to Google's sponsored links, see Fig. 7 ). The incentive mechanism motivated students to submit twice as many ratings as students who didn't have access to the CPoints mechanism in a controlled experiment [ ⁸ ].

Of course, to maintain the existence of the community, we needed to ensure that there were a sufficient number of postings submitted to the system. Instead of rewarding postings through the system, we decided to reward students for postings through the larger incentive system that existed in each university class in terms of course work and marks. We incorporated the use of Comtella in the required coursework for the Ethics and IT class in 2005/2006. Students were required to contribute one new post/article each week, comment on the posts of two of their colleagues, and respond to a question asked by the instructor. The use of the course work incentive mechanism ensured a sufficient level of activity in the Comtella community. Now, we could apply our novel mechanism targeted at yielding more reads and more ratings.

More reads were needed because the original educational purpose of the Comtella system was to make students read additional, more up-to-date material related to the class. In order to ensure quality control and to enable students to more easily find the good posts, we needed students to rate a high number of postings. We encouraged students to rate postings by designing the interface so that an aesthetically pleasing animation appeared after each act of rating: The post that was rated would change colors through a scale from violet to bright orange and finish with a color either brighter or darker than before (depending on if the rating was positive or negative—see Fig. 8 ) . Thus, the student immediately saw the effect of his or her rating on the list of search results that everyone in the community would have seen as well—the posts that had the highest ratings appeared brighter and with larger font. This emphasized the contribution made to the community and created a feeling of self-actualization.

We turned our attention to the Reciprocation and Fairness Theory [ ¹⁰ ] and the Common Bond Theory [ ³⁴ ] to encourage students to read others students' posts. According to many experiments in behavioral economics [ ¹⁰ ], people tend to reciprocate and strive for fairness in their interactions with others. According to the Common Bond Theory [ ³⁴ ], people may contribute to a community because they want to engage in relationships with members of the community. To engage the students in reciprocal relationships, we hypothesized that providing the students with visual feedback about the type of relationships that they develop by reading each other's posts would stimulate them to balance their relationships, making them more symmetrical and "fair." We designed a new social visualization, which represented the symmetry of relationship between the viewer and all other students (see Fig. 9 ).

The visualization divided the 2D space according to dimensions corresponding to how often the viewer-student read the posting of another student (Y-axis) and how often other students read the postings of the viewer-student (X-axis). Each student in the community was represented as a point in the space. The viewer was always at the (0, 0) postion in the lower left corner. The distance between the viewer and a given student along each axis depends on how "close" the relationship was between the viewer and the student. In the beginning, when none of the students had read any posts yet, the distance between the viewer and each of the other students was at its maximum, so the values of both coordinates were (1, 1), or "double invisible" (the viewer hadn't read anything by the student and vice versa). Therefore, at the beginning, all of the other students were clustered at the upper right corner of the square. Later, students moved down and to the left, and asymmetries arose as the students read each other's posts. Some students emerged as "pop-stars" and moved toward the top left corner since their posts were frequently read by the viewer, but they were not aware of the viewer's posts. Other students became "secret admirers" of the viewer's posts and they moved toward the bottom right corner. As seen in Fig. 9 , most students evolved a symmetrical relationship with the viewer by slowly moving toward the lower-left corner along the diagonal. Our hypothesis was that the visualization would stimulate students who were "pop-stars" to look at other students in their "secret admirer" corner, read their postings, and respond or comment on them, thus balancing their relationships. We confirmed the hypothesis in a one-term classroom experiment with a control and test group. We saw that the test group students engaged in more symmetrical relationships with their colleagues and read more articles. More details about the approach and its evaluation can be found in [ ³¹ ].

Of course, we had to make the common assumption that is made in many adaptive and recommender systems: that viewing posts (the student clicks on the post) is the same as reading posts. The system can only track the number of "views" but it cannot know if the viewed posts were actually "read." In general, the evaluation of all three incentive mechanisms in the classroom experiments using the different versions of Comtella with control and test groups of students showed that each of the mechanisms was very effective in stimulating student behavior that was intended (contributing more papers, ratings, or reading papers). Both the Cpoints and the immediate gratification approaches stimulated twice as many ratings in the test group as in the control group. We showed that an adaptive rewards mechanism could orchestrate the desired pattern of collective behavior: The time-adaptation of the rewards stimulated students to make contributions earlier. We learned that it is important to make the student aware of the rewards for different actions at any given time. More details about the evaluation of these systems are available in [ ⁸ ], [ ³⁶ ], [ ³⁷ ], and [ ⁴³ ].