CSDL Home H HICSS 2005 Proceedings of the 38th Annual Hawaii International Conference on System Sciences
Big Island, HI, USA
Jan. 6, 2005 to Jan. 6, 2005
Michelle Potts Steves , National Institute of Standards and Technology
Jean Scholtz , National Institute of Standards and Technology
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/HICSS.2005.15
Collaboration technologies are seeing widespread deployment even though it is difficult to assess the effectiveness of these systems. This paper presents an evaluation method that addresses this issue and the use of the method in the field. The method uses a framework to structure evaluations by mapping system goals to evaluation objectives, metrics, and measures. The upper-most levels of the framework are conceptual in nature, while the bottom level is implementation-specific, i.e., evaluation-specific. Capitalizing on this top-down approach, an evaluation template specifying the conceptual elements can be constructed for a series of evaluations. Then implementation-specific measures are specified for individual experiments. This structure makes use of the framework ideal for comparison of the effectiveness of collaboration tools in a particular environment. We present our findings from use of the method in the field to assess the performance of a particular collaboration technology deployment and its impact on the work process.
Michelle Potts Steves, Jean Scholtz, "A Framework for Evaluating Collaborative Systems in the Real World", HICSS, 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences, Proceedings of the 38th Annual Hawaii International Conference on System Sciences 2005, pp. 29c, doi:10.1109/HICSS.2005.15