Advances in computer technology have made it feasible to provide engineering design students with computer support for learning about and performing various activities in the engineering design process. Important issues for consideration are how to design the computer tools and how to integrate these tools into existing courses. We have focused our efforts on learning how to design collaboration software and also on understanding how to integrate it into an engineering design course. We conducted two quarters of a formative evaluation, during which our software (CaMILE, an Internet based collaboration tool) was used by third year mechanical engineering design students. These students were introduced to and received support for the software in the context of a special voluntary laboratory course focusing more generally on technological support for their efforts in the design course. Our reflections on this experience have led to the identification of the three dilemmas described in the paper: evaluation in or out of the classroom; more with less or more with more; and compatibility of learning about and learning through collaboration. While the three dilemmas were identified in the context of our specific efforts, findings from other educational technology projects as well as other domains support their generality. The paper provides the context for understanding the dilemmas, a description of each, and the impact of each on our current endeavors.