Although highly parallel distributed memory computers exist for several years, the operating systems used on them did not fit the requirements very well. Most of them are designed for sequential, shared memory parallel, or distributed computers. Examples are Unix on the IBM SP/2 and Mach on the Intel Paragon. This results in poor scalability caused by inefficient communication primitives designed for wide area networks or by waste of resources due to huge kernels (e.g. 8 MB per node are reported for Mach on the Paragon), which is harmful especially in highly parallel systems with hundreds or thousands of nodes. With Cosy (Concurrent Operating System) we have shown that a well structured and carefully designed sys tem can be small (70 Kb for the kernel, 372 total memory usage per node), efficient (33ms for communication), and scalable (application run efficient on up to 1024 proces sors).