A system for continuous capture and retrieval of multimedia data from a home-like ubiquitous environment is presented. A large number of cameras and microphones are used for video and audio acquisition. Pressure based sensors mounted on the house floor are used to capture footsteps as context data. Both context and content are analyzed to create indices for efficient retrieval and summarization of the video and audio data. The system can be queried based on date and time, location, persons, sounds and basic activity in terms of footsteps. Key frame extraction adaptive to the activity of a person can be used to summarize the video. The system was evaluated by means of a real-life experiment where a family lived in the house and later used the system to retrieve experiences that they had during their stay. The results of this experiment are reported and discussed. The paper concludes with suggestions for possible future directions.