Issue No.06 - November/December (2003 vol.18)
Published by the IEEE Computer Society
<p>Human memory is more than just file retrieval. How can we duplicate human-like memory processes on computers?</p>
In my last column, I mentioned work underway to frame a particular computer science Grand Challenge. This effort, called Memories for Life, originated in a workshop in Edinburgh last year. The idea is to develop a research
agenda around the challenge of recording an individual's lifetime experience. What science and technology do we need to build systems that store, index, and manage such content?
An Intriguing Topic
This challenge has interested readers and the media alike. I've been thinking why this might be and reflecting on the research agenda that might arise. The interest might originate in the appeal of Grand Challenges themselves. The idea is that they'll lead to a revolutionary shift in thinking or practice and generate enthusiastic support from scientific communities. A good Grand Challenge should confer long-term benefits to science, industry, and society. It should have international scope and invariably will generate interdisciplinary and collaborative research. However, although Memories for Life might meet these criteria, I suspect that a much simpler feature is generating the email—namely, that this Grand Challenge appeals to the imagination.
Everyone is interested in memory. Writers, artists, scientists, and philosophers have all labored mightily on the topic. Our essence resides in our memories. To see this, you only have to look at individuals who have undergone neurological trauma and cannot form long-term memories. Many have perfectly good memories up until the point of the injury or disease. Thereafter, they live in the present, perfectly able to read a news article and a few minutes later come back to it and read it again, totally unaware that they've seen it before. This condition also is a symptom of some neurodegenerative conditions in the elderly. Many of them can recall memories from their youth, but what happened earlier in the day completely escapes them.
Memories are both personal and social: some are unique to the individual, others are held in a collective recollection. Think of your first day at school; think of what you were doing the moment you heard about the Twin Towers. Ask yourself, what is your first memory? Why do we remember so little from before age eight—a condition psychologists call "infantile amnesia." My own daughter, who is nine, would engage me as a three-year-old in detailed recollections of what she had done in her nursery; she could remember in clear detail events that had happened weeks and months before. Most of these are of the vaguest recollection for her now; the clarity and precision of those early recollections are gone. We must have all found ourselves wondering where our memories have gone. Are they lost, temporarily mislaid, diminished, or overlaid? How are they structured? Do we have one or many memory systems? How can a smell evoke a sense of place, or a song a moment in time?
The Science of Memory
There's a lot of science we can draw on here. For example, psychologists studying memory have identified multiple systems. We have working memories that support a variety of information processing. To get to the end of a sentence and know what it's about, it's important to have a short-term memory to store the structure and sense of the sentence's beginning. As you read deeper into an article, other parts of the memory system kick in while the working memory is busy with the sentence you're reading.
Long-term memory appears to have distinct functional capabilities. It appears to involve four types of memory systems:
• Episodic: for example, remembering when you last rode a bicycle
• Semantic: remembering what a bicycle is
• Procedural: remembering how to ride a bicycle
• Recognition: how to recognize an instance of a bicycle
Different brain structures are at work when these components operate, and damage can knock out one component, leaving others intact.
Psychologists have also noted that episodic memories are organized temporally (with respect to the person remembering) and must be remembered consciously. They're also susceptible to being forgotten and are highly context dependent. So, for example, being in the same physical location as the original event can facilitate recollection. Semantic memory seems to be organized with respect to general knowledge and isn't organized temporally. The objects of semantic memory are simply known rather than consciously recollected. They also seem to be relatively permanent—the very last memories to fail in many age-related dementias—and are context independent.
The study of human memory has revealed much else. In a very accessible book, The Seven Sins of Memory (Houghton Mifflin, 2001), Daniel Schacter describes a range of phenomena that clearly put the lie to any notion of memory as analogous to file or content retrieval in a computer. We see transience, which is the weakening or loss of memory over time. Numerous examples exist of blocking, where a person fails to retrieve something in his or her memory. We frequently misattribute with respect to our memories, forgetting when, where, who, or what. We demonstrate suggestibility, where memories are reconstructed or reinterpreted, changing the original recollection considerably. There are numerous examples of bias, where a memory is misrepresented. Often we demonstrate absent-mindedness—a breakdown between attention and memory. Finally, Schacter highlights persistence, which is the inability to suppress or remove memory—a blessing or a curse, depending on the context.
All of which demonstrates, if demonstration were needed, why memory is not simply file retrieval. Other neuroscientists are unraveling both the locus and biomolecular basis of memory. Researchers have shown that the hippocampus is important in the representation of memories with a spatiotemporal component. Place cells in the hippocampus of rats fire most strongly when rats pass through familiar parts of an environment. Imaging techniques such as functional magnetic resonance imaging are providing increasingly detailed views of the brain structures used in a variety of memory tasks. Studies have implicated particular molecules and receptors that appear to modify the efficacy of the connections between nerve cells. This modification appears to be an important means by which memories are formed.
WHAT ABOUT COMPUTERS?
But does any of this inform how we might think about memory in our computational devices? Clearly, the whole neural network and connectionist paradigm has been inspired by the idea that concepts and events can be represented in a distributed fashion in computers. The representations are essentially weight spaces that can be subject to decay, overwriting, attenuation, and strengthening in ways strongly reminiscent of elements of biological memory. Associative retrieval is also a feature of connectionist memory systems. However, it's perhaps less clear how to represent phenomena such as bias and reinterpretation. And if we're to produce memory systems that complement human memory, we need to be aware that these processes are every bit as important as veridical recall. In fact, you could argue that the sins of memory are actually virtues—as humans, we edit and embellish our memories to fit our perception of ourselves and others. Forgetting might serve a range of important purposes. Traumatic events might best be buried and forgotten. The daily trivia of existence wouldn't be welcome if they re-presented themselves at every waking moment.
You could also argue that we can't and shouldn't seek to emulate all facets of human memory. Instead, we should aim to provide memory augmentation systems. Certainly, this has been a more successful approach in the past—rather than building knowledge-based systems that emulate expert behavior, we build decision support systems that augment problem solving. However, even a memory prosthesis gives rise to substantial challenges. Given that the hardware will let us record continuous amounts of experience, how do we represent and model the content? What sort of indexing and metadata can we use to allow contextual retrieval? What are the ontologies for a lifetime? How will these need to evolve and change as an individual's experience, interests, goals, and attitudes develop? How do we summarize experiences so as to provide a gist or précis? There's a requirement to encode content across different modalities—sight, sound, taste, smell, and touch. What sorts of representations will these be?
No lack of challenges here, and, as I've argued in previous editorials, hardware developments will create the requirement for some sort of response to them. Storage systems such as Fujitsu's .8-inch, 80-Gbyte hard drives and the ever-decreasing size and power requirements of cameras and their transmitters will mean that individuals will increasingly be able and willing to commit their lives to bits. Whether they should is another matter. And if a person decides not to recode his or her own experience, what about those who would do it for that person covertly? There are Grand Challenges here of both a technical and social nature.