With the proliferation of multimedia data, it has become necessary to secure this content from illegal use, efficiently detect and reconstruct illegal activities from it, and use it as a source of intelligence. Serious challenges arise from the sheer data volume, however. The multimedia research community has developed many exciting solutions for dealing with video footage, images, audio, and other multimedia content over recent years, including knowledge extraction, automatic categorization, and indexing. Although this work forms an excellent foundation for protecting and analyzing multimedia content, challenges remain in the complexity of the targeted material, the lack of structure and metadata, and other application-specific constraints. This special issue provides an overview of current research following this mission. The articles originally appeared at the ACM Multimedia 2010 Workshop on Multimedia in Forensics, Security, and Intelligence (MiFor). The six high-quality contributions cover various approaches in the field, ranging from the visual recognition of faces and tattoos to the discovery of near duplicates and content tampering.
The last few years have witnessed a multimedia explosion of digital image, video, and audio content. Due to the rapidly increasing spread of mobile devices, accompanied by high-bandwidth Internet and cheaper storage, it is now easy for end users to generate large amounts of multimedia content, store it, and share it via the Internet. Meanwhile, more and more surveillance cameras monitor our public spaces. As a result, digital multimedia has become an integral part of our everyday lives.
With the proliferation of multimedia data, it has become necessary to secure this content from illegal use, efficiently detect and reconstruct illegal activities from it, and use it as a source of intelligence. Serious challenges arise from the sheer data volume, however—tasks that were previously subjected to manual inspection are now far beyond the capacities of forensic experts. Tools are needed to support the protection, management, processing, interpretation, and visualization of multimedia data during the various steps of the investigative process.
The multimedia research community has developed many exciting solutions for dealing with video footage, images, audio, and other multimedia content over recent years, including knowledge extraction, automatic categorization, and indexing. Although this work forms an excellent foundation for protecting and analyzing multimedia content, challenges remain in the complexity of the targeted material, the lack of structure and metadata, and other application-specific constraints. The time is ripe to tailor, adapt, and extend multimedia analysis for forensics, security, and intelligence.
This special issue provides an overview of current research following this mission. The articles originally appeared at the ACM Multimedia 2010 Workshop on Multimedia in Forensics, Security, and Intelligence (MiFor). Out of the almost 40 submissions, we selected six high-quality contributions that cover various approaches in the field, ranging from the visual recognition of faces and tattoos to the discovery of near duplicates and content tampering. The content modalities covered include still images, video streams, and audio data.
The first three articles in this special issue deal with identifying people in a forensic context using different information sources. The two subsequent articles consider the detection of illegal use of multimedia material. Preventing such illegal use is the topic of the final article.
In "Face Matching and Retrieval in Forensics Applications," Anil K. Jain, Brendan Klare, and Unsang Park present an interesting overview of the state of art in the field of face matching and retrieval for forensics application. In particular, their work distinguishes the forensic face-recognition paradigm from typical portrait (2D frontal image) face-recognition scenarios and summarizes recent progress in the field.
The second article, "Image Retrieval in Forensics: Tattoo Image Database Application," by Jung-Eun Lee, Wei Tong, Rong Jin, and Anil K. Jain, addresses the identification of victims and suspects from another angle. Instead of facial clues, their approach exploits body tattoos. Given a tattoo close-up, their system retrieves similar tattoos from a large-scale image database. The approach relies on a matching of local regions of interest and achieves robustness by implementing improvements to standard matching schemes and constraining the search using additional metadata.
In "Profiling Online Auction Sellers Using Image Editing Styles," Lin Yang, Wei-Bang Chen, Chengcui Zhang, John K. Johnstone, Song Gao, and Gary Warner apply image-analysis techniques to product illustrations to detect fraudulent acts on online auction selling platforms. Their key hypothesis is that consistent image-editing styles can serve as a basis for resolving the identity of online sellers, such that automatic style modeling could help detect account takeovers or link multiple accounts. Relying on the identification of repetitive visual elements and local feature matching, their approach is shown to outperform previous methods in the area.
The "Current Developments and Future Trends in Audio Authentication" article by Swati Gupta, Seongho Cho, and C.-C. Jay Kuo addresses the analysis of audio content and its illegal manipulation to falsify evidence in court or hide copyright violations. The authors provide an overview of recent work on audio authentication that covers different steps in the processing pipeline. The article starts with methods for discovering specific recording devices and environments via a spectral analysis, continues with approaches exploiting the electrical frequency of power networks that recording devices are connected to, and concludes with models for compressed-domain audio analysis to detect tampering such as multiple transcoding.
Another interesting issue is addressed in "Efficient Image Copy Detection Using Multiscale Fingerprints" by Hefei Ling, Hongrui Cheng, Qingzhen Ma, Fuhao Zou, and Weiqi Yan. The authors propose an image copy detection method that accounts for a multiresolution signature based on a smart collection of interest point descriptors. The overall method is efficient while still achieving a high degree of discriminability.
While the first five contributions exploit multimedia content for forensic investigations, Liyun Wang, Hefei Ling, Fuhao Zou, and Zhengding Lu's article "Real-Time Compressed-Domain Video Watermarking Resistance to Geometric Distortions" addresses its protection from illegal use. The authors propose a watermarking scheme for video data that operates directly on the compressed video stream and is thus highly efficient. Robustness to geometric distortions in the video, which can occur when an attacker attempts to corrupt a watermark, is achieved by using the histograms of low-frequency wavelet subbands as a channel.
The articles in this issue provide a good overview of the various issues that play a role in the field of multimedia in forensics, security, and intelligence. They also provide a basis for further research in this exciting area. A number of research challenges are still open for future work.
Although techniques for identifying and protecting illegal material are rapidly improving, criminals are also expanding their scope and employing more advanced techniques. Attackers only have to develop one method to circumvent state-of-the-art identification and protection techniques, but research should consider all possible attacks and possibilities. Hence, advancing techniques along the established lines is of great importance.
Even in the multimedia field, we still see many methods that only take one of the information channels into account. The articles in this special issue form no exception. Especially in multimedia forensics, intelligence, and security, where information sources are so diverse, a major step forward will come from employing all possible information channels in a synergetic manner.
Lastly, much work is still necessary in multimedia analytics. Until recently, most investigations have been done manually. The state-of-the-art techniques described in this issue and elsewhere support the analyses using automatic methods. For truly effective investigations, these two fields must come together. The best way to do this is by developing advanced browsing, visualization, and data mining techniques that support every step of the investigation process.
is an associate professor in the Department of Mathematics and Computer Science at the University of Catania, Italy. He is also the director (and cofounder) of the International Computer Vision Summer School (ICVSS) in Sicily, Italy. His research interests include image enhancement and processing, image coding, camera imaging technology, and multimedia forensics. Battiato has a PhD in computer science and applied mathematics from the University of Naples. Contact him at firstname.lastname@example.org.
is an assistant professor in the School of Computer Engineering at Nanyang Technological University, Singapore. His research interests include multimedia and software security and surveillance video processing. Emmanuel has a PhD in computer science from the National University of Singapore. Contact him at ASEmmanuel@ntu.edu.sg.
is a senior researcher with the German Research Center for Artificial Intelligence (DFKI) in Kaiserslautern. His research interests include multimedia analysis, machine learning, and multimedia forensics, where he develops image and video analysis technology that support child abuse investigations. Ulges has a PhD in computer science from the University of Kaiserslautern. Contact him at email@example.com.
is an associate professor at the University of Amsterdam. His research interests include multimedia analytics, bringing together semantic video indexing, interactive retrieval, and advanced browsing and visualization. He has directed several national and international projects, including the EU project I-DASH: the Investigator's Dashboard focusing on fighting child abuse in conjunction with several police forces. Worring has a PhD in computer science from the University of Amsterdam. Contact him at firstname.lastname@example.org.