The recent convergence of image processing, computer vision, and computer graphics has created an emerging field known as image-based rendering. Now widely used in applications ranging from movie special effects (Remember "The Matrix"?) to building virtual environments, image-based rendering has become an essential tool for creating visually exciting content, and yet it is still unfamiliar to many traditional image processing researchers. In this tutorial we strive to serve the ICIP audience by presenting the fundamentals of image-based rendering and summarizing recent developments in the field. The surprising link between image-based rendering and a research field that is well-known to the image processing community, multidimensional multirate signal processing, will be revealed.

With image-based rendering, real-world scenes can be captured and rendered directly using image processing techniques, eliminating the need for computationally intensive 3D model construction. Likewise, virtual environments can be rendered directly from pre-synthesized images, bypassing processes such as ray tracing or texture mapping, hence with much improved rendering efficiency. Furthermore, image-based rendering and model-based rendering can be integrated seamlessly to create a realistic scene such that human eyes can not tell the difference between the two parts of the scene.

Interesting problems in image-based rendering include capturing, representation, and sampling of the visual data, either images or video, needed to render realistically-looking scenes. Also critical to image-based rendering, especially for dynamic environments, are compression and streaming of the large amount of data, both for storage and for transmission over the Internet.

In this tutorial we will introduce how to acquire, represent, and render scenes from digitized images or video. Toward this end, several image-based rendering methods will be presented, with an emphasis on how to use these techniques to build practical systems. We will provide demonstrations of a number of image-based rendering techniques. The audience will have the opportunity to acquire images of various scenes, and use image analysis and synthesis tools provided at the tutorial to render and view these scenes interactively.

The outline of the tutorial is as follows:

I. Fundamentals

• What is image-based rendering?
• Camera models and capturing image-based rendering data
• View interpolation, image mosaics and compositing
• Images warping, morphing, resampling, and texture mapping

II. Image-Based Rendering Methods

• Sampling and representation for image-based rendering
• Parameterization of image-based rendering data
• 7D and 5D plenoptic function
• Light field and lumigraph
• Concentric mosaics
• Panorama and QuickTime VR

III. Application Issues

• Compression for image-based rendering
• Selective coding for image-based rendering over Internet
• Streaming of image-based rendering data
• Compositing real and synthetic scenes
• Virtual teleconferencing
• Special effects in movie production: "The Matrix"

IV. Demonstrations and Experiments

Since October 1997, Tsuhan Chen has been with the Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, Pennsylvania, where he is a Professor. He directs the Advanced Multimedia Processing Laboratory. His research interests include multimedia signal processing and communication, audio-visual interaction, biometrics, processing of 2D/3D graphics, bioinformatics, and building collaborative virtual environments. From August 1993 to October 1997, he worked in the Visual Communications Research Department, AT&T Bell Laboratories, Holmdel, New Jersey, and later at AT&T Labs-Research, Red Bank, New Jersey, as a senior technical staff member and then a principle technical staff member.

Tsuhan has recently been appointed as the Editor-in-Chief for IEEE Transactions on Multimedia for 2002-2004. Before that, he also served in the Editorial Board of IEEE Signal Processing Magazine and as Associate Editor for IEEE Trans. on Circuits and Systems for Video Technology, IEEE Trans. on Image Processing, IEEE Trans. on Signal Processing, and IEEE Trans. on Multimedia. He co-edited a book titled Advances in Multimedia: Systems, Standards, and Networks.

Tsuhan received the B.S. degree in electrical engineering from the National Taiwan University in 1987, and the M.S. and Ph.D. degrees in electrical engineering from the California Institute of Technology, Pasadena, California, in 1990 and 1993, respectively. He received the Charles Wilts Prize for outstanding independent research in Electrical Engineering leading to a Ph.D. degree at the California Institute of Technology. His dissertation was on multidimensional multirate signal processing. He has published many technical papers and holds eleven U.S. patents. He is a recipient of the National Science Foundation CAREER Award.

Tsuhan helped create the Technical Committee on Multimedia Signal Processing, as the founding chair, and the Multimedia Signal Processing Workshop, both in the IEEE Signal Processing Society. His endeavor later evolved into the founding of the IEEE Transactions on Multimedia and the IEEE International Conference on Multimedia and Expo, both joining the efforts of multiple IEEE societies.