In case you thought that Dwayne Phillips had exhausted the topic of imaging in these pages, here's another interesting installment.
This article focuses on visualizing depth images via rendering. It adds another visualization technique to those discussed in my previous article [1]. A depth image is similar to an ordinary grayscale image, except that the brightness of each pixel represents a "depth" or some other quantity that can be represented by a scalar. An example of a depth image is a representation of an ocean floor. Depth images are also called range images because the gray level represents the range or distance from the sensor.
Images 1 and 2 are example depth images (from [2] and [3]). Image 1 is an American 25-cent piece. The brighter areas are closer to the viewer than the darker areas. Image 2 is a depth image of shot-blasted metal. These images are difficult to interpret. Most people can visualize the coin because it is a familiar object. The other object, and the other depth images shown later, are not so easy to understand. It is hard to "see" the surfaces of these objects.
The remainder of this article discusses rendering via Lambertian rules [4]. These fundamental rules of reflection are used in most graphic arts today. These rules determine how computers make mathematical models look like real objects in movies, television, and video games. They allow people to see a depth image or a model of an object as something real.
Rendering and Lambertian Rules
Rendering is the process of generating an image of a surface so that it looks like a real, physical object. In this article, rendering is a transformation applied to depth images; in other contexts, the objects being rendered may be represented by different kinds of underlying models, such as lists of polygons or splines.
Figure 1 illustrates the basics of how light reflects off a surface. The surface of the object is the dark line on the bottom. The light comes in from the right as the vector L. N is the normal vector, which by definition is perpendicular to the surface. The vector R represents the reflection of the light off the surface. Objects become visible when light reflects off of them. The viewer is you and me. We see this interaction of light, surface, and reflection from some vantage point, which is at some point in line with the vector marked Viewer. Notice that the angle q, between incoming light L and normal N, equals the angle between N and reflected light R. L, N, R, and V (viewer) are vectors. They have components in the three dimensions x, y, and z. Equation (1) shows these components.
When light reflects off the surface of an object, we see the object. The reflected light consists of three components, known as ambient, diffuse, and specular light. Equation (2) shows that the intensity of light received at a point is the sum of the intensities of ambient, diffuse, and specular components. Ambient light is light that may have originated from a specific source — sunlight through the window and curtains, overhead light in the hallway, etc. — but has been reflected and diffused so much that it can be approximated as having equal intensity in all directions. Diffuse and specular reflections depend on surface properties of the object. A perfect diffuser will spread light uniformly over a surface. Paper, carpet, and cloth are examples of good diffusers. Shining a light on carpet does not yield much of a bright spot. Specular surfaces are shiny, because they reflect light rays without causing much spreading. Polished brass shows relatively bright spots when a light hits it.
Equation (3) is an expanded version of equation (2). The vectors N, L, V, and R of Figure 1 are components of this equation. Isource represents the intensity of light received from a light source pointed directly at the object (such as from a flashlight or lamp). Iamb represents the intensity of ambient light present in the room. Turning off the lamp and closing all the doors and windows causes both the light terms go to zero — the image becomes zero or black. kdiff is a diffusion constant; it affects the contribution of the ambient light term. kspec is a specular constant that affects the contribution of the specular light term. These constants must be between 0.0 and 1.0. The exponent h at the far right affects how shiny or dull the surface appears. h is small for dull surfaces, large for shiny surfaces. The known values in the equation (because they are entered by the user) are Isource, Iamb, kdiff, kspec, and h.
The N· L and V· R terms on the right end of the equation are "dot products." The dot product is calculated by either Equation (4) or (6). (Reference [5] is one of many sources for the vector calculations common in rendering and modeling.) Equation (4) is a quick way to calculate the dot product, if you know the angle between two vectors. In the equation, the angle q represents the angle between vector V1 and V2. The v1 and v2 terms (lowercase v) of equation (4) represent the magnitudes of the vectors. Equation (5) shows how to calculate the magnitude of a vector. Here is an implementation of Equation (5) in C:
magnitude_of(v, answer) float v[], *answer; { double d, n; n = v[0]*v[0] + v[1]*v[1] + v[2]*v[2]; d = sqrt(n); *answer = d; } /* ends magnitude_of */Equation (6) shows how to calculate the dot product, given two vectors having x, y, and z components as defined in Equation (1). An implementation in C is:
dot_product(v1, v2) float v1[], v2[]; { float result = 0.0; result = v1[0]*v2[0] + v1[1]*v2[1] + v1[2]*v2[2]; return(result); } /* ends dot_product */Notice that the dot product yields a scalar, which is a single value, not a vector.
Figure 2 graphically represents the problem of calculating Equation (3) for a single point of the rendered image. The viewer looks straight down on the surface of an object. (This surface is not explicitly specified anywhere; it will be derived from the intensities of the depth image.) Again, vector L represents incoming light from a directional light source; vector R represents its reflection off the surface. This fictional surface is shown at an arbitrary slant, because the surface of the object being rendered (such as a quarter) will likely be at a slant.
Notice what happens if the angle q, between light ray L and surface normal N, is greater than 90 degrees. That means the light is hitting the back side of the surface. The diffuse and specular components of light reflected to the viewer must be zero, because light does not hit the surface. Equations (2) and (3) indicate that in this case the surface will still be visible as long as the ambient light source and diffusion constant are not zero.
Notice also that the Diffuse reflection component of Equation (3) (Isource*kdiff*N· L) is controlled by the dot product of N and L. The position of the viewer does not matter here; the amount of light reflected is determined by the angle between L and the surface, and the diffusion term kdiff, which depends on the surface properties of the object.
The specular reflection component is controlled by the dot product of V and R; that is, both the reflection angle of the light and the position of the viewer affect whether or not shiny spots on the surface are visible to the viewer.
The program shown later will implement Equation (3), and it will make use of all the terms in that equation. The user will provide the ambient and source light intensities (Iamb and Isource), the diffusion and specular constants (kdiff and kspec), and the h term. The light vector L will also come from the user. It will consist of x, y, and z components, thus allowing the user to specify the precise direction from which the light comes. The viewer vector V is assumed to be straight up.
Finding the Surface
The user specifies six of the values used in Equation (3), which accounts for an infinite variety of possible results. However, still missing are the vectors N and R. The vector N comes from the surface plane, so we need to first find the surface. In this section I explain how this is done.
Figure 3 shows the physical situation modeled by a depth image. The pixel value in a depth image represents a height. The bars in Figure 3 are proportional to the those heights.
Figure 4 represents a depth image pixel and two of its neighbors. The pixel at (x,y) has a certain z coordinate proportional to height, as do its neighbors at (x+1,y) and (x,y+1). The vectors V1 and V2 go from the pixel at (x,y) to its neighbors. V1 and V2 are easy to calculate, given that the image values represent z coordinates. These vectors can be seen to form two sides of a triangle, and this triangle can be seen to lie within a plane. This plane is the surface of Figure 2.
The vector N normal to this plane is produced by the cross product [5] of V1 and V2. Equation (7) shows one way to calculate the cross product, using the three elements of the vectors. (The quantity shown in the middle of the equation represents a matrix determinant.) An implementation in C is shown as follows:
cross_product(v1, v2, result) float v1[], v2[], result[]; { result[0] = v1[1]*v2[2] - v2[1]*v1[2]; result[1] = v1[2]*v2[0] - v2[2]*v1[0]; result[2] = v1[0]*v2[1] - v2[0]*v1[1]; } /* ends cross_product */The program shown later will use this method to find the normal vector N. Notice that the cross product yields a vector, not a scalar.
Now how do we find the vector R? We won't. At first glance, it seems that the vector R is needed to calculate the dot product of V and R. However, this calculation can also be done via Equation (4), given the angle between V and R. That angle can be inferred as follows. First, given two known vectors, such as L and N, it is possible to find the angle between them using Equation (8). The following snippet shows how to do this in C:
angle_between(v1, v2, angle) float *angle, v1[], v2[]; { double d, d1, d2, dt, n, result = 0.0, t; char response[80]; n = v1[0]*v2[0] + v1[1]*v2[1] + v1[2]*v2[2]; dt = v1[0]*v1[0] + v1[1]*v1[1] + v1[2]*v1[2]; d1 = sqrt(dt); dt = v2[0]*v2[0] + v2[1]*v2[1] + v2[2]*v2[2]; d2 = sqrt(dt); d = d1*d2; t = n/d; if(t <= -1.0) t = -0.999; if(t > 1.0) t = 1.0; result = acos(t); *angle = result; } /* ends angle_between */This routine can also be used to calculate the angle between L and V, and between N and V. Some addition and subtraction then provides a good approximation of the angle between V and R.
We now have all the necessary terms to implement Equation (3) and render depth images.
Implementation and Examples
Listing 1 shows parts of the main calling routine. This is a command-line program. The user enters the file names, the diffusion and specular terms, the exponent h, and the light vector L. The main routine calls several functions that create and free arrays and read and write image data. (The full source code is available on the CUJ ftp site. See p. 3 for downloading instructions.)
The main routine also calls the lambert subroutine (Listing 2) that transforms, or "renders" the input depth image into an output image. The lambert subroutine implements Equation (3). This subroutine should be easy to understand, since it follows the preceding discussion closely.
It is important to understand the direction of the axes in this code. The x direction goes from left to right as usual. The y direction goes from top to bottom. This is normal in image processing, but is upside-down of how people usually think of images. This opposite y direction causes the positive z axis to be going away from the viewer — opposite from what is expected again.
The lambert routine first sets the viewer vector V to straight up. The loops over rows and cols run through the input depth image to render it into the output image. The first section of code calculates the V1 and V2 vectors of Figure 4 and uses the cross product to find the normal vector N. The next section finds the angles between L and N, N and V, and L and V. Addition and subtraction of angles produces an approximation of the angle between V and R. V and R produce the specular term while N and L produce the diffuse term. Simple addition and multiplication fills out Equation (3).
Images 3, 4, and 5 show example renderings from depth images. As stated earlier, the user sets six of the variables in Equation (3). Space is not available to show even a small percentage of the possible outputs. Image 3 shows four renderings of the quarter depth image of Image 1. The direction of the light source is different for each rendering. Image 4 shows two different renderings of the shot-blasted metal depth image from Image 2. Image 5 shows a depth image of skin on the left and its rendering on the right.
Conclusion
This article has presented a simple implementation of code that renders objects via Lambertian rules. The program was applied to depth images to produce something that is easier to see. This simple implementation just scratches the surface of rendering and object modeling. It does, however, show the basics of how Hollywood and the scientific communities turn depth images and 3-D models into seemingly real objects on the screen. I invite the readers to explore this topic further on their own. As usual, experiment, learn, and have fun.
Notes and References
[1] Dwayne Phillips. "Visualizing Depth Images," C/C++ Users Journal, December 1999, pp. 18-32.
[2] John C. Russ. The Image Processing Handbook, Third Edition (CRC Press, 1999).
[3] The Image Processing Toolkit, Version 2.5, Reindeer Games Inc., http://members.aol.com/ImagProcTK.
[4] Johann Heinrich Lambert (1728-1777) was a Swiss-German mathematician, astronomer, and physicist. Like many famous and accomplished scientist of his era, Mr. Lambert made fundamental advances in these fields. His work in optics produced the rules that are implicit in Figures 1 and 2; these rules are described more explicitly by the equations that accompany the figures.
[5] William H. Beyer. CRC Standard Mathematical Tables, 26th Edition (CRC Press, 1983).
Dwayne Phillips has worked as a software and systems engineer for the US Government since 1980. He has a Ph.D. in Electrical Engineering from Louisiana State University. In addition to writing about image processing for C/C++ Users Journal, he has written The Software Project Manager's Handbook, Principles that Work at Work, published by the IEEE Computer Society.