Shapes from shading

by Wojciech Chojnacki

The SFS problem, i.e., the problem of reconstructing shape from shading, can be stated in the form of the following question: How can the shape of a smooth surface be reconstructed from its black-and-white photograph? The varying intensity of the picture is called shading.

The problem dates from the 1950s. Initially it consisted in determining the topography of the Moon from its pictures. In the mid-70s it attracted the attention of computer vision experts, who intended to provide robots with an ability of reconstructing a 3D event scene from 2D pictures, a feature necessary for the robot's free displacement around the scene, as the observed fragment of space is termed. A variant of the SFS problem appeared again in the mid-80s as the problem of radar picture interpretation.

The first step in the consideration of the problem is to establish the relation between the shape of the surface and the shading. The relation defines in turn the picture irradiancy equation. Irradiancy is the technical term for the intensity of the picture, while the radiance of the scene, a related term, is the intensity of light emitted by the scene. Picture irradiancy is proportional to the scene radiance, with a coefficient not greater than 1.

To solve the SFS problem effectively, some simplifying assumptions must be added. The basic assumption requires that the surface be smooth. It is also assumed that the surface does not shade itself and that the contribution of repeated light reflections from the surface to the picture is negligible. Moreover, the light source and the device where the picture is generated are usually taken to be at a large distance from the surface. This implies that we can assume the surface image to be produced by a beam of parallel rays hitting the surface perpendicularly. Even more, in that case the shading due to a small patch of the surface depends only on the orientation of the patch in space rather than on the absolute position of the surface in space. This assumption is satisfied in practice when, for instance, the light source is Sun, the device used to generate the picture is a photo camera fixed on board of a plane flying high over the ground and the surface observed is a fragment of the Earth's surface.

Let's fix a Cartesian system of co-ordinates in space so that the axes x and y be in the picture plane. Let u(x, y) be the height of the surface above the picture plane at the point (x, y). Thus every surface point is described by a triple (x, y, u(x, y)) and the image of a point so described is (x, y). With the simplifying assumptions mentioned above the picture irradiancy equation is of the following form:


where E(x, y) is the picture irradiancy at (x, y), R is the so-called reflectivity function, l is the unit vector for the direction of the light source and the last argument is the internal surface reflectivity, i.e., the albedo at (x, y, u(x, y)). The partial derivatives involved in the equation uniquely determine the vector normal to the surface at (x, y, u(x, y)) given by the formula




This vector defines for the plane tangent to the surface at (x, y, u(x, y)) its orientation in space. The tangent plane, when reduced to a small environment of the point (x, y, u(x, y)), is quite good an approximation of the surface patch included in the environment. In consequence we may assume that the partial derivatives determine the orientation of a small surface patch around (x, y, u(x, y)). Given R, l and , the picture irradiancy equation defines all the admissible orientations of the patch that lead to a picture of brightness E(x, y) at (x, y).

As a condition on u(x, y), the picture irradiancy equation is a first-order partial differential equation with two independent variables. As it turns out, with R, l and fixed and for an arbitrary (non-negative) irradiancy E, such an equation usually has infinitely many local solutions defined on a fragment of the image domain. There may also exist many global solutions, defined on the entire domain of the image, and there may be no global solution at all as well. To get a unique global solution, some boundary conditions must be imposed. The determination of global solutions that satisfy the boundary conditions is an interesting mathematical problem itself, still intensely studied.