Image segmentation by object color: a unifying framework and connection to color constancy.

A unifying framework is presented for algorithms that use the bands of a multispectral image to segment the image at material (i.e., reflectance) boundaries while ignoring spatial inhomogeneities incurred by accidents of lighting and viewing geometry. The framework assumes that the visual stimulus (image field) from a uniformly colored object is the sum of a small number of terms, each term being the product of a spatial and a spectral part. Based on this assumption, several quantities depending on the reflected light can be computed that are spatially invariant within object boundaries. For an image field either from two light sources on a matte surface or from a single light source on a dielectric surface with highlights, the invariants are the components of the unit normal to the plane in color space spanned by the pixels from the object. In some limited cases the normal to the plane can be used to estimate spectral-reflectance parameters of the object. However, in general the connection of color-constancy theories with image segmentation by object color is a difficult problem. The concomitant constraints on segmentation and color-constancy algorithms are discussed in light of this fact.