Sparse coding of sensory inputs
Introduction
At any given moment, our senses are receiving vast amounts of information about the environment in the form of light intensities, changes in sound pressure, deformations of the skin, stimulation of taste and olfactory receptors and more. How the brain makes sense of this flood of time-varying information and forms useful internal representations for mediating behavior remains one of the outstanding mysteries in neuroscience. In recent years, a combination of experimental, computational, and theoretical studies have pointed to the existence of a common underlying principle involved in sensory information processing, namely that information is represented by a relatively small number of simultaneously active neurons out of a large population, commonly referred to as ‘sparse coding’.
In this review we discuss the theory of sparse coding, methods for measuring sparsity, and the evidence to date that sparseness constitutes a general principle of sensory coding in the nervous system. We focus here primarily on neural representations in the cortex of mammals, or relatively high levels of processing in other species, but it should be noted that there is substantial evidence for sparse coding occurring at earlier stages of processing across a variety of organisms 1., 2..
Section snippets
Theory of sparse coding
The principle of sparse coding has been advanced and elaborated on by several different authors, for different reasons. Early work on associative memory models, for example, showed that sparse representations are most effective for storing patterns, as they maximize memory capacity because of the fact that there are fewer collisions (less cross-talk) between patterns [3]. Later work has similarly showed that sparse representations would be advantageous for learning associations in neural
How to measure sparseness?
To assess whether or not neurons are utilizing a sparse code, there must be a method for measuring sparseness. Many of the coding models described above employ analog-valued units that could take on both positive and negative values, with responses symmetrically distributed around zero (Figure 3). A standard measure of sparseness for such artificial units is the kurtosis, which measures the 4th moment relative to the variance squared:
where r is the response of the neuron,
Experimental evidence
Experimental evidence for sparse coding has been found in several different sensory modalities in a variety of animals. In the visual system of primates, Vinje and Gallant 39., 42.•• have demonstrated that neurons in V1 produce sparse punctate responses when stimulated with image sequences resembling those that occur during natural vision (Figure 4). Interestingly, when the same neurons are stimulated only within their classical receptive fields (the region of space within which a stimulus
Conclusions
Although the principle of sparse coding has been discussed and elaborated for nearly three decades now, serious empirical investigation of its use in the nervous system has begun only recently. Investigating whether sparse coding is employed in a certain region, however, will require using ecologically valid stimuli (i.e. natural scenes). The studies reviewed here suggest that sparse coding provides an efficient means of representing data found in the natural world. Moreover, it provides a
References and recommended reading
Papers of particular interest, published within the annual period of review, have been highlighted as:
• of special interest
•• of outstanding interest
Acknowledgements
We thank P Kanerva, J Johnson, K O’Connor, and D Warland for helpful comments on the manuscript, and M Bethge and K Koepsell for providing Figure 2.
References (54)
How do visual neurons respond in the real world?
Curr Opin Neurobiol
(2001)Origins of scaling in natural images
Vision Res
(1997)Vision and the statistics of the visual environment
Curr Opin Neurobiol
(2003)- et al.
Independent component filters of natural images compared with simple cells in primary visual cortex
Proc R Soc Lond B Biol Sci
(1998) - et al.
An energy budget for signaling in the grey matter of the brain
J Cereb Blood Flow Metab
(2001) - et al.
Coding of natural scenes in primary visual cortex
Neuron
(2003) - et al.
Dynamic representation of whisker deflection by synaptic potentials in spiny stellate and pyramidal cells in the barrels and septa of layer 4 rat somatosensory cortex
J Physiol
(2002) - et al.
The structure and precision of retinal spike trains
Proc Natl Acad Sci USA
(1997) - et al.
Non-holographic associative memory
Nature
(1969) On associative memory
Biol Cybern
(1980)