Sparse coding of sensory inputs

https://doi.org/10.1016/j.conb.2004.07.007Get rights and content

Several theoretical, computational, and experimental studies suggest that neurons encode sensory information using a small number of active neurons at any given point in time. This strategy, referred to as ‘sparse coding’, could possibly confer several advantages. First, it allows for increased storage capacity in associative memories; second, it makes the structure in natural signals explicit; third, it represents complex data in a way that is easier to read out at subsequent levels of processing; and fourth, it saves energy. Recent physiological recordings from sensory neurons have indicated that sparse coding could be a ubiquitous strategy employed in several different modalities across different organisms.

Introduction

At any given moment, our senses are receiving vast amounts of information about the environment in the form of light intensities, changes in sound pressure, deformations of the skin, stimulation of taste and olfactory receptors and more. How the brain makes sense of this flood of time-varying information and forms useful internal representations for mediating behavior remains one of the outstanding mysteries in neuroscience. In recent years, a combination of experimental, computational, and theoretical studies have pointed to the existence of a common underlying principle involved in sensory information processing, namely that information is represented by a relatively small number of simultaneously active neurons out of a large population, commonly referred to as ‘sparse coding’.

In this review we discuss the theory of sparse coding, methods for measuring sparsity, and the evidence to date that sparseness constitutes a general principle of sensory coding in the nervous system. We focus here primarily on neural representations in the cortex of mammals, or relatively high levels of processing in other species, but it should be noted that there is substantial evidence for sparse coding occurring at earlier stages of processing across a variety of organisms 1., 2..

Section snippets

Theory of sparse coding

The principle of sparse coding has been advanced and elaborated on by several different authors, for different reasons. Early work on associative memory models, for example, showed that sparse representations are most effective for storing patterns, as they maximize memory capacity because of the fact that there are fewer collisions (less cross-talk) between patterns [3]. Later work has similarly showed that sparse representations would be advantageous for learning associations in neural

How to measure sparseness?

To assess whether or not neurons are utilizing a sparse code, there must be a method for measuring sparseness. Many of the coding models described above employ analog-valued units that could take on both positive and negative values, with responses symmetrically distributed around zero (Figure 3). A standard measure of sparseness for such artificial units is the kurtosis, which measures the 4th moment relative to the variance squared:k=1ni=1n(rir¯)4σ43

where r is the response of the neuron, r¯

Experimental evidence

Experimental evidence for sparse coding has been found in several different sensory modalities in a variety of animals. In the visual system of primates, Vinje and Gallant 39., 42.•• have demonstrated that neurons in V1 produce sparse punctate responses when stimulated with image sequences resembling those that occur during natural vision (Figure 4). Interestingly, when the same neurons are stimulated only within their classical receptive fields (the region of space within which a stimulus

Conclusions

Although the principle of sparse coding has been discussed and elaborated for nearly three decades now, serious empirical investigation of its use in the nervous system has begun only recently. Investigating whether sparse coding is employed in a certain region, however, will require using ecologically valid stimuli (i.e. natural scenes). The studies reviewed here suggest that sparse coding provides an efficient means of representing data found in the natural world. Moreover, it provides a

References and recommended reading

Papers of particular interest, published within the annual period of review, have been highlighted as:

  • • of special interest

  • •• of outstanding interest

Acknowledgements

We thank P Kanerva, J Johnson, K O’Connor, and D Warland for helpful comments on the manuscript, and M Bethge and K Koepsell for providing Figure 2.

References (54)

  • E.B. Baum et al.

    Internal representations for associative memory

    Biol Cybern

    (1988)
  • P. Kanerva

    Sparse distributed memory and related models

  • C. Zetzsche

    Sparse coding: the link between low level vision and associative memory

  • D.J. Field

    What is the goal of sensory coding?

    Neural Comput

    (1994)
  • G. Palm et al.

    Associative data storage and retrieval in neural networks

  • H.B. Barlow

    Single units and sensation: a neuron doctrine for perceptual psychology?

    Perception

    (1972)
  • B.A. Olshausen

    Principles of image representation in visual cortex

  • D.J. Field

    Relations between the statistics of natural images and the response properties of cortical cells

    J Opt Soc Am A

    (1987)
  • J.J. Atick

    Could information theory provide an ecological theory of sensory processing?

    Network

    (1992)
  • J.H. van Hateren

    A theory of maximizing sensory information

    Biol Cybern

    (1992)
  • D.L. Ruderman

    The statistics of natural images

    Network: Comput Neural Systems

    (1994)
  • E.P. Simoncelli et al.

    Natural image statistics and neural representation

    Annu Rev Neurosci

    (2001)
  • B.A. Olshausen et al.

    Emergence of simple cell receptive field properties by learning a sparse code for natural images

    Nature

    (1996)
  • A.J. Bell et al.

    The ‘independent components’ of natural scenes are edge filters

    Vision Res

    (1997)
  • J.H. van Hateren et al.

    Independent component analysis of natural image sequences yields spatiotemporal filters similar to simple cells in primary visual cortex

    Proc R Soc Lond B Biol Sci

    (1998)
  • A. Hyvarinen et al.

    Emergence of phase and shift invariant features by decomposition of natural images into independent feature subspaces

    Neural Comput

    (2000)
  • M.S. Lewicki

    Efficient coding of natural sounds

    Nat Neurosci

    (2002)
  • Cited by (0)

    View full text