Important
• Multisensory integration vs. multisensory modulation – The distinction between these two processes is very important. Integration creates a nonlinear response and a multisensory representation, while modulation affects the gain, timing, or tuning of a unisensory response in unisensory areas.
• Inverse effectiveness – An important principle of multisensory integration where the facilitation of the neuronal response is maximal when the responses to the individual unimodal inputs are weakest.
Core concepts
• Unisensory Processing: Refers to areas dedicated to processing information from a single sensory modality, involving modality-specific receptors, modality-specific relay nuclei in the thalamus (e.g., lateral/medial geniculate nucleus), and primary cortices (e.g., primary visual cortex).
• Canonical Unisensory Architecture: The structure of unisensory processing, which includes feature tuning (cells processing specific orientations, like simple and complex cells), hierarchies and streams (e.g., V1, V2, V3, and dorsal/ventral streams), and topographies (systematic spatial or frequency maps).
• Receptive Field: An area of sensory space (or on the retina) in which a stimulus is effective in activating a specific neuron, affecting its firing rate.
• Multisensory Neurons: Neurons found in multisensory areas that have more than one receptive field, and critically, these receptive fields are in spatial register with one another.
• Multisensory Integration: A convergent process in which inputs from two or more sensory modalities are combined within a neuron or circuit to produce a nonlinear response that is greater than the sum of the unimodal responses alone, leading to the formation of a joint multisensory representation.
• Multisensory Modulation: An influence where a stimulus from one modality affects the processing of another modality (e.g., enhancing or inhibiting the response, or altering timing) in a unisensory area, but without creating a multisensory representation.
• Principles of Multisensory Integration: Three basic binding principles that the brain uses to determine which stimuli to integrate, relying likely on hardwired priors (hyperpriors). These are Spatial correspondence (stimuli must originate from the same area in space), Temporal proximity (stimuli must occur within a close temporal window, typically milliseconds), and Inverse effectiveness (weak stimuli elicit maximal enhancement of the neuronal response).
• Cross-modal Associations (or Correspondences): Systematic mappings between features or dimensions from different sensory modalities (e.g., high pitch mapped onto high space) that are common across the population and are not based on specific object knowledge.
• Synaesthesia: An infrequent condition where exposure to a feature in one modality (the inducer) automatically elicits a strong, conscious sensory experience (the concurrent) in a different modality (or within the same modality), such as hearing a sound and experiencing a color.
Theories and Frameworks
• Traditional View of Sensory Processing: Posits that multisensory integration occurs only in multisensory or association areas after extensive, dedicated unisensory processing has taken place.
• Strong Multisensory Account (Ghazanfar & Schroeder, 2006): Argued that the pervasiveness of multisensory influences at all cortical levels compels the abandonment of purely unisensory processing notions, suggesting nearly all of the neocortex is multisensory.
• Current View: Acknowledges the robustness of unisensory processing while emphasizing that multisensory signals intrinsically shape cortical processing; it distinguishes between multisensory integration in multisensory areas and multisensory modulation in unisensory areas.
• Neonatal Synaesthesia Hypothesis (Maurer, 1993): A structural explanation proposing that all individuals are born with dense cross-modal connectivity, but typical pruning leaves only weak functional connections (resulting in cross-modal associations), while synesthetes retain an exaggerated degree of this connectivity.
• A Theory Of Magnitude (ATOM) (Walsh, 2003): A structural explanation suggesting that magnitude-based cross-modal associations (e.g., heavier stimulus associated with darker color) arise because of a common mechanism in the brain responsible for processing magnitude.
• Semantic Mediation (Charles Osgood, 1950s): A mediated explanation proposing that sensory experiences are mapped onto a multidimensional semantic space, primarily defined by potency (large-small), activity (fast-slow), and evaluative meaning (good-bad), and associations occur when sensations map onto the same area in this semantic space.
Notable Individuals
• Salvador Adali: Hypothesized to have been a synesthete.
• Charles Spence: Conducts extensive research on cross-modal associations and correspondences, particularly at Oxford University.
• Vincent Walsh: Proposed the A Theory Of Magnitude (ATOM).
• Ghazanfar & Schroeder: Authored the influential 2006 paper arguing that the neocortex is essentially multisensory.
• Charles Osgood: American psychologist who proposed that perceptual experiences map onto a semantic space with dimensions like potency, activity, and evaluative meaning.
• Maurer (1993): Proposed the Neonatal Synaesthesia Hypothesis regarding the shared origin of synesthesia and cross-modal associations.