Abstract
A visual-to-auditory sensory substitution device initially developed for the blind is known to allow visual-like perception through sequential exploratory strategies. Here we used functional magnetic resonance imaging (fMRI) to test whether processing the location versus the orientation of simple (elementary) “visual” stimuli encoded into sounds using the device modulates the brain activity within the dorsal visual stream in the absence of sequential exploration of these stimuli. Location and orientation detection with the device induced a similar recruitment of frontoparietal brain areas in blindfolded sighted subjects as the corresponding tasks using the same stimuli in the same subjects in vision. We observed a similar preference of the right superior parietal lobule for spatial localization over orientation processing in both sensory modalities. This provides evidence that the parietal cortex activation during the use of the prosthesis is task related and further indicates the multisensory recruitment of the dorsal visual pathway in spatial processing.
1. Introduction
When deprived of its natural input, the “visual” cortex of early-blind subjects is widely activated during a variety of perceptual and cognitive tasks and its cross-modal recruitment may account for the improved perceptual abilities of these subjects in the auditory and tactile modalities. Sensory substitution prostheses translating vision into touch or vision into audition were developed initially for the blind since these devices could take advantage of the cross-modal plasticity changes induced by blindness. It should be noted, however, that cross-modal interactions are increasingly being recognized as playing a vital role in normal perception too. The visual cortex is functionally divided into a ventral/identification and a dorsal/localization stream [1–4]. Interestingly, using a sensory substitution device (SSD) often activates brain areas within both the dorsal and the ventral visual pathways, not only in the blind but also in normally sighted subjects [5–11]. Despite the increasing number of studies on sensory substitution, little is known about the specific versus nonspecific nature of these brain activations as regards to the task performed with the SSD. One may hypothesize that the brain activations elicited during the use of an SSD are partly driven by mechanisms specific to sensory substitution (i.e., the conversion of sounds or tactile sensations into “visual” representations) but also depend on the nature of the task performed with the SSD (e.g., stimulus identification or localization) [12, 13]. It is difficult to dissociate the brain activations related to the sensory substitution process per se from those specific to the task performed with the device since, to date, very few studies have directly compared the brain activation patterns in the same individuals who performed different tasks on identical stimuli with an SSD [6]. Furthermore, spatial and nonspatial aspects are most often jointly engaged when using an SSD. For instance, using an SSD often requires stimulus exploration strategies and a sequential perception and memorization of each single constituting element of a more complex object (due to the relatively narrow perceptual field of the SSD and its low resolution). All these aspects of SSD use involve working memory and spatial components. In the present study, we used a visual-to-auditory SSD to test whether processing the location versus the orientation of identical “visual” stimuli encoded into sounds modulated the brain activity within the dorsal visual stream in proficient SSD users. In vision, while orientation sensitivity has been reported in both dorsal (e.g., [14] for inferior parietal sulcus; [15] for parieto-occipital junction; [16] for the superior parietal lobule) and ventral streams (e.g., [17–19] for the lateral occipital complex; see also [20]), the localization is properly a spatial task that ought to activate the dorsal stream only. In addition, to reduce the importance of the spatial and working memory components inherent to the use of the SSD and to minimize the brain activation relating to decoding the SSD signal during the tasks, we used very simple stimuli: a horizontal and a vertical bar that were presented separately at the middle top or middle bottom of the SSD perceptual field. Such simple stimuli did not require any exploration strategies or self-generated exploratory movements from the subject, and they could be apprehended globally and almost instantly.
2. Materials and Methods
2.1. Subjects
Sixteen right-handed male volunteers, without any history of neurological, psychiatric, or sensory impairment participated in the present study (mean age: years). Fourteen of these volunteers learnt to use the SSD while the two remaining subjects stayed naive regarding the SSD coding scheme and learnt instead the arbitrary association between each SSD sound and its corresponding visual representation (i.e., a vertical and a horizontal bar placed at different locations in the SSD perceptual field). In addition, two of the fourteen subjects were involved in a separate control experiment that took place after the sensory substitution experiment and that used a “visual” version of the stimuli. All volunteers were blindfolded during all the conditions involving the SSD sounds, and they never saw visually the stimuli before the end of all the experimental conditions involving the SSD. The participants gave their written informed consent prior to the experiment, the protocol of which was approved by the Biomedical Ethics Committee of the School of Medicine of the Université catholique de Louvain.
2.2. Sensory Substitution Device
As SSD we used the so-called prosthesis substituting vision with audition (PSVA). Initially developed for the blind, it translates real-time monochrome images from a miniature head-mounted video camera (frame rate: 12.5 Hertz) into sounds using a pixel-frequency relationship [22]. With head movements, visual frames are grabbed in real time and then transformed into a set of corresponding complex sounds enabling recognition [23]. A demonstration is available at http://www.perceptionweb.com/misc.cgi?id=p2607/. Briefly, the camera image is pixelated according to a simplified dual resolution model of the human retina (see Figure 1). This artificial retina consists of a square matrix of 8 × 8 large pixels with the four central ones replaced by 8 × 8 smaller pixels representing the fovea, which has four times the resolution of the periphery. A single sinusoidal tone is assigned to each pixel of the artificial retina with frequencies increasing from left to right and from bottom to top; frequencies range between 50 and 12,526 Hz. The grayscale level of each pixel modulates the amplitude of its corresponding sine wave. The final auditory output of the SSD is the real-time weighted sum of all 124 sine waves that the subject hears through headphones.
2.3. Conditions and Stimuli
There were three conditions in the sensory substitution experiment: localization, orientation detection, and an SSD control condition. The control condition aimed at controlling the brain activation related to auditory processing, general attention aspects, and working memory mechanisms that were present in all sensory substitution conditions. Two subjects naive to the SSD code also underwent the same three conditions. The experiment in naive subjects aimed at testing whether an acoustic association elicited similar brain activation as the one observed when subjects processed the information once actually “decoded” according to the SSD code. The three conditions were also adapted to the visual modality for a control experiment in two subjects in vision. The purpose was to provide a reference regarding the neural substrate of stimulus localization and orientation detection in vision and with the same stimuli.
The stimuli consisted in a single bar (formed by 3 continuous pixels at the SSD fovea scale) presented either horizontally or vertically at the middle top or the middle bottom of the SSD perceptual field (localization and orientation conditions) or three isolated dots placed randomly in the fovea of the SSD perceptual field (control condition) (see Figure 2). It is worth noting that the spatial information was coded using arbitrary differences in frequencies in the 8 × 8 fovea and did not involve any auditory cues that convey naturally information about stimulus location or spatiality, like left/right stereo panning. Therefore, the access to the spatial information could only be done using the SSD code (and not intuitive auditory cues). Stimuli were provided in pairs and presented sequentially one by one with an interstimulus interval of one second. For the SSD experiment, the vertical and horizontal bars were presented successively at the predetermined locations in the SSD perceptual field (without any movement) and the corresponding sounds were then recorded to be delivered later on in the fMRI magnet using headphones (SDS device, the fMRI.pl group, http://www.fmri.pl/). It is worth noting that the sound corresponding to the same horizontal bar (and the same vertical bar) was different if presented at a different location in the SSD perceptual field and the sound corresponding to the horizontal bar was also different from the one of the vertical bar when presented in the same location in the SSD perceptual field (see Figure 3). In the additional visual experiment, the same “visual” stimuli as those encoded into sounds and used in the SSD conditions were backprojected using an MRI compatible projector placed at the rear of the magnet and viewed through a tilted mirror mounted on the head coil (Silent Vision System, Avotec, Inc., http://www.avotec.org). Stimuli were flashed one by one at the middle top or the middle bottom of the screen (experimental conditions), or three isolated square dots were displayed randomly in the half centre of the screen (control condition), with the same time course as in the SSD experiment. A fixation point was displayed at the centre of the screen during inter-stimulus intervals and resting periods. The subjects were instructed to keep their eyes on the fixation point whenever present and to avoid head or eye movements as much as possible throughout the experiment. Although the use of very simple stimuli may not be optimal to probe strong brain activation within the visual cortex, stimuli of a similar level of complexity were successfully used in previous studies on spatial processing in vision [3].
2.4. Task and Procedure
We used a one-back comparison task in all conditions (see Figures 3 and 4). During the experimental conditions (localization and orientation detection), the subjects had to determine whether the second element of each pair was the same or different regarding either its location or its orientation. During the SSD control condition, subjects had to determine whether each dot combination was the same or different as compared to the previous one. It is worth noting that unlike the localization and orientation conditions the control condition did not strictly require from the subject to decode the sensory substitution signal, but could rather be done based on a simple acoustic comparison of the pair of sounds. The spectral acoustic complexity of each stimulus was similar across the three conditions (see Figure 3).
2.5. Training Sessions
2.5.1. Learning the SSD Code
All subjects were trained to use the SSD in five 1-hour sessions with the protocol adapted from Arno and collaborators [23] and successfully used by Renier and collaborators [24–26]. Subjects were trained to recognize simple and complex 2D figures, formed by vertical, horizontal, and oblique lines. One additional 1-hour session was devoted to the familiarization to the task and stimuli that would be used in the fMRI experiment. In order to ensure a good performance level in the noisy environment of fMRI, a score of 85% of correct answers was required at the end of this training period. Subjects who would not reach this criterion at the end of the last training session or familiarization session would have been discarded from the experiment, which did not happen in the present study. All subjects were blindfolded during all conditions involving the sensory substitution device and never saw visually the stimuli before the end of the experimental conditions involving the SSD.
2.5.2. Learning the Acoustic Association
During three 1-hour sessions, the two subjects naive to the SSD code learnt to associate the sound of each stimulus coded by means of the SSD to its corresponding visual representation (the vertical and the horizontal bars presented at the top or the bottom of the SSD perceptual field). Each sound was presented several times, and subjects had to determine whether the sound corresponded to a horizontal or to a vertical bar or was placed in the upper or in the lower part of the SSD perceptual field. After each trial, the correct response was provided verbally to the subjects. We did not present the stimuli visually to prevent inducing visual memories that would have potentially influenced the brain activations during the subsequent fMRI experiment. At the end of the second training session, the two subjects were able to perform the task with >80% of accuracy. During the third session, subjects were familiarized to the fMRI task (one-back comparison). It is worth noting that these subjects received the same instructions as the other subjects for the fMRI experiment; they had to determine whether the second stimulus of each pair was the same or different than the preceding one regarding either its location or its orientation.
2.6. fMRI Paradigm
A block design paradigm was used with conditions alternating with baseline. Each condition was announced verbally at the end of each preceding resting period (i.e., “localization,” “orientation,” “control”). One pair of stimuli was presented in each block. Each stimulus was presented for four seconds with an inter-stimulus interval of one second. Subjects provided their response by pressing the button of one of two response pads, one held in each hand. In order to control any potential effect of the lateralization of brain activation related to the responding hand, we counterbalanced the response buttons across the subjects in the experimental conditions (e.g., right for “same” and left for “different”).
2.7. 3D-MRI and fMRI Acquisition
Structural brain imaging was obtained in all subjects in the bicommissural (AC-PC) orientation [21] on a 3 Tesla MRI unit (Achieva, Philips Medical systems) using a 3D fast T1-weighted gradient echo sequence with an inversion prepulse (Turbo field echo (TFE), TR [repetition time] = 9 ms, TE [echo time] = 4.6 ms, flip angle = 8 degree, 150 slices, 1 mm thickness, in plane resolution = 0.81 × 0.95 mm). The field of view was 220 × 197 mm, and the SENSE factor (parallel imaging) was 1.5. We used an 8-channel phased array head coil. Foam pads restrained the head, and eyes were blindfolded throughout the study, except in the visual control experiment.
Blood-oxygen-level-dependent (BOLD) fMRI data were acquired using a 2D simple shot T2*-weighted gradient echo-planar imaging (EPI) sequence (TR = 3000 ms, TE = 32 ms) with 44 axial slices (slice thickness = 2.6 mm) in the AC-PC orientation. The matrix was 112 × 112 × 44, and the field of view was 220 mm. In plane resolution was 2.12 mm. The fMRI paradigms consisted in 4 runs of 21 alternating epochs of experimental or control conditions and rest (9 s per active epoch, 3 brain volume repetitions, alternating with 12 s [4 repetitions] resting periods). Each condition was assessed 7 times in a separate run, in counterbalanced order. The warning (the name of the task) was provided at the end of the preceding resting period, and stimulus onset was synchronized with the acquisition of the first slice of each epoch. Subject responses were recorded by key pressing at the end of each epoch, when the SSD sound or images were automatically turned off.
2.8. Data Analysis
Data were processed and analyzed using Statistical Parametric Mapping (SPM 2, The Wellcome Department of Imaging Neuroscience, London, UK, http://www.fil.ion.ucl.ac.uk/spm/), implemented in Matlab (MathWorks Inc., Sherborn, MA, USA). The first four volumes of each run were discarded to allow for T1 equilibration. The individual structural (TFE) brain volume was coregistered to the first remaining fMRI volume of the corresponding participant. The 3D structural volume was then spatially normalized into the referential defined by the MRI template supplied by the Montreal Neurological Institute (MNI) based on the atlas of Talairach and Tournoux [21] implemented in SPM. The fMRI data were spatially realigned using a least squares approach to estimate a six-parameter rigid body transformation for each scan (trilinear interpolation [27]) and further spatially normalized in one single step using the normalization parameters derived from the 3D TFE normalization. This procedure resulted in normalized fMRI scans with a cubic voxel size (2 × 2 × 2 mm) for individual and group analysis. Next, a spatial smoothing with a Gaussian kernel of 6 mm (full width at half maximum, FWHM) was applied in order to reduce the residual anatomical and functional variability between participants.
Condition-related changes in regional brain activity were estimated for each participant by a general linear model (GLM) in which the responses evoked by each condition of interest were modeled by a standard hemodynamic response function. The contrasts of interest were first computed at the individual level to identify the cerebral regions significantly activated by each condition. To analyze the data from the sensory substitution experiment, significant brain activation patterns in the group were then examined at the whole-brain level in a random-effects analysis (RFX) using one-sample -tests, with the statistical threshold set at , corrected for false discovery rate (FDR) and extending to at least 10 contiguous voxels. Alternatively, a threshold of (uncorrected for multiple comparisons with a voxels cluster size extending to at least 10 contiguous voxels) was used in the condition comparisons depending on the strictness of the contrasts performed. Given the small size of the sample used in the visual and auditory control experiments, significant brain activation patterns averaged from the two tested subjects were estimated in a fixed-effects analysis (FFX), with the same statistical thresholds as in RFX analysis.
3. Results
3.1. Behavioral Performance
The behavioral performance during the SSD fMRI experiment was 77.8% (SD = 7.1), 78.1% (SD = 9.4), and 99.5% (SD = 2.3) of correct response for the localization, the orientation detection, and the SSD control condition, respectively. A Student -test performed on the scores showed no difference between the localization and the orientation condition (), whereas the performance of the control task was significantly better (). The mean percentages of correct responses were 80.3, 78.5, and 91.06 in the acoustic association experiment and 91.06, 92.84, and 96.42 in the visual control experiment for the localization, the orientation detection, and the control condition, respectively (all values > 0.05, although there was a trend towards better scores in the control condition compared to the orientation condition () during the acoustic association experiment only).
3.2. fMRI Results
3.2.1. Sensory Substitution Experiment
The one-back comparison of dot combinations used as control condition in SSD experiment, that is, the contrast {control minus rest} showed the expected brain activation in primary and secondary auditory brain areas, bilaterally (Table 1). The brain activation pattern also included frontal brain areas such as the superior frontal gyrus bilaterally (BA 9), as well as the insula and the precuneus (BA 7) bilaterally. Additional recruitment of subcortical brain areas was observed in the lentiform nucleus (putamen) bilaterally and the left thalamus (see Figure 5 and Table 1).
(a) Control (dots) minus rest
(b) Localization minus control
(c) Orientation minus control
(d) Localization conjunction orientation
The localization condition, that is, the contrast {localization minus control masked (inclusive) by localization minus rest} revealed a widespread activated network including the right superior frontal gyrus, the middle and inferior frontal gyri bilaterally, the left and right inferior parietal lobule (BA 40), as well as the left and right superior parietal lobule (BA 7) and precuneus (BA 7) and the cerebellum bilaterally (Table 1).
The orientation detection, that is, the contrast {orientation minus control masked (inclusive) by orientation minus rest} showed a roughly similar brain activation pattern including the right superior frontal gyrus, the middle and inferior frontal gyri bilaterally and the left inferior parietal lobule (BA 40), superior parietal lobule (BA 7), and precuneus (BA 7) (Table 1). The left thalamus and the right cerebellum were also activated.
A conjunction analysis between the localization and orientation conditions (minus control) confirmed the equivalent recruitment of the right superior frontal gyrus (BA 6, 8) and the middle and inferior frontal gyri (BA 9, 45, 47) bilaterally, as well as the activation of the precuneus (BA 7) and the superior and inferior parietal lobules (BA 7, 40) in the left hemisphere in both conditions (see Table 2).
The differential contrast between the two experimental conditions, that is, the contrast {localization minus orientation masked (inclusive) by localization minus rest} (at a threshold of uncorrected in combination with a cluster size threshold of 10 continuous voxels), showed that the localization task tended to be associated with larger activation in frontal, parietal, and occipitoparietal brain areas mainly in the right hemisphere, as compared to the orientation detection (Table 2). These brain activation foci included the right superior, middle, and inferior frontal gyri (BA 6, 10, 46), the right superior occipital gyrus/precuneus (BA 19, 7) and inferior parietal lobule (BA 40), the right superior parietal lobule (BA 7), and the middle temporal gyrus bilaterally (BA 19, 21, 39) (see Figure 6 and Table 2). The inverted contrast {orientation minus localization masked (inclusive) by orientation minus rest} did not reveal any significant activation focus at the selected threshold.
(a) Localization minus orientation (SSD)
(b) Localization minus orientation (Vision)
3.2.2. Visual Control
We used an FFX analysis in the two subjects involved in this control, with a threshold of FDR corrected in combination with a cluster size threshold of 10 continuous voxels. The visual control condition as contrasted to baseline was associated with the expected recruitment of a large frontoparietal network, known to be engaged in visuospatial working memory tasks, and including visual association areas (see Table 3). The contrast {localization minus control masked (inclusive) by localization minus rest} disclosed brain activation foci in occipital and parietal areas, included in the superior parietal lobule (BA 7) and the cuneus (BA 17, 18) bilaterally. The contrast {orientation minus control masked (inclusive) by orientation minus rest} did not reveal any significant activation focus at the selected threshold. When we compared the localization condition with the orientation condition, that is, using the contrast {localization minus orientation masked (inclusive) by localization minus rest} (at a threshold of , uncorrected, in combination with a cluster size threshold of 10 continuous voxels), brain activation voxels clusters survived in the right superior parietal lobule (BA 7) and, to a lesser extent, in the right and left superior occipital gyrus/precuneus (BA 19, 7). Obviously the reverse contrast {orientation minus localization} did not reveal any activation focus at the selected threshold.
3.2.3. Acoustic Association
We used an FFX analysis in the two subjects who had learned to associate each sound with its visual representation to estimate the condition-related brain activity changes when based on acoustic association. None of the contrasts performed ({localization minus control}, {orientation minus control}, {localization minus orientation} and {orientation minus localization)} revealed any activation in the parieto-occipital cortex in any of the two subjects (at a threshold of , uncorrected, in combination with a cluster size threshold of 10 continuous voxels). Only the localization task compared to control condition induced some weak activation restricted to the auditory cortex and temporal brain areas (see Table 4).
4. Discussion
This experimental design aimed to test how the brain codes orientation and localization of a visual stimulus transformed into sounds in healthy individuals who were using a sensory substitution device developed for the blind and known to allow visual-like perception through analytic cognitive operations. We measured the task-related activation patterns for two spatial tasks on visual stimuli encoded into sounds and observed a similar recruitment of frontoparietal and occipital brain areas as the corresponding tasks performed in vision. This provides compelling evidence that the parietal and occipital cortex activation during the use of the prosthesis is task related and it further indicates the multisensory recruitment of the dorsal visual pathway in spatial processing.
4.1. Task-Specific Activation beside SSD Signal Decoding
This experiment aimed to address one aspect of the use of a sensory substitution device: the division between activation related to decoding the SSD signal and this relating to the particular task being carried on the information once decoded. Differing from vision, the cognitive operations that allow visual-like perception through sensory substitution are mainly sequential and analytic. Auditory recognition of a visual stimulus through an SSD like the PSVA requires focusing attention on only parts of the stimulus at a time, while the total image has to be mentally synthesized over time. It is therefore not surprising that the use of such SSD to perceive complex environmental scenes, objects, or patterns constantly activated brain areas responsible for spatial processing (in the parietal cortex) and working memory (in frontal lobes) included in blind subjects, whereas little is known about the respective contribution of SSD signal decoding, stimulus perception, and task-specific processing in this brain activation pattern. The present study was designed to fill in this gab by showing how processing the location versus the orientation of identical “visual” stimuli encoded into sounds modulated the brain activity within the dorsal visual stream in proficient SSD users who did not use stimulus exploration strategies or self-generated exploratory movements. Furthermore, the same memorization process was used in each experimental and control condition, and, to minimize the brain activation relating to decoding the SSD signal during the tasks, we used very simple stimuli that could be apprehended globally and almost instantly. Although the use of very simple stimuli may not be optimal to probe strong brain activation within the visual cortex, stimuli of a similar level of complexity were successfully used in previous studies on spatial processing in vision [3] and in the tactile modality [28–30].
4.2. Task-Specific Activation and Hemispheric Specialization
Task-specific activation foci were observed in the dorsal visual pathway when using the SSD to compare the location or the orientation of successive stimuli, which were not present in the SSD control condition. The superior occipital gyrus, precuneus, and the inferior and superior parietal lobules (IPL, SPL) in the right hemisphere were slightly more strongly activated during the localization condition as compared to the orientation condition. Such task-related modulation of brain areas recruitment cannot be due to a difference in the difficulty level since the behavioral performance was equivalent between the localization and the orientation conditions. In addition, since the same stimuli were used in both experimental conditions, no difference in their nature or complexity could account for differences in brain activation patterns. Furthermore, most spatial aspects related to the use of the SSD were controlled and kept to a minimum in all conditions (there was no stimulus exploration and the stimuli did not comprise any spatial arrangement of isolated elements). In vision, the right SPL showed a similar preference for location over orientation processing. Although the activation focus in the right SPL did not perfectly overlap the one observed in the SSD experiment, it was located in the same structure and in its direct vicinity. In the visual location discrimination, as compared to the visual control condition (see Table 3), the SPL was the main activated structure, with, albeit to a lesser extent, the cuneus, in both hemispheres. The weaker effect for the spatial processing tasks observed in the visual modality (i.e., the restricted recruitment of the dorsal stream during location discrimination and its absence during orientation detection as compared to control condition) could be due to the relative simplicity of the tasks in vision that could have led subjects to automatically recognize and localize the stimuli during all conditions, this resulting in a reduction of the specificity of the brain activation patterns. Nevertheless, the task-specific activation patterns for stimulus localization observed both with the SSD and in vision were located mainly in the right hemisphere, in full accordance with the hemispheric specialization hypothesis according to which the right hemisphere would be more involved than the left one in spatial processing, at least in right-handed subjects [31, 32]. Surprisingly, no task-specific activation was observed for the orientation processing (as compared to location discrimination) in any of the sensory modalities although previous studies on sensory substitution that used more complex stimuli revealed clear and strong brain activations in the ventral visual streams during shape recognition tasks [5, 6, 8, 10]. Several explanations could be proposed to that (1) the stimuli used (i.e., horizontal bars and vertical bars) were too simple to engage significantly shape processing, which also might explain the absence of lateral occipital complex (LOC) recruitment and any brain activation foci in the ventral stream during the orientation judgment as compared to control condition and (2) the stimulus orientation was automatically processed during the localization condition.
4.3. Multisensory Integration and Spatial Processing in the Parietal Cortex
The involvement of common parietal brain areas in spatial processing both with the SSD and in vision indicated that part of the parietal cortex could be specialized in the processing of spatial attributes at a supramodal level. This observation was consistent with previous studies. For instance, a bilateral functional recruitment of the SPL and the precuneus was previously reported in auditory and tactile studies involving spatial processing [33–37]. These two brain structures were also described as being part of the cerebral network involved in the processing of spatial information in vision [3, 4]. The IPL, which sustains a wide range of functions related to attention, motion processing, stereo vision, and spatial working memory [38–41], was widely reported as involved in spatial hearing and in tactile localization [36, 42–50]. All these observations indicated that parts of the parietal cortex should play a role of multisensory operator for the processing of spatial information [37]. It is worth noting that a brain area similarly specialized in the processing of object shape whatever the sensory input, the lateral occipital complex (LOC), was previously found in the ventral visual stream [51–55]. Such convergence of the same functional attributes of auditory, visual, and tactile stimuli into one centralized representation would constitute an optimal form of brain organization for the rapid and efficient processing and binding of perceptual information from different sensory modalities. This might play a vital role in our unified perception of the environment and in cross-modal adaptive changes observed in congenital blindness or deafness.
4.4. Potential Role of Mental Imagery and Working Memory in Sensory Substitution Studies
Several frontal and parietal regions located mainly in the left hemisphere were commonly activated during both experimental conditions involving the SSD and were more activated during the experimental conditions than during the control condition. In particular, the inferior, middle, and superior frontal gyri, the left precuneus, and the left IPL, and SPL were similarly recruited by both the localization and the orientation comparison tasks. Interestingly, a similar recruitment of the precuneus, the IPL and the SPL was already found in most studies during the use of either an SSD substituting vision with audition (e.g., the vOICe, the PSVA) or a tactile vision substitution system (e.g., the TVSS in its pneumatic, electric, or vibrotactile variant, the tongue display unit (TDU)) to perform various tasks, for example, to recognize 2D figures [5, 9], to process object shape [10], to estimate object distance in a 3D scene [6], to determine the orientation of a letter [7], to judge the direction of a stimulus in motion [11], or to recognize a virtual route [56]. These brain activation foci and other brain areas located in the ventral and dorsal visual streams were observed in studies that used an SSD, not only in the blind but also in normally sighted subjects (see Supplementary Table 1 for a list of activated voxels clusters that were previously reported on Supplementary material available online at http://dx.doi.org/10.5402/2012/490950). Altogether, we cannot exclude that these as well as some of the brain activations observed in the present study were mediated by mental imagery mechanisms or working memory processes. The precuneus and the SPL were reported as involved in spatial mental imagery tasks [57–59], and the IPL and the SPL were also found to be involved in spatial working memory tasks (see [60] for a review). Although the subtraction of the control condition within the sensory substitution experiment was supposed to neutralize working memory aspects involved in the one-back comparison task, we cannot exclude that the experimental conditions (localization and orientation detection) engaged more strongly working memory than the control condition did. In the present study, no activation within the visual cortex was observed during the acoustic association conditions, which brought support to the hypothesis according to which sensory substitution would specifically recruit visual mechanisms [12] and engage different cognitive processes than a regular sound-object association task [5, 10]. The lack of brain modulation in the auditory paradigm was probably due to the fact that subjects used a same strategy in all tasks: the memorization of meaningless associations. Such strategy did not involve location or orientation processing. By contrast, during the sensory substitution conditions, subjects had to convert sounds into visual representations to perform the task based on the resulting “visual” representation they had built mentally [12]. Interestingly, a modulation of the brain activity according to the nature of the percept was previously observed in a study involving the same SSD and that contrasted the processing of 2D versus 3D attributes of the same stimuli [6]. The present study showed up a modulation of the brain activity according to the performed task on the same stimuli. Altogether, this leads us to favor the hypothesis according to which the brain activation patterns typically observed during the use of an SSD combine some general mechanisms related to sensory substitution and some specific processes related to the nature of the perception provided by the device (e.g., as in [6]) and to the nature of the task performed with the device (as presently).
4.5. Conclusions
Here we showed that regions in the parietal cortex were specifically recruited during spatial processing tasks both in vision and when using a visual-to-auditory sensory substitution device. In particular, the right superior parietal lobule was more activated during localization compared to orientation detection of a same stimulus in both sensory modalities. This indicates that part of the dorsal visual pathway would be a multisensory operator for spatial processing of information coming from different sensory modalities. Additional work with a greater variety of stimuli and paradigms is clearly needed to test whether other multisensory specialized regions can be found along the processing streams. Another challenge for the future is to connect the findings from sensory substitution studies in a coherent framework that defines the nature of such multisensory interactions in the healthy human and their alteration by visual or auditory deprivation.
Acknowledgments
The authors wish to gratefully thank the “Fonds special de Coopération Universitaire au Développement” for supporting P. Plaza and I. Cuevas. They would like to address special thanks to Professor Josef P. Rauschecker for his current support to P. Plaza. Thanks are also due to Pr. C. Veraart, Pr. R. Bruyer, Pr. M. Crommelinck, and Dr. O. Collignon for their comments on the design and a previous version of the paper. They wish to thank Benoit Gérard for technical development of the PSVA. A. G. De Volder is a senior research associate at the National Funds for Scientific Research (Belgium). L. Renier is supported by the Institut d’Encouragement de la Recherche Scientifique et de l’Innovation de Bruxelles (IRSIB) and the Politique Scientifique Fédérale Belge (BELSPO). This study was supported by FRSM Grants 3.4505.04 and 3.4502.08 (Belgium).
Supplementary Materials
Supplementary Table: Overview of the brain activation foci in the ventral and dorsal visual pathways reported in studies that used a sensory substitution device (SSD) in blindfolded sighted subjects.
Supplementary File 1: Auditory version of a pair of stimuli used in the experimental condition (localization or orientation) with the sensory substitution device (SSD). The sound corresponds to a horizontal bar followed by a vertical bar presented both in the middle-lower part of the artificial retina (see Figure 3, left part of the figure).
Supplementary File 2: Auditory version of of a pair of stimuli used in the control condition with the sensory substitution device (SSD). The sound corresponds to three isolated pixels presented in the SSD artificial retina (see Figure 3, right part of the figure).