[Art_beyond_sight_learning_tools] Article: The feeling of colour; You may think there's a picture of the world in glorioustechnicolour inside your head. But it's an illusion

[Shelley L. Rhodes]

New Scientist
Saturday, January 29, 2005

The feeling of colour; You may think there's a picture of the world in 
glorious technicolour inside your head. But it's an illusion

By Helen Phillips

ERIK WEIHENMAYER lost his sight when he was 13. Twenty years on, in Paul 
Bach-y-Rita's lab at the University of Wisconsin Medical School, he caught a 
rolling ball, played a game of rock, paper, scissors, walked through a 
doorway and watched a flickering candle flame. Nothing had changed with his 
eyes. Instead he "saw" with his tongue.

A camera mounted on Weihenmayer's forehead fed a signal into an electronic 
device that turned the pattern of light and dark into electrical pulses. The 
pulses stimulated an array of 144 electrodes on a grid about the size of a 
postage stamp, which zapped the coded image onto his tongue. At first he 
described the sensation as being like candy pop rocks exploding, but later 
he experienced something more "out there" in the world - a sense of space, 
depth and shape.

Cheryl Schiltz danced for the first time in seven years after just 20 
minutes wearing the same device. Normally she is unable to stand upright 
without holding onto something and concentrating on the stable things she 
can see around her, because her sense of balance has been destroyed by an 
antibiotic. If she tilts her head, the world spins and she stumbles.

But when Bach-y-Rita's device translated signals from a sort of 
spirit-level-in-a-hat into patterns of pulses on her tongue, she quickly 
learned how to read them to substitute for her missing sense of balance. The 
effect persisted even when she took the device off, and lingered longer each 
time she tried. "I was normal," she says, still emotional when she 
remembered the first time. "I was completely normal, and I had forgotten 
what it felt like."

These "sensory substitution" devices are based on a technology called the 
BrainPort, which Bach-y-Rita describes as a kind of USB connector into the 
brain. They are close to commercialisation, initially for "wobblers" like 
Schiltz, but eventually as devices for blind people too. The US military is 
interested in developing the system to guide pilots and divers through dark 
skies or murky waters, and others are looking at more frivolous uses in 
virtual reality and games. But in neuroscience and philosophy circles 
sensory substitution has attracted a great deal of interest because of what 
it is revealing about the brain and our senses.

The fact that Weihenmayer sensed something "out there" and forgot the 
tingling on his tongue, and the way Schiltz felt completely normal, even 
although she was working with a different sort of sensory feedback to keep 
her balance, suggests that the traditional separation of the senses - sight, 
hearing, touch and so on - has little bearing on how we experience the 
world. The sense organ that picks up the information, and the way it is 
delivered to the brain, seem less important than the nature of the 
information itself.

Some see this merely as a dramatic demonstration of the brain's flexibility. 
In other words, deprived of a primary source of information such as vision, 
the brain turns to a less prominent source, say touch, and extracts useful 
information from that sense. Others, however, have taken the findings much 
further, going to so far as to suggest that the traditional view of how the 
senses work is completely wrong.

The orthodox view of sensory perception is all about building internal 
pictures of the world. Sensory systems extract information from outside and 
channel it into the brain, which builds up a representation of our 
environment. Sensing is therefore the passive process of picking up signals; 
perception is the active process of turning the signals into useful 
information.

This model certainly chimes with our everyday experience: we talk about our 
mind's eye, of mental images, and so on. And there is some scientific 
evidence that it is correct. Brain imaging reveals that when people see, 
hear, feel, smell or taste, specialised parts of their brains respond, and 
the timing of the response coincides with the moment of conscious 
perception.

Another line of evidence comes from our ability to imagine things. Even in 
the absence of sensory information, we can generate images and sounds in our 
heads, and most researchers believe that the process of imagining mimics 
real sensory perception. When people imagine seeing something, their visual 
cortex lights up. Moreover, using a technique called TMS or transcranial 
magnetic stimulation, which temporarily knocks out activity in the brain 
regions it is targeting, Harvard neuroscientists Steven Kosslyn and Alvaro 
Pascual-Leone have shown that people can't imagine things if their visual 
circuits are switched off.

That's all very well, says Kevin O'Regan, a psychologist with the CNRS, 
France's national research centre at René Descartes University in Paris. But 
he is not convinced that this proves there is a representation of the world 
inside your head. In fact, he argues for a profoundly different view of 
sensory perception, and claims that Bach-y-Rita's sensory substitution 
studies support it.

O'Regan's starting point came some years back, when his interest lay mostly 
with eyes. He was curious as to how the world around us could feel 
completely stable in the face of our almost continuous eye movements, 
particularly large, jerky movements called saccades. O'Regan reasoned that 
saccades must be reported back to the brain so it could compensate for them 
as it built up its internal image of the world.

But he could find no evidence that this actually happens. There are neural 
signals associated with eye movements, but they didn't seem to be involved 
in building up successive visual snapshots into one big picture. Since then 
others have tried, and failed, to find evidence that these signals are used 
to compensate for eye movements in this way. But if the brain doesn't 
compensate for huge shifts in eye position, how can it create a stable image 
of the world?

Another puzzle came from a famous experiment by Dan Simons and Christopher 
Chabris of Harvard University. They asked volunteers to watch a recording of 
a basketball game and count the passes made by one of the teams. Early on in 
the game a man in a gorilla suit walked slowly across the court. Despite the 
fact that he was visible for about 45 seconds, around 40 per cent of the 
viewers failed to notice him. Yet when asked to watch the game with no task 
in mind, they all saw it immediately (New Scientist , 18 November 2000, p 
28). To Simons, this is strong evidence that, despite the impression we have 
of seeing a complete and detailed image of the world, there's a lot missing. 
We rely on the brain to fill in the blanks.

O'Regan goes one step further. He suggests that the mental image is not only 
incomplete, it is completely absent. We don't reconstruct the world in our 
mind, we merely glimpse it in fleeting fragments. "There is no internal 
picture," he says. Where most researchers would argue that seeing is all 
about building up an internal image, O'Regan has us flitting from one visual 
element to the next, only becoming aware of things when we need information. 
In this, O'Regan departs radically from the traditional view of sensory 
perception: sensing becomes an active rather than a passive process, with 
potentially profound ramifications.

O'Regan had been struck by an earlier version of Bach-y-Rita's device, in 
which blind or blindfolded volunteers wore a larger array of electrodes 
taped onto the skin of their back or abdomen. One volunteer, with the array 
on his front, was "viewing" objects using a camera mounted on a tripod, but 
not getting anywhere. Out of frustration he grabbed the camera and started 
waving it around. When he started actively manipulating the camera in this 
way, something dramatic happened. He very quickly moved from merely feeling 
a tingling on his stomach to sensing the presence of external objects. 
Another volunteer, using a head-mounted camera, also suddenly felt the 
outside world become very real when he grabbed the zoom control - and almost 
fell over backwards as objects surged towards him.

What this suggests is that substituting touch information for visual 
information can produce a vision-like experience, but only when people 
actively control the camera in some way. Weihenmayer, for example, could 
almost see objects with his tongue. He didn't taste them, and after a short 
while he didn't feel them either. But this sensation only happened when the 
camera was mounted on his head, so he could move it as if he were scanning 
with his eyes. Similarly, blind people tapping with a cane experience open 
space at the end of their stick, not vibrations on their hands. They, too, 
are seeking information about the space around them.

Results like these have convinced O'Regan that sensory perception is not 
about passively collecting information but actively seeking it, and noticing 
how the information responds to our actions. We sense the world not by 
soaking up information, but by taking physical actions to interrogate it. 
"If the story is right, sensations are not generated in the brain," says 
O'Regan. "They are things we do." Sensory substitution works because it 
matters less to our brain where information comes from than the manner in 
which we gather it.

If it is right, O'Regan's theory doesn't just explain sensory substitution, 
it has philosophical implications too. In particular it suggests a solution 
to one of the "hard problems" of consciousness: why does seeing something 
feel different from touching it? The answer certainly doesn't seem to lie in 
the electrical activity of the brain. Whatever sensory stimulus triggers the 
activity, be it touch, taste, sight or sound, the information is translated 
into electrical pulses. And no one has ever been able to find anything 
unique about these pulses, or where they are sent to in the brain, to 
explain why they produce different sensations.

That spongy feeling

O'Regan believes that his "sensorimotor" theory might provide an answer. 
Maybe, he says, touch, taste, sight and sound feel different because we have 
to perform different actions to collect the information.

Take, for example, the softness of a sponge. Where does the feeling of 
softness come from? No one has ever found a neural mechanism or specific 
part of the brain that exclusively lights up when you feel something soft. 
That, says O'Regan, is because there isn't one. Working with philosopher 
Erik Myin at the University of Antwerp in Belgium, he has proposed that the 
feeling of softness comes from how you go about seeking information about 
the sponge. When you press the surface, it gives way. This is a different 
action from touching a sharp or hard surface, or a liquid.

While the theory seems to make sense for touch, or the difference between 
seeing and touching, what about the hardest "hard problem" of all, the 
sensation of different colours? How can we explain "redness" or "greenness" 
in terms of different actions? To complete his theory, O'Regan needed to 
find unique actions or activities that are associated with perceiving 
different colours.

It seemed an impossible task, but O'Regan and colleague David Philipona from 
the Sony Computer Science Laboratory in Paris were in for a surprise. When 
they looked at the physical properties of coloured surfaces they found 
fundamental differences in the way the different colours interact with 
light. In classical models, reflections from surfaces are the sum of two 
sources: one that behaves like the reflection from a matt surface, and 
another that behaves like the reflection from a sheet of glass laid over the 
matt surface. As we move our eyes, both types of reflection change their 
spectral composition, and the relationship differs according to colour you 
are looking at.

O'Regan suggests that as we move our eyes over a coloured surface, we detect 
something of this change in relationship. And by that we experience colour. 
The key point as far as perception goes is what happens when we probe the 
environment: it's not the brain activation itself that gives the colour. The 
researchers have found that primary colours produce particularly distinctive 
changes, which may explain why they are universally recognised as special.

Already, O'Regan's ideas have doubters. Bach-y-Rita thinks the explanation 
for sensory substitution lies with the remarkable flexibility of the brain. 
There are multiple pathways from all the senses to all the different sensory 
areas in the brain, he says. If you lose the main input from the eyes to the 
visual cortex, say, weaker pathways from the skin, ears, tongue and so on 
take over. This is what happens in the brains of Braille readers, who 
recruit their visual cortex when feeling the forms of the letters.

It may not be long before we know who is right, as O'Regan and colleagues 
are busy thinking up testable predictions of their sensory substitution 
theory. One such prediction is that it should be possible to make the 
substitution feel more convincing by making the information-gathering action 
"mimic" the original as closely as possible in the new medium.

To that end, O'Regan and colleague Malika Auvray have rigged up a video 
camera that represents the visual world in sound. Brighter objects become 
louder sounds, objects high up in the visual field are represented by high 
pitches and object low down by low pitches, while lateral position is 
represented with stereo sound. It is a little hard to imagine, but say the 
camera was looking at a light bulb in the centre of the field of view, you 
would hear powerful noise made up of a limited range of pitches centred in 
space. With a horizontal strip light you'd hear a smaller range of pitches 
over a wider space. As you move the camera, the sound changes.

In preliminary tests with a similar device designed by engineer Peter Meijer 
from Eindhoven in the Netherlands, the signals took a little getting used 
to. But after a couple of hours of feedback either from touching or being 
told what they were viewing, people were able to recognise objects by their 
sound. They could tell plants from statues and crosses from circles. But 
they weren't fooled into thinking they were seeing. O'Regan's prediction is 
that the more they can make the sound information follow the rules of visual 
images, the more like seeing it will feel. For example, Meijer's system has 
a delay between moving the camera and hearing the sound. Another simple 
tweak would be to cut off the sound each time the subjects blink, which is 
exactly what happens to our visual world, though we scarcely notice it.

Perhaps one day blind people will play rock, paper, scissors in stereo 
surround sound. And if O'Regan is right, they could feel almost as if they 
are seeing. Now that really would be a sensation.






--
BlindNews mailing list

Archived at: http://GeoffAndWen.com/blind/
Address message to list by sending mail to: BlindNews at blindprogramming.com

Access your subscription info at: 
http://blindprogramming.com/mailman/listinfo/blindnews_blindprogramming.com



-- 
No virus found in this incoming message.
Checked by AVG Anti-Virus.
Version: 7.0.300 / Virus Database: 265.8.7 - Release Date: 2/10/2005





-- 
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.300 / Virus Database: 265.8.8 - Release Date: 2/14/2005