To pick up from the last post, we are figuratively within the brain and we want to “think like the brain” about how it creates our visual perceptions. We will skip the fine details of neural anatomy and physiology and stay at a highly conceptual level.
The brain has access to an immense range of information, not only from the sensation of your eyes but also from monitoring the rest of your body and from what your brain has learned throughout your life. The most important new information is, of course, the contemporaneous sensations of your two retinas.
As I mentioned before, the form of that information from each retina is a series of “spike discharges” that remind me of a digital signal -- a series of 1's and 0's or in this case a spike and then no spike. If you went to either of the two websites I recommended in my last post, you might have listened to sounds of these “spike discharges” from actual experiments in which certain light patterns were directed at photoreceptors in a retina. Those photoreceptor cells (the rods and cones) that provide information for a particular checkerboard square in your brain, first provide the information to other retina cells that aggregate the information before it leaves that retina. There are about 125 million photoreceptors in each of your retinas but only about one million nerve fibers in the optic nerves that send that information to your brain.
From my reading, as a non-scientist, of sources like Stephen Palmer’s Vision Science book, that I mentioned earlier, there seems to be a consensus among scientists regarding the gross steps by which your brain uses the information from your eyes to create visual perceptions. There are several big steps or tasks that seem sequential when we think about them. More likely, each of these steps not only feeds the next step but also feeds information back to permit the revision of the previous steps.
Although these steps are reasonably well described in Vision Science and elsewhere, they are certainly not completely understood.
Your brain first receives information from your eyes, actually a separate set from each retina. (Remember that your eyes are located at different positions so the visual information from the left and right eyes will not be identical.)
Imagine the organization of the information that your brain receives as a very large spreadsheet or perhaps a checkerboard. The value within each square of the checkerboard represents a recorded measure of the spike "firing rates" from those photoreceptor cells within the retina that are associated with that particular square on the checkerboard. These firing rates for each square in the checkerboard will vary in time according to the changing pattern of light that is striking the photoreceptors. Think of information in each of the checkerboard squares as luminance values (shades of grey, from black to white) that come from those associated photoreceptors in that eye.
As time passes, the overall pattern within the checkerboard changes in what vision scientists call an "optic flow." Some of these changes result from the motion of objects within the scene. Additionally, the motion of your head and particularly the motion of your eyes, even when your head is perfectly still, are other major sources of changes within the "checkerboard." These latter eye movements are called saccades. Saccades ensure that eyes' region of best visual acuity (a 20 degree cone centered on each eye) is always pointed towards the highest interest areas within the visual scene. If you are startled by something in front of you, for instance, your eyes will shift to that location in less than a second. "Routine" saccade movements, like those during reading, occur even more quickly and continuously. Each one of these rapid involuntary eye movements results in an equally rapid change in the contents of the "checkerboard."
Unlike a simple animation, the notion of an “optic flow” more resembles a movie of super high-speed photography, e.g. the classic, slow motion movie of, let's say, a bullet passing through a pane of glass. If I did my arithmetic correctly, the implied data rate of this optic flow is in league with the capacity of the residential high-speed Internet connection to my home.
Upon getting this information from the eyes, your brain first constructs a so-called “Primal Sketch” of the scene from the available information. To construct this Primal Sketch, your brain must search for clues about the objects that make up the scene. This earliest stage has been called a “raw Primal Sketch.” Your brain tries to detect significant cases of sharp linear contrast in the pattern of luminance values within the “checkerboard” array of retinal data from the scene.
One obvious example would be the detection of lines of sharp contrast that are probably edges of some object. Another example of linear sharp contrast is a bar shape, i.e. a simple rectangle, that could be an open door within in the scene. There are also shapes called “blobs” and a fourth category called “line terminations.”
As I hope you saw for yourself in the videos I recommended in my last posting, these patterns are not imagined by scientists as what ought to be. Each pattern is the empirical result of experiments to define that pattern for which a particular photoreceptor cell is tuned to react. There is even one classic example on the site I recommended in which the Nobel laureate, David Hubel, draws the bar shape with a felt-tip pen right on the screen, based upon the Geiger-counter-style sounds he hears as he moves the light source around in the vicinity of the photoreceptor cell.
One final thought. The fact that your brain’s initial understanding of a scene requires a logical distinction between a line and the termination of a line seems to be a compelling demonstration of just how far your brain must go before it can give you a useful visual perception of a scene. And remember, your brain does all this, and more, so quickly that you are unaware it is ongoing.
So now we have the raw Primal Sketch. Next time we will move on to the full Primal Sketch.