Learning where to look: a foveated visuomotor control model

Learning where to look:
A foveated visuomotor control model

Emmanuel Daucé, Pierre Albigès & Laurent Perrinet

CNS*2019, 15/7/2019

Outline

Motivation

Methods

Results

Conclusion

Computer vision

(OBJECTIVE)

Past 5-10 years have seen a huge development of machine learning/deep learning based image processing, indeed artificial vision has been revolutioned by the incredible capability of convolution-based deep networks to capture the semantic content of images/photographs. Their success relies on a reduction of parameter complexity through weight sharing in convolutional neural networks applied over the full image. In order to increase the recognition capability, there has been an inflation in the number of layers needed to process the pixel information. Finally, the processing of large images can be done at a cost that scales quadratically with the image resolution. All regions, even the “boring” ones are systematically scanned and processed in parallel fashion at high computational cost.

Typical ML processing : - bounding boxes around objects of interest - (at best) Linear scaing in #pixels

Human vision

Statistical Viewpoint

Attention vs. Scene Understanding

Given a generative model of the environment, one can define a quantity called the bayesian surprise that tells how different is the visual data from your initial guess. Itti and Koch predict that that the eye is attracted by the bayesian surprise, i.e. by the regions of the image that depart the most from the baseline image statistics. This allows to define salient regions in an image and draw saliency maps over an image that can predict where the eye is attracted the most. This may explain up to 50% of the human scan path, but it is purely phenomenological.

Laurent Itti and Christof Koch. A saliency-based search mechanism for overt and covert shifts of visual attention. In: Vision Research 40.10-12 (2000), pp. 1489--1506.
M. Kümmerer, L. Theis, and M. Bethge Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet ICLR Workshop, 2015

Top down : (sequential decision)

A more detailed modelling originally proposed by Najemnik and Geisler proposes a sequential model of natural vision in a visual search task. Given a generative model of the visual field (an ideal observer that knows everything about how the visual data is generated), and given a statistics over the hypothesis space (Where is Waldo?), the model decides where to look next : choose the next viewpoint that will provide the best information gain. The selection is reiterated several times until enough evidence is gathered.

In general, the active inference setup means using a generative model to quantify the benefit of doing a certain action (changing viewpoint) to reduce the posterior entropy given an history of past actions (viewpoints), that corresponds to a better understanding of the visual scene.

J Najemnik and Wilson S. Geisler. Optimal eye movement strategies in visual search. In: Nature reviews. Neuroscience 434 (2005)
Nicholas J Butko and Javier R Movellan. Infomax control of eye movements. In: Autonomous Mental Development, IEEE Transactions on 2.2 (2010)
Fu, J., Zheng, H., & Mei, T. (2017). Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4438-4446).

Outline

Motivation

Methods

Results

Conclusion

Principles for central and peripheric vision

So what we propose here is to go a little further in a biomimetic implementation of an artificial vision system. (Why : biomimetic systems are the result of a continual optimization throughout ages of evolution: they optimize signal processing under strong material and energy constraints, for specific surfival purposes.) Objective : build an effective artificial foveal vision We concentrate her on the foveal vision case

What is specific with foveal vision? Foveal vision is a trick that was selected by natural selection : a compromise between resource saving and accuracy (budgeted vision) The fovea that concentrates most of the photoreceptors, represents less than 2% of the total visual field In a foveal vision setting, the current view may allow you to tell there is an object of interest in your peripheral vision (for instance a face),that you can not identify, and you need to make a saccade to identify the person.

So in order to analyze a complex visual scene, there are two types of processing that need to be done. On the one side, you need to process in detail what is at the center of fixation, that is the region of interest currently processed. On the other side, you also need to analyze the surrounding part, even if the resolution is low, in order to choose what is the next position of fixation. This basically means making a choice of “what’s interesting next”. You do not necessarily need to know what it is, but you need to that it’s interesting enough, and of course you need to know what action to take to move the center of fixation at the right position.

If we consider now the information gain metric, it shows an interesting correspondence with the central/peripheral processing trade-off. In a sequential setup, the rightmost term can be interpreted as the current state if understanding before the saccade is actuated, that is the information present at the center of the retina -- and the left term can be seen as the future state of understanding after the saccade is executed, that relies on interpreting the peripheral information.

Methods - ''Experimental'' setup

We reproduce in simulation the conditions of a psychophysic experiment.

The problem is to identify a digit that is placed at random over a noisy background, that is : finding the target identity. The agent fixates the center of the screen should give an answer about which digit was present in the image. This corresponds to a classic environment control in psychophysic experiments. Different parameters are controlled, such as the target eccentricity, the background noise and the contrast, in order to var the difficulty of the task.

(B) the agent can make a saccade, in which case the center of fixation moves toward the expected location of the target.

(C) The agent subjective perception is shown on the lower right part. The larger the target eccentrity, the more difficult the identifiction. There is a range of eccentricities for wich it is impossible to identify the target from a single glance, so that a sacade is necessary to issue a propoer response.

DONE-Laurent = génére les frames pour un "film"

Methods: What/Where separation

Consider our simplified visual scene containing a non-centered digit over a noisy background. We consider a separate processing of the central part of the visual field and the periphery, corresponding to a central and peripheral processing consistently with information-gain based action selection.

We consider in our setup a slight simplification, that is sampling the prior and the posterior on the true label. The information gain becomes the difference of the future accuracy and the central accuracy. The accuracy here takes the role of a proxy for the posterior entropy. Importantly, the future accuracy is a score that does not predict the future label. It just tells how correct the response will be while doing saccade a.

The separation into current accuracy and future accuracy is reminiscent of the What/where visual processing separation observed in monkeys and humans... with a separate processing of the object detailed shape and identity through the ventral pathway and the visuo-spatial information through the dorsal pathway. Here we interpret the what/where separation in a slightly different manner, with the what devoted to analyzing the central part of the visual field, and the where devoted to choosing the next saccade. The "Where" is not exactly where but rather: where should I look next in order to increase my accuracy?

Methods: Computational Graph

COMPUTATIONAL GRAPH :
Here is the general computational graph of our active vision system.
Two streams of information are separated from the visual primary layers, one stream for processing the central pixels only, the other for processing the periphery with a logpolar encoding. The two streams converge toward a decision layer that compares the central and the peripheral acuracy, in order to decide wether to issue a saccadic or a categorical response. If a saccade is produced, then the center of vision is displaced toward the region that shows the higher accuracy on the accuracy map.

WHAT :
At the core of the vision system is the identification module (the what).   The what pathway is a classic convolutional clasifier.

It shows some translation invariance. It can quantify its uncertainty. It monitors the where pathway.

TODO: mettre le résultat de l'accuracy map pour faire la transition?

WHERE :
Here we make the assumption that the same logpolar compression pattern is conserved from the retina up to the primary motor layers.
**Each possible future saccade has an expected accuracy, that can be trained from the what pathway output**. To accelerate the training, we use a shortcut that is training the network on a translated accuracy map (with logpolar encoding). The ouput is thus a **logpolar accuracy map**, that tells for each possible visuo-motor displacement the value of the future accuracy.
Thus, the saccadic motor ouput (colliculus) shows a similar log-polar compression than the visual input. The saccades are more precise at short than at long distance (and severals accades may be necessary to precisely reach distant targets).

Methods: What

COMPUTATIONAL GRAPH :
Here is the general computational graph of our active vision system.
Two streams of information are separated from the visual primary layers, one stream for processing the central pixels only, the other for processing the periphery with a logpolar encoding. The two streams converge toward a decision layer that compares the central and the peripheral acuracy, in order to decide wether to issue a saccadic or a categorical response. If a saccade is produced, then the center of vision is displaced toward the region that shows the higher accuracy on the accuracy map.

WHAT :
At the core of the vision system is the identification module (the what).   The what pathway is a classic convolutional clasifier.

It shows some translation invariance. It can quantify its uncertainty. It monitors the where pathway.

TODO: mettre le résultat de l'accuracy map pour faire la transition?

WHERE :
Here we make the assumption that the same logpolar compression pattern is conserved from the retina up to the primary motor layers.
**Each possible future saccade has an expected accuracy, that can be trained from the what pathway output**. To accelerate the training, we use a shortcut that is training the network on a translated accuracy map (with logpolar encoding). The ouput is thus a **logpolar accuracy map**, that tells for each possible visuo-motor displacement the value of the future accuracy.
Thus, the saccadic motor ouput (colliculus) shows a similar log-polar compression than the visual input. The saccades are more precise at short than at long distance (and severals accades may be necessary to precisely reach distant targets).

Methods: Where

COMPUTATIONAL GRAPH :
Here is the general computational graph of our active vision system.
Two streams of information are separated from the visual primary layers, one stream for processing the central pixels only, the other for processing the periphery with a logpolar encoding. The two streams converge toward a decision layer that compares the central and the peripheral acuracy, in order to decide wether to issue a saccadic or a categorical response. If a saccade is produced, then the center of vision is displaced toward the region that shows the higher accuracy on the accuracy map.

WHAT :
At the core of the vision system is the identification module (the what).   The what pathway is a classic convolutional clasifier.

It shows some translation invariance. It can quantify its uncertainty. It monitors the where pathway.

TODO: mettre le résultat de l'accuracy map pour faire la transition?

WHERE :
Here we make the assumption that the same logpolar compression pattern is conserved from the retina up to the primary motor layers.
**Each possible future saccade has an expected accuracy, that can be trained from the what pathway output**. To accelerate the training, we use a shortcut that is training the network on a translated accuracy map (with logpolar encoding). The ouput is thus a **logpolar accuracy map**, that tells for each possible visuo-motor displacement the value of the future accuracy.
Thus, the saccadic motor ouput (colliculus) shows a similar log-polar compression than the visual input. The saccades are more precise at short than at long distance (and severals accades may be necessary to precisely reach distant targets).

Outline

Motivation

Methods

Results

Conclusion

Results: success

Results: failure

Results: one saccade

TODO Manu : insérer résultats avec différents contrastes

Results: role of contrast

TODO Manu : insérer résultats avec différents contrastes

Results: more saccades

IG-based selection of action

       done

Outline

Motivation

Methods

Results

Conclusion