Child pages
  • Model of learning and covert attention
Skip to end of metadata
Go to start of metadata

Model of learning and covert attention

The model

The model is a neural network, where learning and selective attention take place simultaneously. Attention focuses not only momentarily to relevant targets, but it also guides learning. Attention emerges from local competition in all parts of the network; the network implements the biased-competition model. The network is hierarchical, with alternating feature expansion and dimension reduction layers. The upper levels develop increasingly global and invariant representations.

The network can be fed with many kind of information (e.g. visual or auditory or linguistic, or all of these). It then learns to represent independent objects or dynamical events in this data, and learns to selectively attend to a limited set of targets at a time.

A schema of the network is illustrated in Figure 1. The whole network consists of smaller networks, which could roughly correspond to the hypercolumns in the cortex. Neurons in one such unit receive the same set of inputs, which they learn to represent. The units can have for example 10-300 neurons. These neurons compete with each other softly. Few neurons with the most excitation stay active after the competition. Contextual connections bias this competition.


Figure 1. Architecture of the model.

The units can receive many kind of contextual inputs (bias):

  • Lateral bias from the neighbouring units leads to Gestalt like continuous representations across different units.
  • Temporal context leads to temporally regular representations.
  • Context from the above layers allow for choosing those features that are relevant for the high-level intentions of the cortex.
  • Long-range connections can mediate context that allows for representing coherent information between modalities. For instance, motor cortex can bias the visual cortex to represent such objects that relate to the current actions of the animal.

The network has dynamics on three different time-scales:

  1. Instantaneously, selective attention focuses on different targets.
  2. Associations between different features are learnt fast. Because of coherent attention, the features that are represented simulateneously usually relate to each other, even if they are in different sensory modalities. Therefore, associations between them can be learnt easily. These associations are then used as one form of context to the competing neural populations.
  3. Slow tuning of features. The bottom-up weights are adapted in Hebbian style. Attention and lateral associations guide this learning to form relevant invariances and features that represent independent components/objects in the world.
    More detailed description of the model can be found from my master's thesis


The experiments I have done with this model are with abstract vectors. I have only used static data. Therefore things such as novelty detection or learning of dynamics or temporal pattern recognition haven't yet been tested. I am now working with real visual data.

Coherent selection of population codes

In this experiment, the selection and information integration capabilities of the network were. The result was that the network is able to select between different population codes. While neurons compete only on local level, there emerges global competition between large population codes. In addition, when a new target wins the competition in some part of the net, it can cause changes in distal parts from that place; the whole network can start to represent the specific targert.

The bottom-up weights were not adapted, only the contextual (association) weights. The network had two distinct "modalities" with one layer, and then one "association area", that received inputs from both of the former layers. The two lower layers could not communicate directly, but only through the association area (first bottom-up connection to the association area, then contextual connections from there to the other modality).

The network was taught with a set of random vectors. While the first half of the vector is fed to the first modality, the second half is fed to the second modality.

After the network learnt these vectors, it was fed with a superposition of two different vectors it had learnt. Let us call these vector "dog" and "cat". The first modality was fed with a sum of dog and a cat, but on the other modality the sum was weighted. For example, the second modality could receive input that is 0.2 * dog + 0.8 * cat. The mixture ratio of cat and dog to the second modality was altered.

Figure 2 shows the results. The second modality represent both cat and a dog as population vectors. There is soft selection, though. Around mixture ratio 0.5, the representation strengths vary more steeply than the inputs. When the association area receives this information, it performs even more selection. Finally, this information travels to the first modality through contextual connections, where it causes selection between equally stong inputs.

Figure 2. The highest figure depicts the representation strenghts in the association area. The lowest figures depict the representation (solid lines) and input (dashed lines) strengths of the two modalities. On y-axis is the strength and on x-axis is the mixture ratio of two different input vectors to the second modality.

Learning invariances

In this experiment, the bottom-up weights were adaptive, too. The network had six layers, with alternating feature expansion (simple cell) and dimension reduction (complex cell) layers. The data consisted of 204-dimensional vectors, which were generated randomly from 5 different object clusters. Each object had hundreds of thousands of different instantiations. The network learnt to represent the object clusters invariantly with respect to the actual appearance of the object, although it was not given the class information as a teaching signal.

More detailed description can be found in my thesis

Jumping attention

In this experiment, the same network as in the invariance experiment was used. This time, the association weight synapses were adaptive, such that they lose strength temporarily if they are on. This causes the population codes to lose competition after they have been represented a while, and attention will jump to a new target.

Figure 3. Strength of representation of different objects as a function of time. The left figure is with ambiguous input. The right figure is with only one object as input (the "leg" object).

  • No labels


  1. Anonymous

    Handheld consoles

    Coach Outlet Online

    may flank the electronic

    Coach Outlet

    stores in this

    Burberry Outlet

    millennium but if you

    Louis Vuitton Bags

    sat patiently and

    Louis Vuitton Purses

    waited for your

    Coach Outlet

    turn to snag a duck

    Coach Outlet Store

    on your television set instead

    Coach Outlet Store Online

    of giving Barbie a new hairdo

    Coach Factory Outlet

    then this look is for you. Ripped jean

    Coach Factory Outlet

    shorts and a DIY tie

    Coach Outlet Online

    back vintage are perfect attire

    Coach Factory Online

    for Saturday hangouts. Pair with classic

    Coach Factory Online

    Chuck Taylor’s and

    Coach Outlet

    multicolored, knee high, knit socks

    Coach Outlet Online

    to neaten the look. Pile on the plastic

    Coach Outlet Store Online

    bangles and add some neon to your nails to complete.

  2. Anonymous

    The broad

    Coach Outlet Store Online

    rally started last Friday, after President Obama

    Coach Outlet Online

    and congressional leaders wrapped up their first fiscal cliff

    Coach Outlet

    meeting since the election. Investors were encouraged after both Democrats

    Gucci Belt

    and Republicans emerged from the meeting with a confident tone and

    Coach Factory Outlet

    suggested that agreement could be reached before Jan. 1, when

    Coach Factory Outlet

    federal spending cuts and tax increases are scheduled

    Gucci Belts

    to take effect. "The question of whether or not the fiscal cliff

    Coach Factory Outlet

    will be avoided is impossible to answer with any

    Coach Factory Outlet

    degree of certainty, but the political reality does indicate

    Coach Outlet Online

    that some sort of compromise may be possible," said Bob Doll, senior advisor at BlackRock. "The two sides still

    Coach Online Outlet

    appear to be quite far apart on some important issues, particularly

    Coach Factory Outlet Online

    regarding a potential tax increase for higher-income Americans. But the parties are at least attempting

    Coach Outlet Online

    to avoid brinksmanship."While the rhetoric out of Washington is slightly

    Coach Outlet Online

    more positive, experts think stocks will remain under pressure and trading will likely

    Hermes Belts

    be choppy until an actual deal is finalized, which is still likely

    Coach Factory Online

    weeks away.

  3. with 123 nations entering in 2012

    Coach Outlet Online

    said to separate right-wing candidate Jean-Francois Cope and

    Coach Outlet Store Online

    ex-PM Francois Fillon.As the neck-and-neck result emerged, supporters of

    Coach Outlet

    both UMP candidates made accusations of vote fraud.The party has been without

    Coach Outlet

    a leader since Nicolas Sarkozy was defeated in the May presidential election.Mr

    Coach Factory Online

    Cope has been secretary general of the conservative UMP since 2010 while

    Coach Factory Outlet

    his rival was prime minister for five years during Mr Sarkozy's presidency

    Coach Factory Outlet

    .More than 50% of the party's 300,000 members took part in the vote, according

    Coach Factory Outlet

    to early indications, a far higher turnout than expected.When initial

    Coach Outlet Store

    results emerged, Mr Cope was narrowly in the lead, surprising political

    Coach Outlet Online

    J in which he claimed that gangs in the banlieues were fostering "anti-white racism s

    Coach Outlet Store Online

  4. hello i am Topiclaw from panamax company, today i am very happy to tell you about us. And i'm here: Nguyen Vu. We are the cong ty luat biggest in Viet Nam and we have some infomation about this such as: cong bo thuc pham, cong bo thuc pham chuc nang, cong bo thuoc, cong bo ruou, cong bo banh keo, dich vu thanh lap cong ty. We have forum and blog if you love to visit us and we have hot topic such as: cong bo banh keo, dang ky logo cong ty, thu tuc lam ho khau ha noi, quyen han cua giam doc, mang trinh bi rach. And thanks so much for reading my info ! love to see all at vncart.

  5. Anonymous

    This is highly informatics, crisp and clear. I think that Everything has been described in systematic manner so that reader could get maximum information and learn many things


    free term papers psychology

    download ebook free

    drama korea