m8ta
use https for features.
text: sort by
tags: modified
type: chronology
[0] Song S, Consciousness and the consolidation of motor learning.Behav Brain Res 196:2, 180-6 (2009 Jan 23)

{1440}
hide / / print
ref: -2017 tags: attention transformer language model youtube google tech talk date: 02-26-2019 20:28 gmt revision:3 [2] [1] [0] [head]

Attention is all you need

  • Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
  • Attention is all you need neural network models
  • Good summary, along with: The Illustrated Transformer (please refer to this!)
  • Ɓukasz Kaiser mentions a few times how fragile the network is -- how easy it is to make something that doesn't train at all, or how many tricks by google experts were needed to make things work properly. it might be bravado or bluffing, but this is arguably not the way that biology fails.
  • Encoding:
  • Input is words encoded as 512-length vectors.
  • Vectors are transformed into length 64 vectors: query, key and value via differentiable weight matrices.
  • Attention is computed as the dot-product of the query (current input word) with the keys (values of the other words).
    • This value is scaled and passed through a softmax function to result in one attentional signal scaling the value.
  • Multiple heads' output are concatenated together, and this output is passed through a final weight matrix to produce a final value for the next layer.
    • So, attention in this respect looks like a conditional gain field.
  • 'Final value' above is then passed through a single layer feedforward net, with resnet style jump.
  • Decoding:
  • Use the attentional key value from the encoder to determine the first word through the output encoding (?) Not clear.
  • Subsequent causal decodes depend on the already 'spoken' words, plus the key-values from the encoder.
  • Output is a one-hot softmax layer from a feedforward layer; the sum total is differentiable from input to output using cross-entropy loss or KL divergence.

{917}
hide / / print
ref: Doty-1969.01 tags: Doty microstimulation brain behavior macaque conditioned stimulus attention motivation 1969 date: 12-29-2011 23:28 gmt revision:8 [7] [6] [5] [4] [3] [2] [head]

PMID-4888623[0] Electrical stimulation of the brain in behavioral context.

  • Excellent review.
  • Focal stimulation of macaques can induce insect-grabbing responses, after which they will carefully examine their hands to see what was caught!
    • Same thing has been observed in humans -- the patient reported that he wanted to catch 'that butterfly'.
  • Such complicated action must be the effect of downstream / upstream targets of the stimulated site, as the actual stimulation carries no information other than it's spatial locality within the brain.
  • Stimulation of the rostral thalamus in the language hemisphere can elicit phrases: "Now one goes home", "Thank you", "I see something".
    • These are muttered involuntarily and without recollection of having been spoken.
  • Doty stimulated macaques at 20ua for 500us as a CS in postcentral gyrus (S1?) for a lever press CR, which should (he says)only activate a few dozen neurons.
  • Can elicit mating behaviors in oposums with electrical stimulation of the hypothalamus, but only if another opossum or furry object is present.
  • Stimulation of the caudate nucleus in humans causes an arrest reaction: they may speak, smile, or laught inappropriately, but appropriate voluntary responses are brought to a halt.
  • Stimulation of the basolateral amygdala can cause:
    • Hungry cats to immediately stop eating
    • Stop stalking prey
    • Non-hunting animals to stalk prey, and indeed will solve problems to gain access to rats which can be attacked.
  • Prolonged stimulation of almost every place in the brain of a cat at 3-8Hz can put it to sleep, though since lab cats normally sleep 17/24hours, this result may not be significant.
  • Stimulation at most sites in the limbic system has the still mysterious ability to organize motor activity in any fashion required to produce more of the activity or to avoid it, as the case may be.
  • Rats that are stimulated in the periaqueductal gray will self-administer stimulation, but will squeal and otherwise indicate pain and fright during the stimulation. Increasing the duration of stimulation from 0.5 to 1 second makes self-administration of this apparently fearful stimulation stop in both rats and cats.
  • Certain patterns of activity within systems responsible for fearful or aggressive behavior, rather than being aversive are perversely gratifying. This is clearly recognized in the sociology of man...
  • Rats will self-stimulate with the same stimulus trains that will cause them to eat and drink, and under some conditions the self-stimulation occurs only if food or water is available.
  • On the other hand, rats will choose self-stimulation of the lateral hypothalamus instead of food, even when they are starving.
    • Electrically induced hunger is its own reward.
  • The work of Loucks (124, 125) forms the major point of origin for the concept that motivation is essential to learning. with careful and thorough training, Loucks was unable to form CRs to an auditory CS using stimulation of the motor cortex as the US. With this paradigm, the limb movements elicited by the US never appeared to the CS alone; but movements were readily established when each CS-US combination was immediately followed by the presentation of food.
    • However: Kupalov independently proved that stimulation of the motor cortex could be used as the US, at the same time using stimulation at other loci as the CS.
    • Why the difference? Attention -- failures are commonly obtained with animals that consistenly fidget or fight restraint, as most of them do.
    • Cortical stimulation itself is not rewarding or aversive; animals neither seek nor avoid stimulation of most neocortical areas.
  • On classical conditioning: [Bures and colleagues (20, 65) bibtex:Bures-1968 bibtex:Gerbrandt-1968] found that if an anticedent stimulus, which might or might not effect a neuron, were consistently followed by effective intracellular electrical stimulation of that individual neuron, in roughly 10 percent of the cells of the neocortex, hippocampus, thalamus, or mesencephalic reticular formation a change in the response of that cell to the antecedent stimulus could be observed.
  • With an apparent exception of the cerebellum it is possible to electrical excitation any place in the brain as a CS in chickens, rats, rabbits ...
  • Stimulation of group 1 proprioceptive muscle-afferent fibers in cats is ineffective as a CS.
    • Muscle spindles lack clear access to the systems subserving conditioned reflexes. (These instead go to the cerebellum)
  • Macaques can also discriminate between two stimulation sites 1-3 mm apart apparently over the entirety of the cortex, at frequencies between 2 and 100Hz, and over a 4-10fold range of currents.
  • In human cases where electrical stimulation or the cortex elicits specific memories, extirpation of the stimulated area does not effect recall of this memory (156) {973}.

____References____

[0] Doty RW, Electrical stimulation of the brain in behavioral context.Annu Rev Psychol 20no Issue 289-320 (1969)

{199}
hide / / print
ref: Lin-2006.12 tags: nucleus_basalis GABA ACh attention basal_forebrain sleep date: 12-07-2011 03:51 gmt revision:5 [4] [3] [2] [1] [0] [head]

PMID-16928796[0] Fast modulation of prefrontal cortex activity by basal forebrain noncholinergic neuronal ensembles

in the author's own words:

  • in the intro sections, you can find the summary background info you need, both anatomical and functional. Despite the fact that most people think of this as solely the cholinergic projection system, my data is pointing to a very important role for the non-ACh projection system (most likely GABAergic!) in fast cortical modulation and ATTENTION. The relevant thing for you here is that, when people stimulated nucleus basalis and claimed the effect to be cholinergic, I believe most stimulation protocols (short bursts) are in fact mimicking the natural activity pattern of non-ACh projection system, and therefore should be re-interpreted with caution.
  • the intro, as promised, is concise, relevant, and has a lot of references.
  • key hypothesis is that the BF has GABA projections onto GABAergic interneurons in the PFC
    • typically, people focus on ACh projections.. perhaps as a matter of tradition?
    • PFC is reciprocally connected to the BF
  • secondary thing to test was the difference in behavior of the basal-forebrain tonic neurons (BFTN) between sleep and wake states.

____References____

[0] Lin SC, Gervasoni D, Nicolelis MA, Fast modulation of prefrontal cortex activity by basal forebrain noncholinergic neuronal ensembles.J Neurophysiol 96:6, 3209-19 (2006 Dec)

{767}
hide / / print
ref: life-0 tags: attention economy papers authors GOSH date: 07-21-2009 22:45 gmt revision:18 [17] [16] [15] [14] [13] [12] [head]

to read!

  • Jonathan Crary __Suspensions of Perception: Attention, Spectacle, and Modern Culture___
  • Catherine Hales - ___How we became post-human___

to think!

  • DIWO - do it with others
  • openFrameworks - on the iphone! check the demos - so creative, so cool, so fun.
  • Art as research for the future of society and interfaces and life etc.
  • Ars Electronica - superdelux new building with LED facade.
  • POV tracking even with simple / low cost USB cameras. This allows video overlays.
  • Tempt graffiti artist - using eye tracking to enable a graffiti artist to draw again.
  • Inspiring the sense of wonder in children. 9
  • The west is planning ourselves into organized oblivion. Brazil has 600 ponto cultural(s) because "it is disorganized" - there are not really very many limitations; not very much bureaucracy; tthey are open to any and all new ideas for distribution; the policy response is organic but 'flaky'. Brazil may be pedagology more nimble than the west?
  • Open source as ultimately lowering the cost of entry / barrier of entry.
  • Art as a means of getting people interested in their world, interested in creating things, interested in taking things apart (deconstruction seems very relevant to western art)
  • TopoR - not really realated, but would be nice to have this in kicadocaml.
  • Open source needs not only coders, but also the support personnel - the equivalent of all the non-coders in a software corporation. These people's contributions can be as important as the coders'. (e.g. translators).
  • the mesh potato - Seems that many have the same idea at the same time.
  • Via Artigo - very small x86 motherboard / full computer solution. 1.0Ghz processor, video out, ethernet, usb, 2.5" hdd, all in a 5.25" (CD-ROM sized) case. Via Chris Csikszentmihalyi.
  • Goal-direction is best left to partially subconscious control (?) (at least according to some artists at the GOSH conference - but maybe that works best for artists?)
  • Programming languages need to be a good compromise between accurate / terse representation of algorithms & data structures (graphical programming languages are bad at representing complicated datastructures, which is why I don't like them), and understanding / exploiting information-processing strengths of the human brain (e.g. visual, linguistic). I actually find that some of the fold-left & fold-right reduce operators in Ocaml are rather non-obvious to use. But maybe I should just study harder :-)

to make!

to program!

{671}
hide / / print
ref: Song-2009.01 tags: sleep motor learning consolidation attention date: 02-18-2009 17:28 gmt revision:1 [0] [head]

PMID-18951924[0] Consciousness and the consolidation of motor learning

  • Not all consolidation occurs during sleep; in some instances consolidation only occurs during the day; in other times, neither daytime or sleep consolidates a memory.
  • Attention is an important factor that may determine if sleep or daytime replay plays a role in consolidation.
  • In a tapping task, after a night of sleep performance is faster and more accurrate. Without the sleep, but with the same 12-hour interval, the same improvement is absent.
  • Evidence suggests though we experience the sensation of 'voluntary' movement, the conscious wish to move is more an afterthought than the cause.
    • Source: Libet et al 1983. (Subjects could accurately time events, and reported that the will to move preceded actual movement. However, the cortical potentials associated with movement preceded conscious awareness).
    • nonetheless, studies indicate that conscious awareness can affect movements, and how they are consolidated.
  • people with no declarative memory (like HR) can still remember procedural skills.
  • Consolidation = the process by which a fragile memory acquired via practice or exposure is consolidated into a more permanent, stable long-term form. If it occurs in the hours after practice, then it is 'off-line'; likewise for sleep.
    • Consolidation also includes stabilization, or making the memories robust to interference from new memories (retroactive interference).
    • This seems to be dependent on sleep, specifically NREM.
    • In studies where attention was broken using a tone counting task, neither over-night nor over-day enhancements were found to occur for motor sequence learning.
    • Another interesting effect is the development of explicit memory over the course of a night's sleep. Sleep seems to encourage conscious awareness of implicit patterns. -- probably through replay and integration.
  • Regarding "thinking too much" about sports: "As in the studies cited above, motor learning may initially rely on more explicit and prefrontal areas, but after extended practice and expertise, shift to more dorsal areas, but thinking about the movement can shift activity back to the less skilled explicit areas. Although many explanations may be derived, one could argue that these athletes show that even when years of practice has given the implicit system an exquisitely fine tuned memory for a movement, the explicit system can interfere at the time of performance and erase all evidence of implicit memory."
  • Well-written throughout, especially the conclusion paragraph.

____References____