m8ta
You are not authenticated, login. |
|
{1571} | |||||||||
One model for the learning of language
A more interesting result is Deep symbolic regression for recurrent sequences, where the authors (facebook/meta) use a Transformer -- in this case, directly taken from Vaswini 2017 (8-head, 8-layer QKV w/ a latent dimension of 512) to do both symbolic (estimate the algebraic recurrence relation) and numeric (estimate the rest of the sequence) training / evaluation. Symbolic regression generalizes better, unsurprisingly. But both can be made to work even in the presence of (log-scaled) noise! While the language learning paper shows that small generative programs can be inferred from a few samples, the Meta symbolic regression shows that Transformers can evince either amortized memory (less likely) or algorithms for perception -- both new and interesting. It suggests that 'even' abstract symbolic learning tasks are sufficiently decomposable that the sorts of algorithms available to an 8-layer transformer can give a useful search heuristic. (N.B. That the transformer doesn't spit out perfect symbolic or numerical results directly -- it also needs post-processing search. Also, the transformer algorithm has search (in the form of softmax) baked in to it's architecture.) This is not a light architecture: they trained the transformer for 250 epochs, where each epoch was 5M equations in batches of 512. Each epoch took 1 hour on 16 Volta GPUs w 32GB of memory. So, 4k GPU-hours x ~10 TFlops = 1.4e20 Flops. Compare this with grammar learning above; 7 days on 32 cores operating at ~ 3Gops/sec is 1.8e15 ops. Much, much smaller compute. All of this is to suggest a central theme of computer science: a continuum between search and memorization.
Most interesting for a visual neuroscientist (not that I'm one per se, but bear with me) is where on these axes (search, heuristic, memory) visual perception is. Clearly there is a high degree of recurrence, and a high degree of plasticity / learning. But is there search or local optimization? Is this coupled to the recurrence via some form of energy-minimizing system? Is recurrence approximating E-M? | |||||||||
{331} | |||||||||
PMID-17021028[0] Correlations Between the Same Motor Cortex Cells and Arm Muscles During a Trained Task, Free Behavior, and Natural Sleep in the Macaque Monkey
____References____ | |||||||||
{1446} | |||||||||
PMID-29074582 A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAs
| |||||||||
{1213} | |||||||||
PMID-20153370[0] A bio-friendly and economical technique for chronic implantation of multiple microelectrode arrays ____References____
| |||||||||
{994} | |||||||||
PMID-8351520[0] Dynamics of the hippocampal ensemble code for space.
PMID-8036517[1] Reactivation of hippocampal ensemble memories during sleep.
____References____
| |||||||||
{199} | |||||||||
PMID-16928796[0] Fast modulation of prefrontal cortex activity by basal forebrain noncholinergic neuronal ensembles in the author's own words:
____References____
| |||||||||
{702} | |||||||||
PMID-8670641[0] The hippocampo-neocortical dialogue.
| |||||||||
{720} | |||||||||
http://www.the-scientist.com/2009/04/1/34/1/ -- good layperson-level review of the present research on sleep. Includes interviews with Strickgold and other prominents. References:
http://www.the-scientist.com/2009/04/1/15/1/ -- points out that Western sleep style is a relative outlier compared to sleeping in other cultures. More 'primitive' cultures have polyphasic sleep, with different stages of alertness, dozing, napping, disengaged, vigilance, etc.
____References____
| |||||||||
{716} |
ref: Ribeiro-2004.12
tags: Sidarta Ribeiro reverberation sleep consolidation integration replay REM SWS
date: 03-26-2009 03:19 gmt
revision:2
[1] [0] [head]
|
||||||||
PMID-15576886[0] Reverberation, storage, and postsynaptic propagation of memories during sleep
____References____
| |||||||||
{713} | |||||||||
PMID-11250009[0] Sleep and memory: a molecular perspective.
____References____
| |||||||||
{712} | |||||||||
PMID-19245368[0] The influence of learning on sleep slow oscillations and associated spindles and ripples in humans and rats
____References____
| |||||||||
{685} |
ref: BrashersKrug-1996.07
tags: motor learning sleep offline consolidation Bizzi Shadmehr
date: 03-24-2009 15:39 gmt
revision:1
[0] [head]
|
||||||||
PMID-8717039[0] Consolidation in human motor memory.
____References____
| |||||||||
{691} | |||||||||
PMID-18835655[0] Sleep-related improvements in motor learning following mental practice.
____References____
| |||||||||
{701} | |||||||||
PMID-18836440[0] Pharmacological REM sleep suppression paradoxically improves rather than impairs skill memory
____References____
| |||||||||
{708} |
ref: Wagner-2004.01
tags: sleep insight mental restructure integration synthesis consolidation
date: 03-20-2009 21:31 gmt
revision:1
[0] [head]
|
||||||||
PMID-14737168[0] Sleep Inspires Insight.
____References____
| |||||||||
{707} | |||||||||
PMID-11691982[0] The Role of Sleep in Learning and Memory
____References____
| |||||||||
{705} |
ref: Tononi-2006.02
tags: sleep synaptic homeostasis plasticity
date: 03-20-2009 15:45 gmt
revision:1
[0] [head]
|
||||||||
PMID-16376591[0] Sleep function and synaptic homeostasis.
____References____
| |||||||||
{703} | |||||||||
PMID-17167082[0] Elevated sleep spindle density after learning or after retrieval in rats.
____References____
| |||||||||
{700} | |||||||||
PMID-11691983[0] Sleep, Learning, and Dreams: Off-line Memory Reprocessing
____References____
| |||||||||
{692} | |||||||||
PMID-17189946[0] Cortico-hippocampal interaction during up-down states and memory consolidation.
____References____ | |||||||||
{682} | |||||||||
PMID-17173043[0] Coordinated memory replay in the visual cortex and hippocampus during sleep.
____References____ | |||||||||
{680} | |||||||||
PMID-17406665[0] Daytime naps, motor memory consolidation and regionally specific sleep spindles.
____References____ | |||||||||
{679} | |||||||||
PMID-18274267[0] Fast sleep spindle (13-15 hz) activity correlates with sleep-dependent improvement in visuomotor performance.
____References____ | |||||||||
{672} | |||||||||
PMID-18714787[0] Motor sequence learning increases sleep spindles and fast frequencies in post-training sleep.
____References____ | |||||||||
{671} | |||||||||
PMID-18951924[0] Consciousness and the consolidation of motor learning
____References____ | |||||||||
{678} |
ref: Rasch-2009.06
tags: sleep cholinergic acetylcholine REM motor consolidation
date: 02-18-2009 17:27 gmt
revision:0
[head]
|
||||||||
PMID-19194375[0] "Impaired Off-Line Consolidation of Motor Memories After Combined Blockade of Cholinergic Receptors During REM Sleep-Rich Sleep."
____References____ | |||||||||
{669} | |||||||||
PMID-19191602 A New Hypothesis for Sleep: Tuning for Criticality.
| |||||||||
{620} | |||||||||
PMID-17035544[0] Dopaminergic control of sleep-wake states ____References____ | |||||||||
{243} | |||||||||
PMID-17035544 Dopaminergic control of sleep-wake states.
| |||||||||
{167} |
ref: GarciaRill-1991.01
tags: PPN pedunculopontine nucleus brainstem sleep locomotion consciousness 1991
date: 0-0-2007 0:0
revision:0
[head]
|
||||||||
PMID-1887068 The Pedunculopontine nucleus
|