m8ta
You are not authenticated, login. |
|
{1576} |
ref: -0
tags: GFlowNet Bengio probabilty modelling reinforcement learing
date: 10-29-2023 19:17 gmt
revision:3
[2] [1] [0] [head]
|
||||
| |||||
{842} | |||||
Distilling free-form natural laws from experimental data
Since his Phd, Michael Schmidt has gone on to found Nutonian, which produced Eurequa software, apparently without dramatic new features other than being able to use the cloud for equation search. (Probably he improved many other detailed facets of the software..). Nutonian received $4M in seed funding, according to Crunchbase. In 2017, Nutonian was acquired by Data Robot (for an undisclosed amount), where Michael has worked since, rising to the title of CTO. Always interesting to follow up on the authors of these classic papers! | |||||
{1556} |
ref: -0
tags: concept net NLP transformers graph representation knowledge
date: 11-04-2021 17:48 gmt
revision:0
[head]
|
||||
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models
Human-designed knowledge graphs are described here: ConceptNet 5.5: An Open Multilingual Graph of General Knowledge And employed for profit here: https://www.luminoso.com/ | |||||
{1545} | |||||
Self-organizaton in a perceptual network
One may critically challenge the infomax idea: we very much need to (and do) throw away spurious or irrelevant information in our sensory streams; what upper layers 'care about' when making decisions is certainly relevant to the lower layers. This credit-assignment is neatly solved by backprop, and there are a number 'biologically plausible' means of performing it, but both this and infomax are maybe avoiding the problem. What might the upper layers really care about? Likely 'care about' is an emergent property of the interacting local learning rules and network structure. Can you search directly in these domains, within biological limits, and motivated by statistical reality, to find unsupervised-learning networks? You'll still need a way to rank the networks, hence an objective 'care about' function. Sigh. Either way, I don't per se put a lot of weight in the infomax principle. It could be useful, but is only part of the story. Otherwise Linsker's discussion is accessible, lucid, and prescient. Lol. | |||||
{1543} |
ref: -2019
tags: backprop neural networks deep learning coordinate descent alternating minimization
date: 07-21-2021 03:07 gmt
revision:1
[0] [head]
|
||||
Beyond Backprop: Online Alternating Minimization with Auxiliary Variables
This is interesting in that the weight updates can be cone in parallel - perhaps more efficient - but you are still propagating errors backward, albeit via optimizing 'codes'. Given the vast infractructure devoted to auto-diff + backprop, I can't see this being adopted broadly. That said, the idea of alternating minimization (which is used eg for EM clustering) is powerful, and this paper does describe (though I didn't read it) how there are guarantees on the convexity of the alternating minimization. Likewise, the authors show how to improve the performance of the online / minibatch algorithm by keeping around memory variables, in the form of covariance matrices. | |||||
{1534} | |||||
Going in circles is the way forward: the role of recurrence in visual inference I think the best part of this article are the references -- a nicely complete listing of, well, the current opinion in Neurobiology! (Note that this issue is edited by our own Karel Svoboda, hence there are a good number of Janelians in the author list..) The gestalt of the review is that deep neural networks need to be recurrent, not purely feed-forward. This results in savings in overall network size, and increase in the achievable computational complexity, perhaps via the incorporation of priors and temporal-spatial information. All this again makes perfect sense and matches my sense of prevailing opinion. Of course, we are left wanting more: all this recurrence ought to be structured in some way. To me, a rather naive way of thinking about it is that feed-forward layers cause weak activations, which are 'amplified' or 'selected for' in downstream neurons. These neurons proximally code for 'causes' or local reasons, based on the supported hypothesis that the brain has a good temporal-spatial model of the visuo-motor world. The causes then can either explain away the visual input, leading to balanced E-I, or fail to explain it, in which the excess activity is either rectified by engaging more circuits or engaging synaptic plasticity. A critical part of this hypothesis is some degree of binding / disentanglement / spatio-temporal re-assignment. While not all models of computation require registers / variables -- RNNs are Turning-complete, e.g., I remain stuck on the idea that, to explain phenomenological experience and practical cognition, the brain much have some means of 'binding'. A reasonable place to look is the apical tuft dendrites, which are capable of storing temporary state (calcium spikes, NMDA spikes), undergo rapid synaptic plasticity, and are so dense that they can reasonably store the outer-product space of binding. There is mounting evidence for apical tufts working independently / in parallel is investigations of high-gamma in ECoG: PMID-32851172 Dissociation of broadband high-frequency activity and neuronal firing in the neocortex. "High gamma" shows little correlation with MUA when you differentiate early-deep and late-superficial responses, "consistent with the view it reflects dendritic processing separable from local neuronal firing" | |||||
{1530} |
ref: -2017
tags: deep neuroevolution jeff clune Uber genetic algorithms
date: 02-18-2021 18:27 gmt
revision:1
[0] [head]
|
||||
Deep Neuroevolution: genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning* Uber AI labs; Jeff Clune.
The result is indeed surprising, but it also feels lazy -- the total effort or information that they put into writing the actual algorithm is small; as mentioned in the introduction, this is a case of old algorithms with modern levels of compute. Analogously, compare Go-Explore, also by Uber AI labs, vs Agent57 by DeepMind; the Agent57 paper blithely dismisses the otherwise breathless Go-Explore result as feature engineering and unrealistic free backtracking / game-resetting (which is true..) It's strange that they did not incorporate crossover aka recombination, as David MacKay clearly shows that recombination allows for much higher mutation rates and much better transmission of information through a population. (Chapter 'Why have sex'). They also perhaps more reasonably omit developmental encoding, where network weights are tied or controlled through development, again in an analogy to biology. A better solution, as they point out, would be some sort of hybrid GA / ES / A3C system which used both gradient-based tuning, random stochastic gradient-based exploration, and straight genetic optimization, possibly all in parallel, with global selection as the umbrella. They mention this, but to my current knowledge this has not been done. | |||||
{1522} | |||||
Schema networks: zero-shot transfer with a generative causal model of intuitive physics
| |||||
{1517} | |||||
PMID-26621426 Causal Inference and Explaining Away in a Spiking Network
| |||||
{1516} | |||||
Inductive representation learning on large graphs
This is a well-put together paper, with some proofs of convergence etc -- but it still feels only lightly tested. As with many of these papers, could benefit from a positive control, where the generating function is known & you can see how well the algorithm discovers it. Otherwise, the structure / algorithm feels rather intuitive; surprising to me that it was not developed before the matrix factorization methods. Worth comparing this to word2vec embeddings, where local words are used to predict the current word & the resulting vector in the neck-down of the NN is the representation. | |||||
{1511} | |||||
| |||||
{1507} | |||||
| |||||
{1428} | |||||
PMID-30420685 Fast in-vivo voltage imaging using a red fluorescent indicator
| |||||
{1492} | |||||
PMID: Spiking neurons can discover predictive features by aggregate-label learning
Editorializing a bit: I said this was interesting, but why? The first part of the paper is another form of SGD, albeit in a spiking neural network, where the gradient is harder compute hence is done numerically. It's the aggregate part that is new -- pulling in repeated patterns through synaptic learning rules. Of course, to do this, the full trace of pre and post synaptic activity must be recorded (??) for estimating the STS (i think). An eligibility trace moves in the right direction as a biologically plausible approximation, but as always nothing matches the precision of SGD. Can the eligibility trace be amended with e.g. neuromodulators to push the performance near that of SGD? The next step of adding self supervised singular and multiple features is perhaps toward the way the brain organizes itself -- small local feedback loops. These features annotate repeated occurrences of stimuli, or tile a continuous feature space. Still, the fact that I haven't seen any follow-up work is suggestive... Editorializing further, there is a limited quantity of work that a single human can do. In this paper, it's a great deal of work, no doubt, and the author offers some good intuitions for the design decisions. Yet still, the total complexity that even a very determined individual can amass is limited, and likely far below the structural complexity of a mammalian brain. This implies that inference either must be distributed and compositional (the normal path of science), or the process of evaluating & constraining models must be significantly accelerated. This later option is appealing, as current progress in neuroscience seems highly technology limited -- old results become less meaningful when the next wave of measurement tools comes around, irrespective of how much work went into it. (Though: the impedtus for measuring a particular thing in biology is only discovered through these 'less meaningful' studies...). A third option, perhaps one which many theoretical neuroscientists believe in, is that there are some broader, physics-level organizing principles to the brain. Karl Friston's free energy principle is a good example of this. Perhaps at a meta level some organizing theory can be found, or likely a set of theories; but IMHO, you'll need at least one theory per brain area, at least, just the same as each area is morphologically, cytoarchitecturaly, and topologically distinct. (There may be only a few theories of the cortex, despite all the areas, which is why so many are eager to investigate it!) So what constitutes a theory? Well, you have to meaningfully describe what a brain region does. (Why is almost as important; how more important to the path there.) From a sensory standpoint: what information is stored? What processing gain is enacted? How does the stored information impress itself on behavior? From a motor standpoint: how are goals selected? How are the behavioral segments to attain them sequenced? Is the goal / behavior even a reasonable way of factoring the problem? Our dual problem, building the bridge from the other direction, is perhaps easier. Or it could be a lot more money has gone into it. Either way, much progress has been made in AI. One arm is deep function approximation / database compression for fast and organized indexing, aka deep learning. Many people are thinking about that; no need to add to the pile; anyway, as OpenAI has proven, the common solution to many problems is to simply throw more compute at it. A second is deep reinforcement learning, which is hideously sample and path inefficient, hence ripe for improvement. One side is motor: rather than indexing raw motor variables (LRUD in a video game, or joint torques with a robot..) you can index motor primitives, perhaps hierarchically built; likewise, for the sensory input, the model needs to infer structure about the world. This inference should decompose overwhelming sensory experience into navigable causes ... But how can we do this decomposition? The cortex is more than adept at it, but now we're at the original problem, one that the paper above purports to make a stab at. | |||||
{1485} | |||||
PMID-26352471 Labelling and optical erasure of synaptic memory traces in the motor cortex
| |||||
{1475} |
ref: -2017
tags: two photon holographic imaging Arch optogenetics GCaMP6
date: 09-12-2019 19:24 gmt
revision:1
[0] [head]
|
||||
PMID-28053310 Simultaneous high-speed imaging and optogenetic inhibition in the intact mouse brain.
| |||||
{1418} |
ref: -0
tags: nanophotonics interferometry neural network mach zehnder interferometer optics
date: 06-13-2019 21:55 gmt
revision:3
[2] [1] [0] [head]
|
||||
Deep Learning with Coherent Nanophotonic Circuits
| |||||
{1463} | |||||
All-optical spiking neurosynaptic networks with self-learning capabilities
| |||||
{1460} | |||||
A friend postulated using the triplet state phosphorescence as a magnetically-modulatable dye. E.g. magnetically slice a scattering biological sample, rather than slicing optically (light sheet, 2p) or mechanically. After a little digging:
I'd imagine that it should be possible to design a molecule -- a protein cage, perhaps a (fully unsaturated) terpine -- which isolates the excited state from oxygen quenching. Adam Cohen at Harvard has been working a bit on this very idea, albeit with fluorescence not phosphorescence --
Yet! Magnetic field effects do exist in solution:
| |||||
{1456} | |||||
PMID-21360044 Robust penetrating microelectrodes for neural interfaces realized by titanium micromachining
| |||||
{1446} | |||||
PMID-29074582 A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAs
| |||||
{1439} |
ref: -2006
tags: hinton contrastive divergence deep belief nets
date: 02-20-2019 02:38 gmt
revision:0
[head]
|
||||
PMID-16764513 A fast learning algorithm for deep belief nets.
| |||||
{1434} | |||||
Audio AI: isolating vocals from stereo music using Convolutional Neural Networks
| |||||
{1426} | |||||
Training neural networks with local error signals
| |||||
{1419} | |||||
All-optical machine learning using diffractive deep neural networks
| |||||
{1174} | |||||
Brains, sex, and machine learning -- Hinton google tech talk.
| |||||
{1425} | |||||
PMID-29375323 Fear learning regulates cortical sensory representation by suppressing habituation
| |||||
{1409} | |||||
Coevolution of Fitness Predictors
| |||||
{1408} | |||||
LDMNet: Low dimensional manifold regularized neural nets.
| |||||
{1384} | |||||
PMID-28246640 Ultraflexible nanoelectronic probes form reliable, glial scar–free neural integration
| |||||
{1236} | |||||
PMID-23580530 Injectable, cellular-scale optoelectronics with applications for wireless optogenetics.
| |||||
{1391} | |||||
Evolutionary Plasticity and Innovations in Complex Metabolic Reaction Networks
Summary thoughts: This is a highly interesting study, insofar that the authors show substantial support for their hypotheses that phenotypes can be explored through random-walk non-lethal mutations of the genotype, and this is somewhat invariant to the source of carbon for known biochemical reactions. What gives me pause is the use of linear programming / optimization when setting the relative concentrations of biomolecules, and the permissive criteria for accepting these networks; real life (I would imagine) is far more constrained. Relative and absolute concentrations matter. Still, the study does reflect some robustness. I suggest that a good control would be to ‘fuzz’ the list of available reactions based on statistical criteria, and see if the results still hold. Then, go back and make the reactions un-biological or less networked, and see if this destroys the measured degrees of robustness. | |||||
{1354} |
ref: -0
tags: David Kleinfeld penetrating arterioles perfusion cortex vasculature
date: 10-17-2016 23:24 gmt
revision:1
[0] [head]
|
||||
PMID-17190804 Penetrating arterioles are a bottleneck in the perfusion of neocortex.
| |||||
{1348} | |||||
Heller Lecture - Prof. David Kleinfeld
| |||||
{711} | |||||
PMID-19299587[0] Optical Deconstruction of Parkinsonian Neural Circuitry.
____References____
| |||||
{1334} | |||||
PMID-26627311 Monolithically Integrated μLEDs on Silicon Neural Probes for High-Resolution Optogenetic Studies in Behaving Animals.
| |||||
{1287} |
ref: -0
tags: maleimide azobenzine glutamate photoswitch optogenetics
date: 06-16-2014 21:19 gmt
revision:0
[head]
|
||||
PMID-16408092 Allosteric control of an ionotropic glutamate receptor with an optical switch
| |||||
{1283} | |||||
PMID-17521567 Remote control of neuronal activity with a light-gated glutamate receptor.
| |||||
{1269} |
ref: -0
tags: hinton convolutional deep networks image recognition 2012
date: 01-11-2014 20:14 gmt
revision:0
[head]
|
||||
{1257} |
ref: -0
tags: Anna Roe optogenetics artificial dura monkeys intrinisic imaging
date: 09-30-2013 19:08 gmt
revision:3
[2] [1] [0] [head]
|
||||
PMID-23761700 Optogenetics through windows on the brain in nonhuman primates
| |||||
{1255} |
ref: -0
tags: Disseroth Kreitzer parkinsons optogenetics D1 D2 6OHDA
date: 09-30-2013 18:15 gmt
revision:0
[head]
|
||||
PMID-20613723 Regulation of parkinsonian motor behaviors by optogenetic control of basal ganglia circuitry
| |||||
{1177} | |||||
IEEE-1196780 (pdf) 3D flexible multichannel neural probe array
| |||||
{1214} | |||||
PMID-7972766 Brain and cerebrospinal fluid motion: real-time quantification with M-mode MR imaging.
| |||||
{54} | |||||
!:
| |||||
{913} | |||||
PMID-21499255[0] Reversible large-scale modification of cortical networks during neuroprosthetic control.
Other notes:
____References____
| |||||
{1058} | |||||
PMID-19596378 Magnetic insertion system for flexible electrode implantation.
| |||||
{1183} | |||||
PMID-22574249 High spatial and temporal resolution wide-field imaging of neuron activity using quantum NV-diamond.
| |||||
{1125} |
ref: -0
tags: active filter design Netherlands Gerrit Groenewold
date: 02-17-2012 20:27 gmt
revision:0
[head]
|
||||
IEEE-04268406 (pdf) Noise and Group Delay in Actvie Filters
| |||||
{425} | |||||
images/425_1.pdf August 2007
| |||||
{1007} | |||||
IEEE-5910570 (pdf) Spiking neural network decoder for brain-machine interfaces
____References____ Dethier, J. and Gilja, V. and Nuyujukian, P. and Elassaad, S.A. and Shenoy, K.V. and Boahen, K. Neural Engineering (NER), 2011 5th International IEEE/EMBS Conference on 396 -399 (2011) | |||||
{998} | |||||
The Coming War on General Computation "M.P.s and Congressmen and so on are elected to represent districts and people, not disciplines and issues. We don't have a Member of Parliament for biochemistry, and we don't have a Senator from the great state of urban planning, and we don't have an M.E.P. from child welfare. " | |||||
{993} | |||||
IEEE-1439548 (pdf) Interpreting spatial and temporal neural activity through a recurrent neural network brain-machine interface
____References____ Sanchez, J.C. and Erdogmus, D. and Nicolelis, M.A.L. and Wessberg, J. and Principe, J.C. Interpreting spatial and temporal neural activity through a recurrent neural network brain-machine interface Neural Systems and Rehabilitation Engineering, IEEE Transactions on 13 2 213 -219 (2005) | |||||
{968} |
ref: Bassett-2009.07
tags: Weinberger congnitive efficiency beta band neuroimagaing EEG task performance optimization network size effort
date: 12-28-2011 20:39 gmt
revision:1
[0] [head]
|
||||
PMID-19564605[0] Cognitive fitness of cost-efficient brain functional networks.
____References____
| |||||
{323} |
ref: Loewenstein-2006.1
tags: reinforcement learning operant conditioning neural networks theory
date: 12-07-2011 03:36 gmt
revision:4
[3] [2] [1] [0] [head]
|
||||
PMID-17008410[0] Operant matching is a generic outcome of synaptic plasticity based on the covariance between reward and neural activity
____References____ | |||||
{883} | |||||
Response to Jonah Lehrer's The Web and the Wisdom of Crowds: Lehrer is right on one thing: culture. We're all consuming similar things (e.g. Rebecca Black) via the strong positive feedback of sharing things that you like, liking things that you share, and becoming more like the things that are shared with you. Will this lead to a cultural convergence, or stable n-ary system? To early to tell, but probably not: likely this is nothing new. Would you expect music to collapse to a single genre? No way. Sure, there will be pop culture via the mechanisms Lehrer suggests, but meanwhile there is too much to explore, and we like novelty too much. Regarding decision making through stochastic averaging as implemented in democracy, I have to agree with John Hawk here. The growing availability of knowledge, news, and other opinions should be a good thing. This ought to be more than enough to counteract the problem of everyone reading say the NYTimes instead of many varied local newspapers; there should be no impoverishment of opinion. Furthermore, we read blogs (like Lehrer's) which have to compete increasingly honestly in the attention economy. The cost of redirecting our attention has gone from that of a subscription to free. Plus, this attention economy ties communication to reality at more points - each reader, as opposed to each publisher, is partially responsible for information amplification and dissemination. (I mean I just published this damn thing and almost zero cost - is that not a great thing?) | |||||
{862} |
ref: -0
tags: backpropagation cascade correlation neural networks
date: 12-20-2010 06:28 gmt
revision:1
[0] [head]
|
||||
The Cascade-Correlation Learning Architecture
| |||||
{795} |
ref: work-0
tags: machine learning reinforcement genetic algorithms
date: 10-26-2009 04:49 gmt
revision:1
[0] [head]
|
||||
I just had dinner with Jesse, and the we had a good/productive discussion/brainstorm about algorithms, learning, and neurobio. Two things worth repeating, one simpler than the other: 1. Gradient descent / Newton-Rhapson like techniques should be tried with genetic algorithms. As of my current understanding, genetic algorithms perform an semi-directed search, randomly exploring the space of solutions with natural selection exerting a pressure to improve. What if you took the partial derivative of each of the organism's genes, and used that to direct mutation, rather than random selection of the mutated element? What if you looked before mating and crossover? Seems like this would speed up the algorithm greatly (though it might get it stuck in local minima, too). Not sure if this has been done before - if it has, edit this to indicate where! 2. Most supervised machine learning algorithms seem to rely on one single, externally applied objective function which they then attempt to optimize. (Rather this is what convex programming is. Unsupervised learning of course exists, like PCA, ICA, and other means of learning correlative structure) There are a great many ways to do optimization, but all are exactly that - optimization, search through a space for some set of weights / set of rules / decision tree that maximizes or minimizes an objective function. What Jesse and I have arrived at is that there is no real utility function in the world, (Corollary #1: life is not an optimization problem (**)) -- we generate these utility functions, just as we generate our own behavior. What would happen if an algorithm iteratively estimated, checked, cross-validated its utility function based on the small rewards actually found in the world / its synthetic environment? Would we get generative behavior greater than the complexity of the inputs? (Jesse and I also had an in-depth talk about information generation / destruction in non-linear systems.) Put another way, perhaps part of learning is to structure internal valuation / utility functions to set up reinforcement learning problems where the reinforcement signal comes according to satisfaction of sub-goals (= local utility functions). Or, the gradient signal comes by evaluating partial derivatives of actions wrt Creating these goals is natural but not always easy, which is why one reason (of very many!) sports are so great - the utility function is clean, external, and immutable. The recursive, introspective creation of valuation / utility functions is what drives a lot of my internal monologues, mixed with a hefty dose of taking partial derivatives (see {780}) based on models of the world. (Stated this way, they seem so similar that perhaps they are the same thing?) To my limited knowledge, there has been some work as of recent in the creation of sub-goals in reinforcement learning. One paper I read used a system to look for states that had a high ratio of ultimately rewarded paths to unrewarded paths, and selected these as subgoals (e.g. rewarded the agent when this state was reached.) I'm not talking about these sorts of sub-goals. In these systems, there is an ultimate goal that the researcher wants the agent to achieve, and it is the algorithm's (or s') task to make a policy for generating/selecting behavior. Rather, I'm interested in even more unstructured tasks - make a utility function, and a behavioral policy, based on small continuous (possibly irrelevant?) rewards in the environment. Why would I want to do this? The pet project I have in mind is a 'cognitive' PCB part placement / layout / routing algorithm to add to my pet project, kicadocaml, to finally get some people to use it (the attention economy :-) In the course of thinking about how to do this, I've realized that a substantial problem is simply determining what board layouts are good, and what are not. I have a rough aesthetic idea + some heuristics that I learned from my dad + some heuristics I've learned through practice of what is good layout and what is not - but, how to code these up? And what if these aren't the best rules, anyway? If i just code up the rules I've internalized as utility functions, then the board layout will be pretty much as I do it - boring! Well, I've stated my sub-goal in the form of a problem statement and some criteria to meet. Now, to go and search for a decent solution to it. (Have to keep this blog m8ta!) (Or, realistically, to go back and see if the problem statement is sensible). (**) Corollary #2 - There is no god. nod, Dawkins. | |||||
{789} | |||||
I've been reading Computational Explorations in Cognitive Neuroscience, and decided to try the code that comes with / is associated with the book. This used to be called "PDP+", but was re-written, and is now called Emergent. It's a rather large program - links to Qt, GSL, Coin3D, Quarter, Open Dynamics Library, and others. The GUI itself seems obtuse and too heavy; it's not clear why they need to make this so customized / panneled / tabbed. Also, it depends on relatively recent versions of each of these libraries - which made the install on my Debian Lenny system a bit of a chore (kinda like windows). A really strange thing is that programs are stored in tree lists - woah - a natural folding editor built in! I've never seen a programming language that doesn't rely on simple text files. Not a bad idea, but still foreign to me. (But I guess programs are inherently hierarchal anyway.) Below, a screenshot of the whole program - note they use a Coin3D window to graph things / interact with the model. The colored boxes in each network layer indicate local activations, and they update as the network is trained. I don't mind this interface, but again it seems a bit too 'heavy' for things that are inherently 2D (like 2D network activations and the output plot). It's good for seeing hierarchies, though, like the network model. All in all looks like something that could be more easily accomplished with some python (or ocaml), where the language itself is used for customization, and not a GUI. With this approach, you spend more time learning about how networks work, and less time programming GUIs. On the other hand, if you use this program for teaching, the gui is essential for debugging your neural networks, or other people use it a lot, maybe then it is worth it ... In any case, the book is very good. I've learned about GeneRec, which uses different activation phases to compute local errors for the purposes of error-minimization, as well as the virtues of using both Hebbian and error-based learning (like GeneRec). Specifically, the authors show that error-based learning can be rather 'lazy', purely moving down the error gradient, whereas Hebbian learning can internalize some of the correlational structure of the input space. You can look at this internalization as 'weight constraint' which limits the space that error-based learning has to search. Cool idea! Inhibition also is a constraint - one which constrains the network to be sparse. To use his/their own words: ... given the explanation above about the network's poor generalization, it should be clear why both Hebbian learning and kWTA (k winner take all) inhibitory competition can improve generalization performance. At the most general level, they constitute additional biases that place important constraints on the learning and the development of representations. Mroe specifically, Hebbian learning constrains the weights to represent the correlational structure of the inputs to a given unit, producing systematic weight patterns (e.g. cleanly separated clusters of strong correlations). Inhibitory competition helps in two ways. First, it encourages individual units to specialize in representing a subset of items, thus parcelling up the task in a much cleaner and more systematic way than would occur in an otherwise unconstrained network. Second, inhibition greatly restricts the settling dynamics of the network, greatly constraining the number of states the network can settle into, and thus eliminating a large proportion of the attractors that can hijack generalization.." | |||||
{787} | |||||
My theory on the Flynn effect - human intelligence IS increasing, and this is NOT stopping. Look at it from a ML perspective: there is more free time to get data, the data (and world) has almost unlimited complexity, the data is much higher quality and much easier to get (the vast internet & world!(travel)), there is (hopefully) more fuel to process that data (food!). Therefore, we are getting more complex, sophisticated, and intelligent. Also, the idea that less-intelligent people having more kids will somehow 'dilute' our genetic IQ is bullshit - intelligence is mostly a product of environment and education, and is tailored to the tasks we need to do; it is not (or only very weakly, except at the extremes) tied to the wetware. Besides, things are changing far too fast for genetics to follow. Regarding this social media, like facebook and others, you could posit that social intelligence is increasing, along similar arguments to above: social data is seemingly more prevalent, more available, and people spend more time examining it. Yet this feels to be a weaker argument, as people have always been socializing, talking, etc., and I'm not sure if any of these social media have really increased it. Irregardless, people enjoy it - that's the important part. My utopia for today :-) | |||||
{690} | |||||
PMID-10404201 Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex.
| |||||
{776} | |||||
http://www.willamette.edu/~gorr/classes/cs449/intro.html -- descent resource, good explanation of the equations associated with artificial neural networks. | |||||
{756} |
ref: life-0
tags: education wikinomics internet age college university pedagogy
date: 06-11-2009 12:52 gmt
revision:0
[head]
|
||||
Will universities stay relevant? and the rest of the wikinomics blog
| |||||
{724} | |||||
| |||||
{695} | |||||
Alopex: A Correlation-Based Learning Algorithm for Feed-Forward and Recurrent Neural Networks (1994)
| |||||
{669} | |||||
PMID-19191602 A New Hypothesis for Sleep: Tuning for Criticality.
| |||||
{503} | |||||
quote: Consumers also pay high taxes for telecommunication services, averaging about 13 percent on some telecom services, similar to the tax rate on tobacco and alcohol, Mehlman said. One tax on telecom service has remained in place since the 1898 Spanish-American War, when few U.S. residents had telephones, he noted. "We think it's a mistake to treat telecom like a luxury and tax it like a sin," he said. from: The internet could run out of capacity in two years comments:
| |||||
{497} | |||||
http://dotpublic.istumbler.net/
| |||||
{479} | |||||
http://pespmc1.vub.ac.be/books/IntroCyb.pdf -- dated, but still interesting, useful, a book in and of itself!
| |||||
{467} | |||||
Self-learning fuzzy neural network with optimal on-line leaning for water injection control of a turbocharged automobile.
| |||||
{465} | |||||
good explanation of 32-bit CRC (from the blackfin BF537 hardware ref): | |||||
{401} | |||||
http://hardm.ath.cx:88/pdf/RFpenetrationInTissue.pdf
even more interesting: wireless brain machine interface | |||||
{384} | |||||
notes on reading magstripe cards:
| |||||
{277} | |||||
PMID-15888522[0] Motor cortex neural correlates of output kinematics and kinetics during isometric-force and arm-reaching tasks.
____References____ | |||||
{230} |
ref: engineering notes-0
tags: homopolar generator motor superconducting magnet
date: 03-09-2007 14:39 gmt
revision:0
[head]
|
||||
http://hardm.ath.cx:88/pdf/homopolar.pdf
| |||||
{223} | |||||
calculations for a strong DC loop magnet using 1/8" copper capillary tubing:
| |||||
{7} |
ref: bookmark-0
tags: book information_theory machine_learning bayes probability neural_networks mackay
date: 0-0-2007 0:0
revision:0
[head]
|
||||
http://www.inference.phy.cam.ac.uk/mackay/itila/book.html -- free! (but i liked the book, so I bought it :) | |||||
{92} | |||||
with the extended kalman filter, from '92: http://ftp.ccs.neu.edu/pub/people/rjw/kalman-ijcnn-92.ps with the unscented kalman filter : http://hardm.ath.cx/pdf/NNTrainingwithUnscentedKalmanFilter.pdf | |||||
{40} |
ref: bookmark-0
tags: Bayes Baysian_networks probability probabalistic_networks Kalman ICA PCA HMM Dynamic_programming inference learning
date: 0-0-2006 0:0
revision:0
[head]
|
||||
http://www.cs.ubc.ca/~murphyk/Bayes/bnintro.html very, very good! many references, well explained too. | |||||
{39} | |||||
http://www.numenta.com/Numenta_HTM_Concepts.pdf
| |||||
{20} |
ref: bookmark-0
tags: neural_networks machine_learning matlab toolbox supervised_learning PCA perceptron SOM EM
date: 0-0-2006 0:0
revision:0
[head]
|
||||
http://www.ncrg.aston.ac.uk/netlab/index.php n.b. kinda old. (or does that just mean well established?) |