You are not authenticated, login.
text: sort by
tags: modified
type: chronology
{816} is owned by tlh24.
[0] Vyssotski AL, Serkov AN, Itskov PM, Dell'Omo G, Latanov AV, Wolfer DP, Lipp HP, Miniature neurologgers for flying pigeons: multichannel EEG and action and field potentials in combination with GPS recording.J Neurophysiol 95:2, 1263-73 (2006 Feb)[1] Otto KJ, Johnson MD, Kipke DR, Voltage pulses change neural interface properties and improve unit recordings with chronically implanted microelectrodes.IEEE Trans Biomed Eng 53:2, 333-40 (2006 Feb)

hide / / print
ref: -2017 tags: attention transformer language model youtube google tech talk date: 02-26-2019 20:28 gmt revision:3 [2] [1] [0] [head]

Attention is all you need

  • Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
  • Attention is all you need neural network models
  • Good summary, along with: The Illustrated Transformer (please refer to this!)
  • Łukasz Kaiser mentions a few times how fragile the network is -- how easy it is to make something that doesn't train at all, or how many tricks by google experts were needed to make things work properly. it might be bravado or bluffing, but this is arguably not the way that biology fails.
  • Encoding:
  • Input is words encoded as 512-length vectors.
  • Vectors are transformed into length 64 vectors: query, key and value via differentiable weight matrices.
  • Attention is computed as the dot-product of the query (current input word) with the keys (values of the other words).
    • This value is scaled and passed through a softmax function to result in one attentional signal scaling the value.
  • Multiple heads' output are concatenated together, and this output is passed through a final weight matrix to produce a final value for the next layer.
    • So, attention in this respect looks like a conditional gain field.
  • 'Final value' above is then passed through a single layer feedforward net, with resnet style jump.
  • Decoding:
  • Use the attentional key value from the encoder to determine the first word through the output encoding (?) Not clear.
  • Subsequent causal decodes depend on the already 'spoken' words, plus the key-values from the encoder.
  • Output is a one-hot softmax layer from a feedforward layer; the sum total is differentiable from input to output using cross-entropy loss or KL divergence.

hide / / print
ref: -0 tags: Hinton google tech talk dropout deep neural networks Boltzmann date: 02-12-2019 08:03 gmt revision:2 [1] [0] [head]

Brains, sex, and machine learning -- Hinton google tech talk.

  • Hinton believes in the the power of crowds -- he thinks that the brain fits many, many different models to the data, then selects afterward.
    • Random forests, as used in predator, is an example of this: they average many simple to fit and simple to run decision trees. (is apparently what Kinect does)
  • Talk focuses on dropout, a clever new form of model averaging where only half of the units in the hidden layers are trained for a given example.
    • He is inspired by biological evolution, where sexual reproduction often spontaneously adds or removes genes, hence individual genes or small linked genes must be self-sufficient. This equates to a 'rugged individualism' of units.
    • Likewise, dropout forces neurons to be robust to the loss of co-workers.
    • This is also great for parallelization: each unit or sub-network can be trained independently, on it's own core, with little need for communication! Later, the units can be combined via genetic algorithms then re-trained.
  • Hinton then observes that sending a real value p (output of logistic function) with probability 0.5 is the same as sending 0.5 with probability p. Hence, it makes sense to try pure binary neurons, like biological neurons in the brain.
    • Indeed, if you replace the backpropagation with single bit propagation, the resulting neural network is trained more slowly and needs to be bigger, but it generalizes better.
    • Neurons (allegedly) do something very similar to this by poisson spiking. Hinton claims this is the right thing to do (rather than sending real numbers via precise spike timing) if you want to robustly fit models to data.
      • Sending stochastic spikes is a very good way to average over the large number of models fit to incoming data.
      • Yes but this really explains little in neuroscience...
  • Paper referred to in intro: Livnat, Papadimitriou and Feldman, PMID-19073912 and later by the same authors PMID-20080594
    • A mixability theory for the role of sex in evolution. -- "We define a measure that represents the ability of alleles to perform well across different combinations and, using numerical iterations within a classical population-genetic framework, show that selection in the presence of sex favors this ability in a highly robust manner"
    • Plus David MacKay's concise illustration of why you need sex, pg 269, __Information theory, inference, and learning algorithms__
      • With rather simple assumptions, asexual reproduction yields 1 bit per generation,
      • Whereas sexual reproduction yields G\sqrt G , where G is the genome size.

hide / / print
ref: -0 tags: palladium metal glass tought strong caltech date: 02-25-2014 19:02 gmt revision:1 [0] [head]

A damage-tolerant glass

  • Perhaps useful for the inserter needle?
  • WC-Co Tungesten carbide-cobalt cermet is another alternative.

hide / / print
ref: -0 tags: Moshe looks automatic programming google tech talk links date: 11-07-2012 07:38 gmt revision:3 [2] [1] [0] [head]

List of links from Moshe Looks google tech talk:

hide / / print
ref: Wattanapanitch-2007 tags: recording tech amplifier cascode MOS-bipolar pseudoresistor MIT date: 01-15-2012 18:13 gmt revision:5 [4] [3] [2] [1] [0] [head]

IEEE-4358095 (pdf) An Ultra-Low-Power Neural Recording Amplifier and its use in Adaptively-Biased Multi-Amplifier Arrays.

  • images/729_1.pdf -- copy, just in case.
  • Masters thesis - shows the development of, as the title explains, an ultra low power neural amplifier.
  • Probably the best amplifier out there. NEF 2.67; theoretical limit 2.02.
  • Final design uses folded cascode operational transconductance amplifier (OTA)
    • Design employs a capacitor-feedback gain stage of 40db followed by a lowpass stage.
    • Majority of the current is passed through large subthreshold PMOS input transistors.
      • PMOS has lower noise than NMOS in most processes.
      • Subthreshold has the highest transconductance-to-current ratio. (ratio of a ratio)
    • Cascode transistors self-shunt their own current noise sources.
    • Design takes 0.16 mm^2 in 0.5 um AMI CMOS process, uses 2.7 uA from a ~2.8V supply, input referred noise of 3 uVrms
    • Thesis gives all design parameters for the transistors.
    • Input is AC coupled, DC path through gigaohm MOS-bipolar psudoresistor.
      • this path gracefully decays to diode-connected MOS or bipolar transistors if the voltage is high.
    • images/729_1.pdf
  • Last chapter details the use of envelope detection to adaptively change the bias current of the input stage
    • That is, if an electrode is noisy, the bias current is decreased!

hide / / print
ref: notes-2000.09 tags: BMI recording technology Chapin Nicolelis battery Wolf date: 01-06-2012 03:09 gmt revision:4 [3] [2] [1] [0] [head]

from the book "Neural Prostheses for Restoration of Sensory and Motor Function" edited by John Chapin and Karen Moxon.

Phillip Kennedy's one-channel neurotrophic glass electrode BMI (axons apparently grew into the electrode, and he recorded from them)

Pat Wolf on neural amplification / telemetry technology

battery technology for powering the neural telemetry

hide / / print
ref: life-notes-2007 tags: electrode assay technology electrophysiology hack ad-hoc date: 01-03-2012 07:10 gmt revision:3 [2] [1] [0] [head]

properties of electrodes that are to penetrate the pia mater of a rhesus macaque:

  1. must easily go into a canned peach (in heavy or light sauce, it does not matter)
  2. does not go into pineapple cross-grain
  3. does go into the end-grain of pineapple
  4. penetrates the skin of a red grape (somewhat fresh) ~= pia
    1. The pia is a bit more tough than this, but is much less firm - if you are implanting electrodes that are any less than extremely sharp - e.g. etched - it will dimple the surface and not penetrate. Very sharp electrodes are key for getting through this tough membrane - which is even tougher in humans!
      • dimpling seems to silence cortical activity (observational evidence for this only)
      • however, once implanted lower-impedance electrodes work better. Low current microstimulation may be able to round the sharp tips of tungsten electrodes - we may want to test this.
    1. microdissection of the pia often damages the surface vasulature of the cortex, leading to localized infarctions, and hence should be avoided (unless you are really good)
    2. Bunching multiple elctrodes into one shaft - that is, making the shaft thicker and duller (albiet staggered) is not a good strategy for entering the brain (need to test the present monkeys).
  1. Cortical layer V (location of large pyramidal cells + betz cells in M1) in humans is 3-3.5mm below the surface, and ~1.6mm deep in rhesus. microwire/microwire arrays should have at least 2mm free wire length if intended for monkeys, and 4mm free wire if intended for humans.
    1. M1/S1 / central sulcus region is mostly inactive under isoflouro anesthesia, somewhat mangled/depressed with light ketamine, and silent with fentanyl. So, be careful with intraoperative recordings - the monkey/rat may be too deep, hence no cells to listen to!

hide / / print
ref: Perelman-2007.01 tags: Technion recording silicon date: 01-03-2012 01:07 gmt revision:2 [1] [0] [head]

PMID-17260864[0] An integrated system for multichannel neuronal recording with spike/LFP separation, integrated A/D conversion and threshold detection.

  • Use an RC filter (5MOhm resistor (polysilicon) + 160pf cap (gate oxide)) to split spike and LFP signals.
  • Weak-inversion MOS transistor to vary the high-pass pole. This can be varied over several orders of magnitude with a DAC (and can be varied to compensate for process variation).
  • Have some good debugging notes on their chip - how the weak inversion MOS transistors leaked more current than expected.


[0] Perelman Y, Ginosar R, An integrated system for multichannel neuronal recording with spike/LFP separation, integrated A/D conversion and threshold detection.IEEE Trans Biomed Eng 54:1, 130-7 (2007 Jan)

hide / / print
ref: Merletti-2009.02 tags: surface EMG multielectrode recording technology italy date: 01-03-2012 01:07 gmt revision:2 [1] [0] [head]

PMID-19042063[0] Technology and instrumentation for detection and conditioning of the surface electromyographic signal: state of the art

  • good background & review of surface EMG (sEMG) - noise levels, electrodes, electronics. eg. Instrumentation amplifiers with an input resistance < 100MOhm are not recommended, and the lower the input capacitance, the better: the impedance of a 10pf capacitor at 100hz is 160MOhm.
  • Low and balanced input impedances are required to reduce asymmetric filtering of common-mode power-line noise.


[0] Merletti R, Botter A, Troiano A, Merlo E, Minetto MA, Technology and instrumentation for detection and conditioning of the surface electromyographic signal: state of the art.Clin Biomech (Bristol, Avon) 24:2, 122-34 (2009 Feb)

hide / / print
ref: Akin-1995.06 tags: Najafi neural recording technology micromachined digital TETS 1995 PNS schematics date: 01-01-2012 20:23 gmt revision:8 [7] [6] [5] [4] [3] [2] [head]

IEEE-717081 (pdf) An Implantable Multichannel Digital neural recording system for a micromachined sieve electrode

  • Later pub: IEEE-654942 (pdf) -- apparently putting on-chip isolated diodes is a difficult task.
  • 90mw of power @ 5V, 4x4mm of area (!!)
  • targeted for regenerated peripheral neurons grown through a micromachined silicon sieve electrode.
    • PNS nerves are deliberately severed and allowed to regrow through the sieve.
  • 8bit low-power current-mode ADC. seems like a clever design to me - though I can't really follow the operation from the description written there.
  • class e transmitter amplifier.
  • 3um BiCMOS process. (you get vertical BJTs and Zener diodes)
  • has excellent schematics. - including the voltage regulator, envelop detector & ADC.
  • most of the power is dissipated in the voltage regulator (!!) - 80mW of 90mW.
  • tiny!
  • rather than using pseudoresistors, they use diode-capacitor input filter which avoids the need for chopping or off-chip hybrid components.
  • can record from any two of 32 input channels. I think the multiplexer is after the preamp - right?


Akin, T. and Najafi, K. and Bradley, R.M. Solid-State Sensors and Actuators, 1995 and Eurosensors IX.. Transducers '95. The 8th International Conference on 1 51 -54 (1995)

hide / / print
ref: Tehovnik-1996.03 tags: ICMS technique Tehovnik MIT 1996 current density microstimulation date: 12-29-2011 05:11 gmt revision:2 [1] [0] [head]

PMID-8815302[0] Electrical stimulation of neural tissue to evoke behavioral responses

  • reference to justify our current levels.
  • radial dispersion of current, inverse square falloff of excitability.
  • low currents (10 ua) can activate 10-1000 of neurons in cat M1 (allegedly).


[0] Tehovnik EJ, Electrical stimulation of neural tissue to evoke behavioral responses.J Neurosci Methods 65:1, 1-17 (1996 Mar)

hide / / print
ref: -0 tags: caltech right date: 01-11-2011 05:15 gmt revision:0 [head]

Commentary on Why Caltech Is in a Class by Itself (I do not go to Caltech, but this is not the reason the article rubbed me):

This is a good argument, especially given the real need to educate students in STEM subjects. However I would like to make four counterpoints:

1. Only accepting students based on test scores - as your article suggests - strikes me as a rather narrow criteria. There are many forms of intelligence, and the variety of problems is much broader than those posed within a SAT - surely there should be some leeway in accepting individuals? Any single criteria would seem to impoverish the student body.

2. Maybe affirmative action will not heal wounds; I have not read your book (obviously). But what would happen if all universities stopped accepting minority students? Our society is already stratified, and this would make it worse. (The counterargument would be that by making the criteria equal, minorities would be forced to rise up - yes, eventually, but slow enough to not set off a positive feedback loop)

3. I don't watch sports myself, but a lot of my friends do. Sure, there may be an excess now of section-9 type stuff, but is there anything wrong with cultivating athletic excellence? Is it not inspiring to be on campus with these people as well as to watch them? More importantly, people *really enjoy* athletics, which seems a good enough reason to me.

4. Where's your soul, man? My criteria would be: "Caltech is great because the students there are motivated and happy, and when they graduate they go on to contribute to and greatly enjoy the f- out of life." which is true, btw.

hide / / print
ref: -0 tags: radiolab what does technology want Kevin Kelley teleology date: 12-19-2010 06:23 gmt revision:1 [0] [head]

What does technology want? An interview with Steven Johnson and Kevin Kelly at the New York Public Library, usefully condensed to a half-hour segment for the RadioLab podcast.

  • Many of the ideas are not new - its teleology: a means of understanding the world by interpreting everything in terms of 'wants' and 'desires'. As Douglas Hofstadter explains in his book, __I am a Strange Loop__, this is really just a cognitive shortcut - not much more, not much less - which allows us to interpret things which exhibit attractor-like dynamics. Hence, as the comments on the page note, the title is to some degree just a semantic trick.
  • That said, the idea behind the title is very interesting: technology, by virtue of being subject to recursive selection, iterative refinement, code reuse (aggressive copy-paste, idea promiscuity) just like biological organisms will open-system violate the second law of thermodynamics. (They don't actually say this, but that's my interpretation).
  • Here's my logic: Imagine a statistical distribution - a population of things, animals, ideas, products, whatever. Pass them through a statistical selection process, be it evolution, the marketplace, the predictive models of your mind & associated decision making processes, a political system (ok, maybe here), the immune system, the modern attention economy (maybe here, too). This gives you a new population of things, which the selective process has impinged a degree of information about itself (the real world, usually) upon. Duplicate, spawn some more, run it through the selective process recursively ad infinitum, and the (Shannon) information contained in the resulting populations will increase. Things do not tend toward disorder.
  • That carries some heavy caveats - the information content of the selection system (which may be interpreted as applying a 'fitness' or 'objective' function) must be, at every point, much higher than that of the population for the transfer to occur. In the real world, that's easy - the information content of even a minute of life is far greater than that of our DNA! Furthermore, due to {825 coevolution} -- other organisms are our world - the information content of the selective process continually increases.
    • This implies that in some stable evolutionary niches, e.g. algae, the Shannon information of the genome must be approximately the same as the expected (as in, integral) information content of the selectively-important events of it's life. (Yea, I don't know about that either..too wishy-washy and intuitive to be useful; also algorithmic complexity doesn't scale linearly).
  • Some selection systems don't seem to be evolving to increasing complexity, however.
    • The political system: lots of problems. (1) The population is small. (2) The population (candidates) has strong incentive to mislead the selective process (the voters) (3) The information passed from selective process to population is low (a few bits every 4 years, times however many senate seats there are, divided by partisanness/statistical dependence between the bits).
    • I am happy to say, the communication issue (3) seems to be getting better - we know more about what out leaders are doing, and they know more about us - but it is imperfect, filtered through a system (the media) who holds a different objective (garners interest) than the ultimate population (who wants, roughly, security and wealth).
    • The attention economy: very strong highpass characteristics (novelty rather than truthiness), strong limits on individual complexity in the population (the ideas must be conveyable). Yet! there is strong co-evolution. (Anyway, hell, it's supposed to keep us amused, doesn't it do that? It has never been proposed to be able to solve global problems..)
    • In both politics and media, our desire for novelty may be good - it directly forces the investigation of new ideas (new members of the population). New interpretations of events are continually sought; perhaps this is a worthy price to pay for losing a bit of objective reality?
      • This should be quantifiable using Bayes' rule, then tested in an experiment. That said, the loss-function for the reality weighting is dependent on estimates of environmental change.
  • Once again I have wandered away from the original subject. Oops. Yea, the other sections - about __Where Good Ideas Come From__ is basically common knowledge now / well covered by Thomas Kuhn. All good ideas are conceived by different people in different places at the same time; insight takes time and effort, and is seldom a eureka moment; often what is required is a perceptual shift, as per the discovery of air, again what Kuhn has covered.

This blog is probably failing in the attention economy. Again, oops ;-)

hide / / print
ref: Linderman-2006.01 tags: neural recording technology compact flash stanford Shenoy 2006 date: 04-15-2009 20:55 gmt revision:3 [2] [1] [0] [head]

PMID-17946450[0] An Autonomous, broadband, multi-channel neural recording system for freely behaving primates

  • goal: recording system for freely-behaving animals.
    • problems: battery life, size
    • cannot sample broadband.
    • non autonomous.
  • solution:
    • compact flash, ARM core
    • accelerometer?
    • mounted inside the monkey's skull in the dental cement.
  • specs


[0] Linderman MD, Gilja V, Santhanam G, Afshar A, Ryu S, Meng TH, Shenoy KV, An autonomous, broadband, multi-channel neural recording system for freely behaving primates.Conf Proc IEEE Eng Med Biol Soc 1no Issue 1212-5 (2006)

hide / / print
ref: notes-0 tags: robots Tokyo Institute of Technology date: 09-04-2008 17:30 gmt revision:5 [4] [3] [2] [1] [0] [head]

Robots & others designed & made at the Tokyo Institute of Technology (from the Hirose / Fukushima Robotics lab)

  • snake robots
  • gripper
      • with human-crushing, kid-grabbing power. frightening!
  • walking robots
      • -- 1994. can climb a 70 degree slope!
    • --window washing robot.
    • -- skating robot: walk ; skate. movie -- wow!
  • wheeled robots
  • other
    • -- prismatic, variable-speed eccentric linear drive. sorta like harmonic drive for linear motion? link

hide / / print
ref: bookmarks-0 tags: neurotechnology companies date: 08-30-2007 17:02 gmt revision:0 [head]

hide / / print
ref: notes-0 tags: low-power microprocessor design techniques ieee DSP date: 05-29-2007 03:30 gmt revision:2 [1] [0] [head]


also see IBM's eLite DSP project.

hide / / print
ref: notes-0 tags: recording tech tbsi biosignal telemetry date: 05-20-2007 16:40 gmt revision:1 [0] [head]

hide / / print
ref: Vyssotski-2006.02 tags: neurologger neural_recording recording_technology EEG SUA LFP electrical engineering date: 02-05-2007 06:21 gmt revision:6 [5] [4] [3] [2] [1] [0] [head]

PMID-16236777[0] Miniature neurologgers for flying pigeons: multichannel EEG and action and field potentials in combination with GPS recording.

Recording neuronal activity of animals moving through their natural habitat is difficult to achieve by means of conventional radiotelemetry. This illustration shows a new approach, exemplified by a homing pigeon carrying both a small GPS path recorder and a miniaturized action and field potential logger (“neurologger”), the entire assembly weighing maximally 35 g, a load carried easily by a pigeon over a distance of up to 50 km. Before release at a distant location, the devices are activated and store both positional and neuronal activity data during the entire flight. On return to the loft, all data are downloaded and can be analyzed using software for path analysis and electrical brain activity. Thus single unit activity or EEG patterns can be matched to the flight path superimposed on topographical maps. Such neurologgers may also be useful for a variety of studies using unrestrained laboratory animals in different environments or test apparatuses. The prototype on the hand-held pigeon records and stores EEG simultaneously from eight channels up to 47 h, or single unit activity from two channels during 9 h, but the number of channels can be increased without much gain in weight by sandwiching several of these devices. Further miniaturization can be expected. For details, see Vyssotski AL, Serkov AN, Itskov PM, Dell Omo G, Latanov AV, Wolfer DP, and Lipp H-P. Miniature neurologgers for flying pigeons: multichannel EEG and action and field potentials in combination with GPS recording. [1]


hide / / print
ref: bookmark-0 tags: neural_recording recording_technology electrical engineering DSP date: 0-0-2006 0:0 revision:0 [head]