m8ta
You are not authenticated, login. |
|
{1546} | ||
Local synaptic learning rules suffice to maximize mutual information in a linear network
x = randn(1000, 10); Q = x' * x; a = 0.001; Y = randn(10, 1); y = zeros(10, 1); for i = 1:1000 y = Y + (eye(10) - a*Q)*y; end y - pinv(Q)*Y / a % should be zero.
To this is added a 'sensing' learning and 'noise' unlearning phase -- one optimizes , the other minimizes . Everything is then applied, similar to before, to a gaussian-filtered one-dimensional white-noise stimuli. He shows this results in bandpass filter behavior -- quite weak sauce in an era where ML papers are expected to test on five or so datasets. Even if this was 1992 (nearly forty years ago!), it would have been nice to see this applied to a more realistic dataset; perhaps some of the following papers? Olshausen & Field came out in 1996 -- but they applied their algorithm to real images. In both Olshausen & this work, no affordances are made for multiple layers. There have to be solutions out there... | ||
{1545} | ||
Self-organizaton in a perceptual network
One may critically challenge the infomax idea: we very much need to (and do) throw away spurious or irrelevant information in our sensory streams; what upper layers 'care about' when making decisions is certainly relevant to the lower layers. This credit-assignment is neatly solved by backprop, and there are a number 'biologically plausible' means of performing it, but both this and infomax are maybe avoiding the problem. What might the upper layers really care about? Likely 'care about' is an emergent property of the interacting local learning rules and network structure. Can you search directly in these domains, within biological limits, and motivated by statistical reality, to find unsupervised-learning networks? You'll still need a way to rank the networks, hence an objective 'care about' function. Sigh. Either way, I don't per se put a lot of weight in the infomax principle. It could be useful, but is only part of the story. Otherwise Linsker's discussion is accessible, lucid, and prescient. Lol. | ||
{1493} | ||
PMID-27690349 Nonlinear Hebbian Learning as a Unifying Principle in Receptive Field Formation
| ||
{760} |
ref: -0
tags: LDA myopen linear discriminant analysis classification
date: 01-03-2012 02:36 gmt
revision:2
[1] [0] [head]
|
|
How does LDA (Linear discriminant analysis) work? It works by projecting data points onto a series of planes, one per class of output, and then deciding based which projection plane is the largest. Below, to the left is a top-view of this projection with 9 different classes of 2D data each in a different color. Right is a size 3D view of the projection - note the surfaces seem to form a parabola. Here is the matlab code that computes the LDA (from myopen's ceven % TrainData and TrainClass are inputs, column major here. % (observations on columns) N = size(TrainData,1); Ptrain = size(TrainData,2); Ptest = size(TestData,2); % add a bit of interpolating noise to the data. sc = std(TrainData(:)); TrainData = TrainData + sc./1000.*randn(size(TrainData)); K = max(TrainClass); % number of classes. %%-- Compute the means and the pooled covariance matrix --%% C = zeros(N,N); for l = 1:K; idx = find(TrainClass==l); % measure the mean per class Mi(:,l) = mean(TrainData(:,idx)')'; % sum all covariance matrices per class C = C + cov((TrainData(:,idx)-Mi(:,l)*ones(1,length(idx)))'); end C = C./K; % turn sum into average covariance matrix Pphi = 1/K; Cinv = inv(C); %%-- Compute the LDA weights --%% for i = 1:K Wg(:,i) = Cinv*Mi(:,i); % this is the slope of the plane Cg(:,i) = -1/2*Mi(:,i)'*Cinv*Mi(:,i) + log(Pphi)'; % and this, the origin-intersect. end %%-- Compute the decision functions --%% Atr = TrainData'*Wg + ones(Ptrain,1)*Cg; % see - just a simple linear function! Ate = TestData'*Wg + ones(Ptest,1)*Cg; errtr = 0; AAtr = compet(Atr'); % this compet function returns a sparse matrix with a 1 % in the position of the largest element per row. % convert to indices with vec2ind, below. TrainPredict = vec2ind(AAtr); errtr = errtr + sum(sum(abs(AAtr-ind2vec(TrainClass))))/2; netr = errtr/Ptrain; PeTrain = 1-netr; | ||
{796} | ||
An interesting field in ML is nonlinear dimensionality reduction - data may appear to be in a high-dimensional space, but mostly lies along a nonlinear lower-dimensional subspace or manifold. (Linear subspaces are easily discovered with PCA or SVD(*)). Dimensionality reduction projects high-dimensional data into a low-dimensional space with minimum information loss -> maximal reconstruction accuracy; nonlinear dim reduction does this (surprise!) using nonlinear mappings. These techniques set out to find the manifold(s):
(*) SVD maps into 'concept space', an interesting interpretation as per Leskovec's lecture presentation. | ||
{588} | ||
images/588_1.pdf -- Good lecture on LDA. Below, simple LDA implementation in matlab based on the same: % data matrix in this case is 36 x 16, % with 4 examples of each of 9 classes along the rows, % and the axes of the measurement (here the AR coef) % along the columns. Sw = zeros(16, 16); % within-class scatter covariance matrix. means = zeros(9,16); for k = 0:8 m = data(1+k*4:4+k*4, :); % change for different counts / class Sw = Sw + cov( m ); % sum the means(k+1, :) = mean( m ); %means of the individual classes end % compute the class-independent transform, % e.g. one transform applied to all points % to project them into one plane. Sw = Sw ./ 9; % 9 classes criterion = inv(Sw) * cov(means); [eigvec2, eigval2] = eig(criterion); See {587} for results on EMG data. | ||
{409} |
ref: bookmark-0
tags: optimization function search matlab linear nonlinear programming
date: 08-09-2007 02:21 gmt
revision:0
[head]
|
|
http://www.mat.univie.ac.at/~neum/ very nice collection of links!! | ||
{141} |
ref: learning-0
tags: motor control primitives nonlinear feedback systems optimization
date: 0-0-2007 0:0
revision:0
[head]
|
|
http://hardm.ath.cx:88/pdf/Schaal2003_LearningMotor.pdf not in pubmed. | ||
{34} |
ref: bookmark-0
tags: linear_algebra solution simultaneous_equations GPGPU GPU LUdecomposition clever
date: 0-0-2006 0:0
revision:0
[head]
|
|