Information, stability and learning complexity in associative memories
Abstract
The relationship between amount of information and learning complexity is studied for the discrete Hopfield model of associative memory. More precisely, we analize, from a statistical point of view, the relation between the probability of a state of the system to be a stable equilibrium (i.e. a memory) and a value of entropy or uncertainty associated to it. Computer experiments are made to confirm these results.Downloads
References
J. J. Hopfield, "Neural networks and physical systems with emergent computational abilities", Proc. Nat. Acad. Sc., USA, 79, pp. 2554-8, April 1982 (Biophysics).
T. Hastie, R. Tibshirani and J. Friedman, The Ekements of Statistical Learning: Data Mining, Inference, and Prediction. Berlin, Germany: Springer, 2013.
A. C. Rencher and W. F. Christensen, Methods of Multivariate Analysis. New Jersey, USA: Wiley, 2012. http://dx.doi.org/10.1002/9781118391686
C. E. Shannon; "A mathematical theory of communication", Bell System Technical Journal, vol. XXVII, no. 3, pp. 379-423, 1948. http://dx.doi.org/10.1002/j.1538-7305.1948.tb01338.x
G. Grimmett and D. Stirzaker, Probability and random processes. Oxford, UK: Oxford University Press, 3rd. edition, 2001.
D. J. Amit, H. Gutfreund and H. Sompolinsky, Statistical Mechanics of Neural Networks near Saturation,} Annals of Physics 173, pp. 30-67, 1987. http://dx.doi.org/10.1016/0003-4916(87)90092-3
D. J. Amit, Modeling Brain Function. Cambridge, USA: Cambridge University Press, 1989. http://dx.doi.org/10.1017/CBO9780511623257
S. Haykin, Neural Networks and Learning Machines Upper Saddle River, USA: Pearson–Prentice Hall, 2011.
Y. S. Abu-Mustafa and J-M. St. Jacques, "Information capacity of the Hopfield model", IEEE Trans. Inf. Th., vol. IT-31, no. 41, pp. 461-464, 1985. http://dx.doi.org/10.1109/TIT.1985.1057069
R. J. McEliece, E. C. Posner, E. R. Rodemich and S. S. Venkatesh, "The Capacity of the Hopfield Associative Memory", California Institute of Technology, 1986.
Published papers are the exclusive responsibility of their authors and do not necessary reflect the opinions of the editorial committee.
IJMSOR respects the moral rights of its authors, whom must cede the editorial committee the patrimonial rights of the published material. In turn, the authors inform that the current work is unpublished and has not been previously published.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.
