Neuromemrisitive Architecture of HTM with On-Device Learning and Neurogenesis

Abstract

Hierarchical temporal memory (HTM) is a biomimetic sequence memory algorithm that holds promise for invariant representations of spatial and spatio-temporal inputs. This article presents a comprehensive neuromemristive crossbar architecture for the spatial pooler (SP) and the sparse distributed representation classifier, which are fundamental to the algorithm. There are several unique features in the proposed architecture that tightly link with the HTM algorithm. A memristor that is suitable for emulating the HTM synapses is identified and a new Z-window function is proposed. The architecture exploits the concept of synthetic synapses to enable potential synapses in the HTM. The crossbar for the SP avoids dark spots caused by unutilized crossbar regions and supports rapid on-chip training within two clock cycles. This research also leverages plasticity mechanisms such as neurogenesis and homeostatic intrinsic plasticity to strengthen the robustness and performance of the SP. The proposed design is benchmarked for image recognition tasks using Modified National Institute of Standards and Technology (MNIST) and Yale faces datasets, and is evaluated using different metrics including entropy, sparseness, and noise robustness. Detailed power analysis at different stages of the SP operations is performed to demonstrate the suitability for mobile platforms.

References

1. Subutai Ahmad and Jeff Hawkins. 2015. Properties of sparse distributed representations and their application to hierarchical temporal memory. arXiv preprint arXiv:1503.07469.
2. Julien Borghetti, Gregory S. Snider, Philip J. Kuekes, J. Joshua Yang, Duncan R. Stewart, and R. Stanley Williams. 2010. Memristive switches enable stateful logic operations via material implication. Nat. 464, 7290 (2010), 873--876.
3. S. Brivio, E. Covi, A. Serb, T. Prodromakis, M. Fanciulli, and S. Spiga. 2016. Experimental study of gradual/abrupt dynamics of HfO2-based memristive devices. Appl. Phys. Lett. 109, 13 (2016), 133504.
4. Deng Cai, Xiaofei He, Yuxiao Hu, Jiawei Han, and Thomas Huang. 2007. Learning a spatially smooth subspace for face recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’07). IEEE, 1--7.
5. Leon Chua. 1971. Memristor-the missing circuit element. IEEE Trans. Circuit Theory 18, 5 (1971), 507--519.
6. Jianwei Cui and Qinru Qiu. 2016. Towards memristor based accelerator for sparse matrix vector multiplication. In IEEE International Symposium on Circuits and Systems (ISCAS’16). IEEE, 121--124.
7. Yuwei Cui, Subutai Ahmad, and Jeff Hawkins. 2017. The HTM spatial pooler—A neocortical algorithm for online sparse distributed coding. Front. Comput. Neurosci. 11 (2017), 111.
8. Yuwei Cui, Chetan Surpur, Subutai Ahmad, and Jeff Hawkins. 2016. A comparative study of HTM and other neural network models for online sequence learning with streaming data. In International Joint Conference on Neural Networks (IJCNN’16). IEEE, 1530--1538.
9. Russell Eberhart and James Kennedy. 1995. A new optimizer using particle swarm theory. In Proceedings of the 6th International Symposium on Micro Machine and Human Science (MHS’95). IEEE, 39--43.
10. Alexander Fish, Vadim Milrud, and Orly Yadid-Pecht. 2005. High-speed and high-precision current winner-take-all circuit. IEEE Trans. Circuits Syst. II Express Briefs 52, 3 (2005), 131--135.
11. Peter Földiak. 1990. Forming sparse representations by local anti-Hebbian learning. Biol. Cybern. 64, 2 (1990), 165--170.
12. Dileep George and Jeff Hawkins. 2009. Towards a mathematical theory of cortical micro-circuits. PLoS Comput. Biol. 5, 10 (2009), e1000532.
13. Jeff Hawkins and Subutai Ahmad. 2016. Why neurons have thousands of synapses, a theory of sequence memory in neocortex. Front. Neural Circuits 10 (2016), 23.
14. Jeff Hawkins and Sandra Blakeslee. 2004. On Intelligence: How a New Understanding of the Brain Will Lead to Truly Intelligent Machines. New York: Henry Holt 8 Co (2004).
15. Jeff Hawkins, Dileep George, and Jamie Niemasik. 2009. Sequence memory for prediction, inference and behaviour. Philos. Trans. R. Soc. London, Ser. B 364, 1521 (2009), 1203--1209.
16. Donald Hebb. 1988. 0.(1949) The Organization of Behavior. Wiley, New York.
17. Alex Pappachen James, Irina Fedorova, Timur Ibrayev, and Dhireesha Kudithipudi. 2017. HTM spatial pooler with memristor crossbar circuits for sparse biometric recognition. IEEE Trans. Biomed. Circuits Syst. 11, 3 (2017), 640--651.
18. Sung Hyun Jo, Ting Chang, Idongesit Ebong, Bhavitavya B. Bhadviya, Pinaki Mazumder, and Wei Lu. 2010. Nanoscale memristor device as synapse in neuromorphic systems. Nano Lett. 10, 4 (2010), 1297--1301.
19. Tomasz Kulej and Fabian Khateb. 2017. Sub 0.5-V bulk-driven winner-take-all circuit based on a new voltage follower. Analog Integr. Circuits Signal Process. 90, 3 (2017), 687--691.
20. Shahar Kvatinsky, Misbah Ramadan, Eby G. Friedman, and Avinoam Kolodny. 2015. VTEAM: A general model for voltage-controlled memristors. IEEE Trans. Circuits Syst. II Express Briefs 62, 8 (2015), 786--790.
21. Alexander Lavin and Subutai Ahmad. 2015. Evaluating real-time anomaly detection algorithms—The Numenta Anomaly Benchmark. In IEEE 14th International Conference on Machine Learning and Applications (ICMLA’15). IEEE, 38--44.
22. John Lazzaro, Sylvie Ryckebusch, Misha Anne Mahowald, and Caver A. Mead. 1989. Winner-take-all networks of O (n) complexity. In Adv. Neural Inf. Process. Syst. 703--711.
23. Wei Lu, Kuk-Hwan Kim, Ting Chang, and Siddharth Gaba. 2011. Two-terminal resistive switches (memristors) for memory and logic applications. In Proceedings of the 16th Asia and South Pacific Design Automation Conference. IEEE Press, 217--223.
24. Wim J. C. Melis and Michitaka Kameyama. 2009. A study of the different uses of colour channels for traffic sign recognition on hierarchical temporal memory. In 4th International Conference on Innovative Computing, Information and Control (ICICIC’09). IEEE, 111--114.
25. Numenta. 2014. The Science of Anomaly Detection (How HTM Enables Anomaly Detection in Streaming Data).
26. Daniel E. Padilla, Russell Brinkworth, and Mark D. McDonnell. 2013. Performance of a hierarchical temporal memory network in noisy sequence learning. In IEEE International Conference on Computational Intelligence and Cybernetics (CYBERNETICSCOM’13). IEEE, 45--51.
27. Daniel E. Padilla-Baez. 2015. Analysis and Spiking Implementation of the Hierarchical Temporal Memory Model for Pattern and Sequence Recognition. Ph.D. Dissertation. University of South Australia.
28. Mirko Prezioso, Farnood Merrikh-Bayat, B. D. Hoskins, G. C. Adam, Konstantin K. Likharev, and Dmitri B. Strukov. 2015. Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nat. 521, 7550 (2015), 61--64.
29. Themistoklis Prodromakis, Boon Pin Peh, Christos Papavassiliou, and Christofer Toumazou. 2011. A versatile memristor model with nonlinear dopant kinetics. IEEE Trans. Electron Devices 58, 9 (2011), 3099--3105.
30. Shubha Ramakrishnan and Jennifer Hasler. 2014. Vector-matrix multiply and winner-take-all as an analog classifier. IEEE Trans. Very Large Scale Integr. VLSI Syst. 22, 2 (2014), 353--361.
31. Behzad Razavi. 2017. Design of Analog CMOS Integrated Circuits. McGraw-Hill.
32. Kenneth L. Rice, Tarek M. Taha, and Christopher N. Vutsinas. 2008. Hardware acceleration of image recognition through a visual cortex model. Opt. Laser Technol. 40, 6 (2008), 795--802.
33. Nicholas Soures, Abdullah M. Zyarah, Kristofor David Carlson, James Bradley Aimone, and Dhireesha Kudithipudi. 2017. How neural plasticity boosts performance of spiking neural networks. Sandia National Lab. (SNL-NM), Albuquerque, NM (United States).
34. Lennard Streat, Dhireesha Kudithipudi, and Kevin Gomez. 2016. Non-volatile hierarchical temporal memory: Hardware for spatial pooling. arXiv preprint arXiv:1611.02792.
35. Dmitri B. Strukov, Gregory S. Snider, Duncan R. Stewart, and R. Stanley Williams. 2008. The missing memristor found. Nat. 453, 7191 (2008), 80.
36. Son Ngoc Truong, Khoa Van Pham, and Kyeong-Sik Min. 2018. Spatial-pooling memristor crossbar converting sensory information to sparse distributed representation of cortical neurons. IEEE Trans. Nanotechnol. 17, 3 (2018), 482--491.
37. Pavan Vyas. 2013. Verilog implementation of a node of hierarchical temporal memory. Threshold 1, 2 (2013), 2.
38. Walt Woods, Jens Bürger, and Christof Teuscher. 2015. Synaptic weight states in a locally competitive algorithm for neuromorphic memristive hardware. IEEE Trans. Nanotechnol. 14, 6 (2015), 945--953.
39. Jianguo Xing, Tao Wang, Yang Leng, and Jun Fu. 2012. A bio-inspired olfactory model using hierarchical temporal memory. In 5th International Conference on Biomedical Engineering and Informatics (BMEI’12). IEEE, 923--927.
40. Jinxiang Zha, He Huang, and Yujie Liu. 2016. A novel window function for memristor model with application in programming analog circuits. IEEE Trans. Circuits Syst. II Express Briefs 63, 5 (2016), 423--427.
41. Wei Zhang and David J. Linden. 2003. The other side of the engram: Experience-driven changes in neuronal intrinsic excitability. Nat. Rev. Neurosci. 4, 11 (2003), 885.
42. Abdullah M. Zyarah. 2015. Design and Analysis of a Reconfigurable Hierarchical Temporal Memory Architecture. Rochester Institute of Technology.
43. Abdullah M. Zyarah and Dhireesha Kudithipudi. 2015. Reconfigurable hardware architecture of the spatial pooler for hierarchical temporal memory. In 28th IEEE International System-on-Chip Conference (SOCC’15). IEEE, 143--153.
44. Abdullah M. Zyarah and Dhireesha Kudithipudi. 2019. Neuromorphic architecture for the hierarchical temporal memory. IEEE Trans. Emerging Top. Comput. Intell. 3, 1 (2019), 4--14.
45. Abdullah M. Zyarah, Nicholas Soures, Lydia Hays, Robin B. Jacobs-Gedrim, Sapan Agarwal, Matthew Marinella, and Dhireesha Kudithipudi. 2017. Ziksa: On-chip learning accelerator with memristor crossbars for multilevel neural networks. In IEEE International Symposium on Circuits and Systems (ISCAS’17). IEEE, 1--4.

Index Terms

• Computing methodologies

  • Machine learning

    • Machine learning approaches

      • Neural networks

• Hardware

  • Emerging technologies