Ehsan Hesamifard
ABSTRACT
Deep Neural Networks (DNNs) have overtaken classic machine learning algorithms due to their superior performance in big data analysis in a broad range of applications. On the other hand, in recent years Machine Learning as a Service (MLaaS) has become more widespread in which a client uses cloud services for analyzing its data. However, the client's data may be sensitive which raises privacy concerns. In this paper, we address the issue of privacy preserving classification in a Machine Learning as a Service (MLaaS) settings and focus on convolutional neural networks (CNN). To achieve this goal, we develop new techniques to run CNNs over encrypted data. First, we design methods to approximate commonly used activation functions in CNNs (i.e. ReLU, Sigmoid, and Tanh) with low degree polynomials which is essential for a practical and efficient solution. Then, we train CNNs with approximation polynomials instead of original activation functions and implement CNNs classification over encrypted data. We evaluate the performance of our modified models at each step. The results of our experiments using several CNNs with a varying number of layers and structures are promising. When applied to the MNIST optical character recognition tasks, our approach achieved 99.25% accuracy which significantly outperforms state-of-the-art solutions and is close to the accuracy of the best non-private version. Furthermore, it can make up to 164000 predictions per hour. These results show that our approach provides accurate, efficient, and scalable privacy-preserving predictions in CNNs.
- Louis J. M. Aslett, Pedro M. Esperança, and Chris C. Holmes. 2015. Encrypted statistical machine learning: new privacy preserving methods. CoRR (2015).Google Scholar
- L. J. M. Aslett, P. M. Esperança, and C. C. Holmes. 2015. A review of homomorphic encryption and software tools for encrypted statistical machine learning. Technical Report. University of Oxford.Google Scholar
- K. Atkinson and W. Han. 2009. Theoretical Numerical Analysis: A Functional Analysis Framework. Springer New York.Google Scholar
- James Bergstra and Yoshua Bengio. 2012. Random Search for Hyper-parameter Optimization. J. Mach. Learn. Res. 13 (Feb. 2012), 281--305. http://dl.acm.org/ citation.cfm?id=2188385.2188395 Google ScholarDigital Library
- Raphael Bost, Raluca Ada Popa, Stephen Tu, and Shafi Goldwasser. 2015. Machine Learning Classification over Encrypted Data. In 22nd Annual Network and Distributed System Security Symposium, NDSS, San Diego, California, USA.Google ScholarCross Ref
- Zvika Brakerski, Craig Gentry, and Vinod Vaikuntanathan. 2012. (Leveled) Fully Homomorphic Encryption Without Bootstrapping. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference (ITCS '12). ACM, New York, NY, USA, 309--325. Google ScholarDigital Library
- Fanyu Bu, Yu Ma, Zhikui Chen, and Han Xu. 2015. Privacy Preserving Back- Propagation Based on BGV on Cloud. In 17th IEEE International Conference on High Performance Computing and Communications, HPCC 2015, 7th IEEE International Symposiumon Cyberspace Safety and Security, CSS 2015, and 12th IEEE International Conference on Embedded Software and Systems, ICESS 2015, New York, NY, USA, August 24--26, 2015. 1791--1795. Google ScholarDigital Library
- Hervé Chabanne, Amaury de Wargny, Jonathan Milgram, Constance Morel, and Emmanuel Prouff. 2017. Privacy-Preserving Classification on Deep Neural Network. Cryptology ePrint Archive, Report 2017/035.Google Scholar
- Nishanth Chandran, Divya Gupta, Aseem Rastogi, Rahul Sharma, and Shardul Tripathi. 2017. EzPC: Programmable, Efficient, and Scalable Secure Two-Party Computation. IACR Cryptology ePrint Archive 2017 (2017), 1109.Google Scholar
- Jung Hee Cheon, Andrey Kim, Miran Kim, and Yongsoo Song. 2016. Homomorphic Encryption for Arithmetic of Approximate Numbers. Cryptology ePrint Archive, Report 2016/421. https://eprint.iacr.org/2016/421.Google Scholar
- François Chollet et al. 2017. Keras. https://github.com/fchollet/keras.Google Scholar
- Jack L.H. Crawford, Craig Gentry, Shai Halevi, Daniel Platt, and Victor Shoup. 2018. Doing Real Work with FHE: The Case of Logistic Regression. Cryptology ePrint Archive, Report 2018/202. (2018). https://eprint.iacr.org/2018/202.Google Scholar
- Daniel Demmler, Thomas Schneider, and Michael Zohner. 2015. ABY - A Framework for Efficient Mixed-Protocol Secure Two-Party Computation. In 22nd Annual Network and Distributed System Security Symposium, NDSS 2015, San Diego, California, USA, February 8--11, 2015.Google Scholar
- Nathan Dowlin, Ran Gilad-Bachrach, Kim Laine, Kristin Lauter Michael Naehrig, and John Wernsing. 2016. CryptoNets: Applying Neural Networks to Encrypted Data with High Throughput and Accuracy. Technical Report MSR-TR-2016--3.Google Scholar
- Craig Gentry. 2009. A Fully Homomorphic Encryption Scheme. Ph.D. Dissertation. Stanford, CA, USA. Advisor(s) Boneh, Dan. AAI3382729. Google Scholar
- Google. 2017. Google Prediction API. (2017). https://cloud.google.com/ prediction/Google Scholar
- Thore Graepel, Kristin Lauter, and Michael Naehrig. 2013. ML Confidential: Machine Learning on Encrypted Data. In Proceedings of the 15th International Conference on Information Security and Cryptology (ICISC'12). Springer-Verlag. Google ScholarDigital Library
- Shai Halevi and Victor Shoup. 2014. Algorithms in HElib. In Advances in Cryptology - CRYPTO - 34th Annual Cryptology Conference, CA, USA, Proceedings.Google Scholar
- Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan. 2018. Gazelle: A Low Latency Framework for Secure Neural Network Inference. CoRR abs/1801.05507 (2018). arXiv:1801.05507 http://arxiv.org/abs/1801.05507Google Scholar
- Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A Convolutional Neural Network for Modelling Sentences. CoRR abs/1404.2188 (2014).Google Scholar
- Andrey Kim, Yongsoo Song, Miran Kim, Keewoo Lee, and Jung Hee Cheon. 2018. Logistic Regression Model Training based on the Approximate Homomorphic Encryption. Cryptology ePrint Archive, Report 2018/254. (2018). https://eprint. iacr.org/2018/254.Google Scholar
- Miran Kim, Yongsoo Song, Shuang Wang, Yuhou Xia, and Xiaoqian Jiang. 2018. Secure Logistic Regression Based on Homomorphic Encryption: Design and Evaluation. Cryptology ePrint Archive, Report 2018/074. (2018). https://eprint. iacr.org/2018/074.Google Scholar
- Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. 2019. CIFAR-10 (Canadian Institute for Advanced Research). (2019). www.cs.toronto.edu/~kriz/cifar.htmlGoogle Scholar
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.). Curran Associates, Inc. Google ScholarDigital Library
- Ersatz Labs. 2017. Ersatz. (2017). http://www.ersatzlabs.com/Google Scholar
- Yann LeCun and Corinna Cortes. 2010. MNIST handwritten digit database. (2010). http://yann.lecun.com/exdb/mnist/Google Scholar
- Jian Liu, Mika Juuti, Yao Lu, and N. Asokan. 2017. Oblivious Neural Network Predictions via MiniONN Transformations. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS '17). ACM, New York, NY, USA, 619--631. Google ScholarDigital Library
- Roi Livni, Shai Shalev-Shwartz, and Ohad Shamir. 2014. On the Computational Efficiency of Training Neural Networks. CoRR (2014).Google Scholar
- Microsft. 2017. Microsoft Azure Machine Learning. (2017). https://azure. microsoft.com/en-us/services/machine-learning/Google Scholar
- Microsft. 2018. Simple encrypted arithmetic library - SEAL. (2018). https: //sealcrypto.orgGoogle Scholar
- P. Mohassel and Y. Zhang. 2017. SecureML: A System for Scalable Privacy- Preserving Machine Learning. In IEEE Symposium on Security and Privacy (SP).Google Scholar
- Pascal Paillier. 1999. Public-key Cryptosystems Based on Composite Degree Residuosity Classes. In 17th International Conference on Theory and Application of Cryptographic Techniques (EUROCRYPT'99). Springer-Verlag, Berlin, Heidelberg. Google ScholarDigital Library
- M. Sadegh Riazi, ChristianWeinert, Oleksandr Tkachenko, Ebrahim M. Songhori, Thomas Schneider, and Farinaz Koushanfar. 2018. Chameleon: A Hybrid Secure Computation Framework for Machine Learning Applications. CoRR abs/1801.03239 (2018). arXiv:1801.03239 http://arxiv.org/abs/1801.03239Google ScholarDigital Library
- R.L. Rivest, L. Adleman, and M.L. Dertouzos. 1978. On data banks and privacy homomorphisms. In Foundations on Secure Computation, Academia Press.Google Scholar
- Kurt Rohloff. accessed August 2018. The PALISADE lattice cryptography library. Retrieved from https://git.njit.edu/palisade/PALISADE.Google Scholar
- Bita Darvish Rouhani, M. Sadegh Riazi, and Farinaz Koushanfar. 2017. DeepSecure: Scalable Provably-Secure Deep Learning. CoRR abs/1705.08963 (2017).Google Scholar
- Thomas Shortell and Ali Shokoufandeh. 2015. Secure Signal Processing Using Fully Homomorphic Encryption. In Advanced Concepts for Intelligent Vision Systems - 16th International Conference, ACIVS, Italy, Proceedings. Google ScholarDigital Library
- Turi. 2017. GraphLab. (2017). http://www.select.cs.cmu.edu/code/graphlab/Google Scholar
- Pengtao Xie, Misha Bilenko, Tom Finley, Ran Gilad-Bachrach, Kristin E. Lauter, and Michael Naehrig. 2014. Crypto-Nets: Neural Networks over Encrypted Data. CoRR (2014).Google Scholar
- Yuan Xu. 2001. Orthogonal Polynomials of Several Variables. Encyclopedia of Mathematics and its Applications 81 (2001).Google Scholar
- Q. Zhang, L. T. Yang, and Z. Chen. 2016. Privacy Preserving Deep Computation Model on Cloud for Big Data Feature Learning. IEEE Trans. Comput. 65, 5 (May 2016). Google ScholarDigital Library
Index Terms
- Deep Neural Networks Classification over Encrypted Data
Recommendations
Research on improved wavelet convolutional wavelet neural networks
AbstractConvolutional neural network (CNN) is recognized as state of the art of deep learning algorithm, which has a good ability on the image classification and recognition. The problems of CNN are as follows: the precision, accuracy and efficiency of ...
Deep CNN for Classification of Image Contents
IPMV '21: Proceedings of the 2021 3rd International Conference on Image Processing and Machine VisionIn recent years the classification of images has made great progress and has been used in many fields. However, it may not be possible to classify images perfectly through the CNN because of overfitting and gradient vanishing. Most existing CNNs have ...
Towards a Better Compromise Between Shallow and Deep CNN for Binary Classification Problems of Unstructured Data
Machine Learning for NetworkingAbstractDeep Neural Network is a large scale neural network. Deep Learning, refers to training very large Neural Networks in order to discover good representations, at multiple levels, with higher-level learned features. The rise of deep learning is ...
Reviews
Fjodor J. Ruzic
When we speak about a convolutional neural network (CNN) as a more complex deep learning algorithm, there are privacy-preserving issues that could be addressed in any study on the topic. In deep learning, CNNs are used to analyze complex cloud data, where privacy preservation is a big challenge to machine learning classification services. Thus, the authors have developed a technique that provides functionality and efficiency for privacy-preserving classification in machine learning as a service (MLaaS) when cloud services for analyzing big data are in use. Further, deep learning and especially CNNs are more effective than traditional computer algorithms, meaning they are more accurate than a machine learning algorithm in a big data classification. CNNs as deep artificial neural networks (ANNs) are used in many cases to classify complex big data. However, it is hard to achieve an acceptable level of privacy preservation within CNNs that use sensitive data over the cloud. This issue makes the study more valuable, because it is strongly focused on privacy preserving with a well-defined model. It is evidently clear that "preserving the privacy of sensitive data in different machine learning algorithms" exists in theory and real-life deployment. However, when we are confronted with huge datasets located in the cloud, there are many challenges to keeping data anonymous. In addition to a brief introduction to deep learning and CNNs, the authors describe the model and training environment, focusing on a client-server structure where the encrypted data is in the communication process, thus ensuring invisibility of input data. In this way, the proposed model provides privacy preservation in the classification process during the prediction outputs. Literature on deep learning exists in many fields; however, research on data encrypted training models using deep neural networks and especially CNNs is rare. Thus, this study deserves particular attention from readers interested in system privacy. "Protect[ing] the privacy of the data during the learning process," using the appropriate "encryption scheme to support the secure computation of the high-order back-propagation algorithm efficiently for deep computation model training on the cloud," is only the starting point of the study. Because operations in a neural network can be implemented over encrypted data but are not practical within standard neuron schemes, the authors propose other complementary functions to operate over encrypted data, ensuring privacy preservation. The approach is presented in a very clear way, making the privacy-preserving classification process through CNNs accessible to many readers. It is definitely recommended reading for computer science (CS) students and professionals in artificial intelligence (AI)-supported machine learning development.
Access critical reviews of Computing literature here
Become a reviewer for Computing Reviews.
Comments