SenseCap: Synchronized Data Collection with Microsoft Kinect2 and LeapMotion

ABSTRACT

We present a new recording tool to capture synchronized video and skeletal data streams from cheap sensors such as the Microsoft Kinect2, and LeapMotion. While other recording tools act as virtual playback devices for testing on-line real-time applications, we target multi-media data collection for off-line processing. Images are encoded in common video formats, and skeletal data as flat text tables. This approach enables long duration recordings (e.g. over 30 minutes), and supports post-hoc mapping of the Kinect2 depth video to the color space if needed. By using common file formats, the data can be played back and analyzed on any other computer, without requiring sensor specific SDKs to be installed. The project is released under a 3-clause BSD license, and consists of an extensible C++11 framework, with support for the official Microsoft Kinect 2 and LeapMotion APIs to record, a command-line interface, and a Matlab GUI to initiate, inspect, and load Kinect2 recordings.

References

1. Z. Cheng, L. Qin, Y. Ye, Q. Huang, and Q. Tian. Human daily action analysis with multi-view and color-depth data. In Computer Vision--ECCV 2012. Workshops and Demonstrations, pages 52--61. Springer, 2012.
2. J. Han, L. Shao, D. Xu, and J. Shotton. Enhanced computer vision with microsoft kinect sensor: A review. Cybernetics, IEEE Transactions on, 43(5):1318--1334, 2013.
3. S. Izadi et al. Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 559--568. ACM, 2011.
4. J. F. P. Kooij and J. C. van Gemert. Depth-aware motion magnification. In European Conference on Computer Vision (ECCV), 2016. Springer, (to appear).
5. G. Marin, F. Dominio, and P. Zanuttigh. Hand gesture recognition with leap motion and kinect devices. In Image Processing (ICIP), 2014 IEEE International Conference on, pages 1565--1569. IEEE, 2014.
6. A. Prest, C. Leistner, J. Civera, C. Schmid, and V. Ferrari. Learning object class detectors from weakly annotated video. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 3282--3289. IEEE, 2012.
7. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211--252, 2015.
8. J. Shotton, T. Sharp, A. Kipman, A. Fitzgibbon, M. Finocchio, A. Blake, M. Cook, and R. Moore. Real-time human pose recognition in parts from single depth images. Communications of the ACM, 56(1):116--124, 2013.
9. S. Song, S. P. Lichtenberg, and J. Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 567--576, 2015.
10. F. Weichert, D. Bachmann, B. Rudak, and D. Fisseler. Analysis of the accuracy and robustness of the leap motion controller. Sensors, 13(5):6380--6393, 2013.

Index Terms

• Hardware

  • Communication hardware, interfaces and storage

    • Sensor applications and deployments

• Software and its engineering

  • Software organization and properties

    • Software system structures

      • Software architectures

        • Data flow architectures

        • Object oriented architectures