Hang Qiu
Ramesh Govindan
ABSTRACT
Autonomous vehicle prototypes today come with line-of-sight depth perception sensors like 3D cameras. These 3D sensors are used for improving vehicular safety in autonomous driving, but have fundamentally limited visibility due to occlusions, sensing range, and extreme weather and lighting conditions. To improve visibility and performance, not just for autonomous vehicles but for other Advanced Driving Assistance Systems (ADAS), we explore a capability called Augmented Vehicular Reality (AVR). AVR broadens the vehicle's visual horizon by enabling it to wirelessly share visual information with other nearby vehicles, but requires the design of novel relative positioning techniques, new perspective transformation methods, approaches to isolate and predict the motion of dynamic objects in order to hide latency, and adaptive transmission strategies to cope with wireless bandwidth variability. We show that AVR is feasible using off-the-shelf wireless technologies, and it can qualitatively change the decisions made by autonomous vehicle path planning algorithms. Our AVR prototype achieves positioning accuracies that are within a few percent of car lengths and lane widths, and is optimized to process frames at 30fps.
Supplemental Material
- 18 Awesome Innovations in the New Mercedes E-Class. http://www.businessinsider.com/mercedes-e-class-2017-features-2016-6/ (Cited on page 12).Google Scholar
- F. Ahmad et al. "QuickSketch: Building 3D Representations in Unknown Environments using Crowdsourcing". In: 2018 21st International Conference on Information Fusion (Fusion). 2018, pp. 1--8 (Cited on page 5).Google Scholar
- Mani Amoozadeh et al. "Platoon Management with Cooperative Adaptive Cruise Control Enabled by VANET". In: Vehicular Communications 2.2 () (Cited on page 12). Google ScholarDigital Library
- Cheng Bo et al. "SmartLoc: Push the Limit of the Inertial Sensor Based Metropolitan Localization Using Smartphone". In: Proceedings of the 19th Annual International Conference on Mobile Computing and Networking. MobiCom '13. 2013 (Cited on page 12). Google ScholarDigital Library
- C. Chen et al. "DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving". In: 2015 IEEE International Conference on Computer Vision (ICCV). 2015, pp. 2722--2730. (Cited on page 3). Google ScholarDigital Library
- Dongyao Chen et al. "Invisible Sensing of Vehicle Steering with Smartphones". In: Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services. MobiSys '15. 2015 (Cited on page 12). Google ScholarDigital Library
- Connected Ann Arbor. http://www.mtc.umich.edu/deployments/connected-ann-arbor (Cited on page 12).Google Scholar
- DARPA Grand Challenge 2007. http://archive.darpa.mil/grandchallenge/ (Cited on page 12).Google Scholar
- Andrew J. Davison et al. "MonoSLAM: Real-Time Single Camera SLAM". In: IEEE Trans. Pattern Anal. Mach. Intell. 29.6 (June 2007) (Cited on page 12). Google ScholarDigital Library
- Falko Dressler et al. "Inter-vehicle Communication: Quo Vadis". In: IEEE Communications Magazine 52.6 (2014), pp. 170--177 (Cited on page 12).Google ScholarCross Ref
- Hugh Durrant-Whyte and Tim Bailey. "Simultaneous Localization and Mapping: Part I". In: IEEE robotics & automation magazine 13.2 (2006), pp. 99--110 (Cited on page 12).Google Scholar
- J. Engel, J. Stueckler, and D. Cremers. "Large-Scale Direct SLAM with Stereo Cameras". In: iros. 2015 (Cited on page 12).Google Scholar
- Tarak Gandhi and Mohan Trivedi. "Parametric Egomotion Estimation for Vehicle Surround Analysis using an Omnidirectional Camera". In: Machine Vision and Applications 16.2 (2005), pp. 85--95 (Cited on page 12). Google ScholarDigital Library
- Tarak Gandhi and Mohan M Trivedi. "Motion Based Vehicle Surround Analysis using an Omni-Directional Camera". In: Intelligent Vehicles Symposium, 2004 IEEE. IEEE. 2004, pp. 560--565 (Cited on page 12).Google ScholarCross Ref
- Andreas Geiger, Philip Lenz, and Raquel Urtasun. "Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite". In: Conference on Computer Vision and Pattern Recognition (CVPR). 2012 (Cited on page 7). Google ScholarDigital Library
- A. Gern, R. Moebus, and U. Franke. "Vision-based Lane Recognition Under Adverse Weather Conditions using Optical Flow". In: Intelligent Vehicle Symposium, 2002. IEEE. Vol. 2. 2002, 652--657 vol.2 (Cited on page 3).Google Scholar
- Get Under the Hood of Parker, Our Newest SOC for Autonomous Vehicles. https://blogs.nvidia.com/blog/2016/08/22/parker-for-self-driving-cars/ (Cited on page 11).Google Scholar
- P. Gomes, F. Vieira, and M. Ferreira. "The See-Through System: From Implementation to Test-drive". In: 2012 IEEE Vehicular Networking Conference (VNC). 2012, pp. 40--47. (Cited on page 12).Google ScholarCross Ref
- Google Self-Driving Car Project Monthly Report September 2016. https://static.googleusercontent.com/media/www.google.com/en//selfdrivingcar/files/reports/report-0916.pdf (Cited on page 2).Google Scholar
- Mahanth Gowda et al. "Tracking Drone Orientation with Multiple GPS Receivers". In: Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. MobiCom '16. 2016 (Cited on page 12). Google ScholarDigital Library
- Santanu Guha et al. "AutoWitness: Locating and Tracking Stolen Property While Tolerating GPS and Radio Outages". In: Proceedings of the 8th ACM Conference on Embedded Networked Sensor Systems. SenSys '10. 2010 (Cited on page 12). Google ScholarDigital Library
- Heads Up Display. http://continental-head-up-display.com/ (Cited on page 12).Google Scholar
- Wolfgang Hess et al. "Real-Time Loop Closure in 2D LIDAR SLAM". In: 2016 IEEE International Conference on Robotics and Automation (ICRA). 2016, pp. 1271--1278 (Cited on page 12).Google Scholar
- How Uber's First Self-Driving Car Works. http://www.businessinsider.com/how-ubers-driverless-cars-work-2016-9 (Cited on page 2).Google Scholar
- Jing Huang and Suya You. "Point Cloud Matching based on 3D Self-similarity". In: Computer Vision and Pattern Recognition Workshops (CVPRW), 2012 IEEE Computer Society Conference on. IEEE. 2012, pp. 41--48 (Cited on page 5).Google Scholar
- Dirk Hübner and G Riegelhuth. "A New System Architecture for Cooperative Traffic Centres-the Simtd Field Trial". In: ITS World Congress. 2012 (Cited on page 12).Google Scholar
- Bret Hull et al. "CarTel: a Distributed Mobile Sensor Computing System". In: Proceedings of the 4th international conference on Embedded networked sensor systems. SenSys '06. ACM. 2006, pp. 125--138 (Cited on page 12). Google ScholarDigital Library
- Yurong Jiang et al. "CARLOC: Precise Positioning of Automobiles". In: Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems. SenSys '15. 2015 (Cited on pages 4 and 12). Google ScholarDigital Library
- J. Kammerl et al. "Real-time Compression of Point Cloud Streams". In: 2012 IEEE International Conference on Robotics and Automation. 2012, pp. 778--785. (Cited on page 12).Google Scholar
- KITTI Visual Odometry Benchmark. http://www.cvlibs.net/datasets/kitti/eval__odometry.php (Cited on page 9).Google Scholar
- Swarun Kumar, Shyamnath Gollakota, and Dina Katabi. "A Cloud-assisted Design for Autonomous Driving". In: Proceedings of the First Edition of the MCC Workshop on Mobile Cloud Computing. MCC '12. 2012 (Cited on page 12). Google ScholarDigital Library
- Swarun Kumar et al. "CarSpeak: A Content-centric Network for Autonomous Driving". In: SIGCOMM Comput. Commun. Rev. 42.4 (Aug. 2012) (Cited on page 12). Google ScholarDigital Library
- Mathieu Labbé and François Michaud. "Online Global Loop Closure Detection for Large-scale Multi-session Graph-based Slam". In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE. 2014, pp. 2661--2666 (Cited on page 5).Google ScholarCross Ref
- Lidar-Monocular Visual Odometry. https://github.com/johannes-graeter/limo (Cited on page 12).Google Scholar
- Sathiya Kumaran Mani et al. "MNTP: Enhancing Time Synchronization for Mobile Devices". In: Proceedings of the 2016 Internet Measurement Conference. IMC '16. 2016 (Cited on page 5). Google ScholarDigital Library
- Leanne Matuszyk et al. "Stereo Panoramic Vision for Monitoring Vehicle Blind-spots". In: Intelligent Vehicles Symposium, 2004 IEEE. IEEE. 2004, pp. 31--36 (Cited on page 12).Google Scholar
- D. L. Mills. "Internet Time Synchronization: the Network Time Protocol". In: IEEE Transactions on Communications 39.10 (1991), pp. 1482--1493 (Cited on page 5).Google ScholarCross Ref
- Raul MurArtal, J. M. M. Montiel, and Juan D. Tardos. "ORB-SLAM: a Versatile and Accurate Monocular SLAM System". In: IEEE Transactions on Robotics (2015) (Cited on pages 4, 5, and 12).Google Scholar
- Richard A. Newcombe et al. "KinectFusion: Real-time Dense Surface Mapping and Tracking". In: Proceedings of the 2011 10th IEEE International Symposium on Mixed and Augmented Reality. ISMAR '11. 2011 (Cited on page 12). Google ScholarDigital Library
- NHTSA Federal Accident Reporting System. https://www-fars.nhtsa.dot.gov/Main/index.aspx (Cited on page 1).Google Scholar
- NVidia Drive PX 2. http://www.nvidia.com/object/drive-px.html (Cited on page 2).Google Scholar
- C. Olaverri-Monreal et al. "The See-Through System: A VANET-enabled Assistant for Overtaking Maneuvers". In: 2010 IEEE Intelligent Vehicles Symposium. 2010, pp. 123--128. (Cited on page 12).Google Scholar
- ORB-SLAM Code. http://webdiis.unizar.es/~raulmur/orbslam/ (Cited on page 7).Google Scholar
- H. Qiu et al. "Towards Robust Vehicular Context Sensing". In: IEEE Transactions on Vehicular Technology 67.3 (2018), pp. 1909--1922. ISSN: 0018-9545. (Cited on page 12).Google ScholarCross Ref
- Hang Qiu et al. "Augmented Vehicular Reality: Enabling Extended Vision for Future Vehicles". In: Proceedings of the 18th International Workshop on Mobile Computing Systems and Applications. ACM. 2017, pp. 67--72 (Cited on page 12). Google ScholarDigital Library
- Joseph Redmon et al. "You Only Look Once: Unified, Real-Time Object Detection". In: CoRR abs/1506.02640 (2015). URL: http://arxiv.org/abs/1506.02640 (Cited on page 3).Google Scholar
- Dirk Reichardt et al. "CarTALK 2000: Safe and Comfortable Driving based upon Inter-vehicle-communication". In: Intelligent Vehicle Symposium, 2002. IEEE. Vol. 2. IEEE. 2002, pp. 545--550 (Cited on page 12).Google Scholar
- Radu Bogdan Rusu and Steve Cousins. "3D is here: Point Cloud Library (PCL)". In: IEEE International Conference on Robotics and Automation (ICRA). Shanghai, China, 2011 (Cited on page 7).Google Scholar
- Andrzej Ruta et al. "In-vehicle Camera Traffic Sign Detection and Recognition". In: Machine Vision and Applications 22.2 (2011), pp. 359--375 (Cited on page 12). Google ScholarDigital Library
- Ruwen Schnabel and Reinhard Klein. "Octree-based Point-Cloud Compression." In: Spbg. 2006, pp. 111--120 (Cited on pages 5 and 6). Google ScholarDigital Library
- Miguel Angel Sotelo et al. "A Color Vision-Based Lane Tracking System for Autonomous Driving on Unmarked Roads". In: Auton. Robots 16.1 (Jan. 2004), pp. 95--116. ISSN: 0929-5593 (Cited on page 3). Google ScholarDigital Library
- R Stahlmann et al. "Starting European Field Tests for Car-2-X Communication: the DRIVE C2X Framework". In: 18th ITS World Congress and Exhibition. 2011 (Cited on page 12).Google Scholar
- Tesla Autopilot. http://www.businessinsider.com/how-teslas-autopilot-works-2016-7 (Cited on page 2).Google Scholar
- Arvind Thiagarajan et al. "VTrack: Accurate, Energy-aware Road Traffic Delay Estimation Using Mobile Phones". In: Proceedings of the 7th ACM Conference on Embedded Networked Sensor Systems. SenSys '09. 2009 (Cited on page 12). Google ScholarDigital Library
- Sebastian Thrun et al. "Stanley: The Robot That Won the DARPA Grand Challenge: Research Articles". In: J. Robot. Syst. 23.9 (Sept. 2006), pp. 661--692. ISSN: 0741-2223 (Cited on pages 2 and 12). Google ScholarDigital Library
- TP-Link Talon AD7200 Multi-Band WiFi Router. https://www.tp-link.com/us/products/details/cat-9_AD7200.html (Cited on page 7).Google Scholar
- Two Second Rule. https://en.wikipedia.org/wiki/Two-second_rule (Cited on page 9).Google Scholar
- C. Urmson and W. ". Whittaker. "Self-Driving Cars and the Urban Challenge". In: IEEE Intelligent Systems 23.2 (2008), pp. 66--68 (Cited on page 12). Google ScholarDigital Library
- Chris Urmson et al. "Autonomous Driving in Urban Environments: Boss and the Urban Challenge". In: J. Field Robot. 25.8 (Aug. 2008), pp. 425--466. ISSN: 1556-4959 (Cited on page 2). Google ScholarDigital Library
- U.S. Details Plans for Car-to-car Safety Communications. http://www.autonews.com/article/20140818/OEM11/140819888/u.s.-details-plans-for-car-to-car-safety-communications (Cited on page 12).Google Scholar
- V2V Safety Technology Now Standard on Cadillac CTS Sedans. http://media.cadillac.com/media/us/en/cadillac/news.detail.html/content/Pages/news/us/en/2017/mar/0309-v2v.html (Cited on page 12).Google Scholar
- Vehicle Average Length. https://www.reference.com/vehicles/average-length-car-2e853812726d079d (Cited on page 10).Google Scholar
- Vehicle Stopping Distance and Time. https://nacto.org/docs/usdg/vehicle__stopping__distance__and__time__upenn.pdf (Cited on page 9).Google Scholar
- Velodyne LiDAR HDL-64E Datasheet. http://velodynelidar.com/docs/datasheet/63-9194%20Rev-E__HDL-64E__S3__Spec%20Sheet__Web.pdf (Cited on page 2).Google Scholar
- Christoph Vogel, Konrad Schindler, and Stefan Roth. "3D Scene Flow Estimation with a Piecewise Rigid Scene Model". In: Int. J. Comput. Vision 115.1 (Oct. 2015) (Cited on page 6). Google ScholarDigital Library
- Y. Xu et al. "3D Point Cloud Map Based Vehicle Localization using Stereo Camera". In: 2017 IEEE Intelligent Vehicles Symposium (IV). 2017, pp. 487--492. (Cited on page 12).Google Scholar
- ZED Stereo Camera. https://www.stereolabs.com/ (Cited on pages 7 and 12).Google Scholar
- ZED Stereo Camera Datasheet. https://www.stereolabs.com/zed/specs/ (Cited on page 3).Google Scholar
Index Terms
- AVR: Augmented Vehicular Reality
Recommendations
Augmented Vehicular Reality: Enabling Extended Vision for Future Vehicles
HotMobile '17: Proceedings of the 18th International Workshop on Mobile Computing Systems and ApplicationsLike today's autonomous vehicle prototypes, vehicles in the future will have rich sensors to map and identify objects in the environment. For example, many autonomous vehicle prototypes today come with line-of-sight depth perception sensors like 3D ...
The Trouble with Autopilots: Assisted and Autonomous Driving on the Social Road
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing SystemsAs self-driving cars have grown in sophistication and ability, they have been deployed on the road in both localised tests and as regular private vehicles. In this paper we draw upon publicly available videos of autonomous and assisted driving (...
Priming Drivers before Handover in Semi-Autonomous Cars
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing SystemsSemi-autonomous vehicles occasionally require control to be handed over to the driver in situations where the vehicle is unable to operate safely. Currently, such handover requests require the driver to take control almost instantaneously. We ...
Comments