Real-time and Accurate Lane Detection

Watch that line!

Real-time lane detection


This project was independently done by me in summer 2018 through my internship at Aptiv, Mountain View. Above was a demo of my algorithm running on caltech lanes dataset.

This algorithm runs at 48Hz on an NVIDIA 1070Ti and Intel i5 desktop, which is state-of-the-art result as of September 2018.

In self-driving, lane marking detection is an important task. In this project, I made use of (and modified) VPGNet as the neural network for lane detection with an input of a raw RGP picture. I wrote the computer vision algorithm for post-processing of the output from VPGNet and maps the results into a bird’s eye view. The final results are stored as a set of polylines, which can be fed into a vehicle controller.

Apart from feeding into controller, we also have a seprate workflow which utilizes the lane detection for HD Map creation. This requires high accuracy instead of high speed. Below is a demo of the performance of my algorithm when it comes to a demand for high accuracy.
Lane detection for HD Map. From left to right: output from neural net, lane detection (marked in red lines) in bird's eye view, lane detection (marked in red lines) in driver's perspective

Neural Network

The neural network makes use of VPGNet. Below is an output example directly from VPGNet, without post processing.
Output from neural network

VPGNet runs in caffe. It is poorly documented. Also, redundancy is present in the network so we need some truncating. We trained the network on caltech dataset.


The post-processing for this project poses the greatest challenge. From the output (pixelwise) of neural network to a mathematical form of polylines, tons of work is required. I use houghlines algorithm, clustering algorithm, inverse perspective mapping (IPM), and a lot of filteing and adjustment to obtain the final results.

Speed Performance

In order to achieve fast performance, I spent a lot of time optimizing the code.

I truncated the original version of VPGNet and customized it for our needs. The resulted network runs at ~68Hz.

I rewrote the most time-consuming part of the project in C++ and used swig for the interface between python and C++.

Also, I optimized the code in IPM (inverse perspective mapping) and reduced redundancy in code to the largest extent.

The final algorithm runs at 48Hz on an NVIDIA 1070Ti and Intel i5 desktop, which is state-of-the-art result as of September 2018.


The final code was delivered in the form of a standalone software package for the team to use. I wrote detailed documentation regarding its usage (python flags, etc.).