End to End Learning for Self-Driving Cars

End to End Learning for Self-Driving Cars Mariusz Bojarski Davide Del Testa Daniel Dworakowski Bernhard Firner

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

End to End Learning for Self-Driving Cars 2016 arXiv (NVIDIA technical report) 1604.07316 bojarski2016-self-driving bojarski2016-self-driving.txt, the project's staged copy 232fcab40e13c827c2b8541902ce6624aa0c6a5d arxiv-open robotics

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them method Quotation withheld because this record is not cleared for redistribution. end-to-end learning of lane and road following Quotation withheld because this record is not cleared for redistribution. system-or-tool-building system-or-tool-building Quotation withheld because this record is not cleared for redistribution. mechanisms the steering command is stored as 1/r rather than r, which removes the singularity at straight-ahead and makes the label continuous through zero across left and right turns Quotation withheld because this record is not cleared for redistribution. mechanisms recovery is taught rather than encountered: the training set is augmented with images of the car already displaced from the lane centre, because human driving alone never shows the network a mistake Quotation withheld because this record is not cleared for redistribution. mechanisms off-centre viewpoints are synthesised under an approximation the paper states it knows to be wrong - everything below the horizon treated as flat ground, everything above it as infinitely far - accepted because the resulting distortions do not hurt training Quotation withheld because this record is not cleared for redistribution. mechanisms the label for a synthesised viewpoint is not the driver's command but the command that would return the car to the correct place within a fixed two-second horizon Quotation withheld because this record is not cleared for redistribution. mechanisms the training frames are filtered by the driver's activity label and then downsampled, because consecutive frames at full rate are near-duplicates Quotation withheld because this record is not cleared for redistribution. mechanisms the perturbations are drawn from a normal distribution deliberately twice as wide as the variation measured in human drivers Quotation withheld because this record is not cleared for redistribution. mechanisms autonomy is computed from a COUNTED event and an ASSUMED constant: each simulated intervention is charged a fixed six seconds of human takeover time Quotation withheld because this record is not cleared for redistribution. mechanisms the normalisation layer is inside the network but hard-coded, so it is not learned and can be changed with the architecture and run on the GPU Quotation withheld because this record is not cleared for redistribution. more work is needed to improve robustness and to verify/visualize the network's internal processing what this limits: the deployment readiness of the system: Quotation withheld Quotation withheld because this record is not cleared for redistribution. The viewpoint transformation used for augmentation is an approximation because the true 3D scene geometry is unknown. what this limits: every augmented training example: PRECISE VIEWPOINT TRANSFORMATION REQUIRES 3D SCENE KNOWLEDGE WHICH WE DON'T HAVE Quotation withheld because this record is not cleared for redistribution. That approximation distorts objects standing above the ground plane. what this limits: the augmentation for anything but flat road: IT INTRODUCES DISTORTIONS FOR OBJECTS THAT STICK ABOVE THE GROUND, SUCH AS CARS, POLES, TREES, AND BUILDINGS - the distorted objects are the ones that matter Quotation withheld because this record is not cleared for redistribution. Larger augmentation magnitudes introduce undesirable artifacts into the training data. what this limits: the benefit of larger augmentation: ARTIFICIALLY AUGMENTING THE DATA DOES ADD UNDESIRABLE ARTIFACTS AS THE MAGNITUDE INCREASES Quotation withheld because this record is not cleared for redistribution. Robustness of the network, methods to verify it, and internal-state visualisation all remain open. what this limits: any safety or verification claim: MORE WORK IS NEEDED ... TO FIND METHODS TO VERIFY THE ROBUSTNESS, AND TO IMPROVE VISUALIZATION OF THE NETWORK-INTERNAL PROCESSING STEPS Quotation withheld because this record is not cleared for redistribution. The results are explicitly preliminary. what this limits: the standing of every result reported: THIS PAPER DESCRIBES PRELIMINARY RESULTS OF THIS NEW EFFORT Quotation withheld because this record is not cleared for redistribution. The reported autonomy percentage is built on an assumed six-second cost per intervention, which the authors state as an assumption rather than measure. what this limits: any autonomy percentage from this paper: it multiplies the intervention count by an assumed six seconds - the time the authors take a human to retake control, re-centre and restart self-steering - so the figure moves with that assumption Quotation withheld because this record is not cleared for redistribution. An intervention is a threshold crossing in simulation, not a human decision: the car counts as having failed when it passes one metre from the centre line. what this limits: the intervention count behind the autonomy figure: an intervention is triggered automatically when the simulated vehicle departs the centre line by more than one metre, so the number depends on that threshold Quotation withheld because this record is not cleared for redistribution. After every simulated intervention the car is placed back on the recorded ground truth, so the measure cannot observe how a failure would have developed. what this limits: simulated autonomy as evidence of safety: after each intervention the vehicle is reset to the ground truth of the original video, so errors never compound and the metric counts how often the car left the lane rather than how badly Quotation withheld because this record is not cleared for redistribution. a convolutional network taking raw camera pixels, used in place of the perception-plus-control pipeline of robotics so that steering is learned end to end Quotation withheld because this record is not cleared for redistribution. manually calibrating the lane centre for each frame used by the simulator, rather than taking the human driver's position as the target eliminates: the human drivers' own off-centre driving being learned as the correct steering target · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. convolutional neural network convolutional neural network Quotation withheld because this record is not cleared for redistribution. autonomy autonomy Quotation withheld because this record is not cleared for redistribution. intervention intervention Quotation withheld because this record is not cleared for redistribution. road features road features Quotation withheld because this record is not cleared for redistribution. training signal training signal Quotation withheld because this record is not cleared for redistribution. autonomy - AS A STIPULATED FORMULA: one minus the intervention count times six seconds over elapsed time, as a percentage; the word names an arithmetic result, not an observed capability Quotation withheld because this record is not cleared for redistribution. a performance criterion - CHOSEN FOR LEGIBILITY, WHICH IS NOT THE SAME AS BEING RIGHT: the paper's argument for end-to-end training is that human-selected intermediate criteria are picked because a person can interpret them, and that this does not guarantee system performance Quotation withheld because this record is not cleared for redistribution. road features - LEARNED WITHOUT EVER BEING NAMED: the network detects them from the steering angle alone, and the authors state they never trained it to detect road outlines Quotation withheld because this record is not cleared for redistribution. builds-on → Pomerleau ALVINN (1989) Quotation withheld because this record is not cleared for redistribution. builds-on → DARPA Autonomous Vehicle (DAVE) [5] Quotation withheld because this record is not cleared for redistribution. builds-on → Pomerleau [6], the ALVINN system Quotation withheld because this record is not cleared for redistribution. extends → Pomerleau [6], the ALVINN system Quotation withheld because this record is not cleared for redistribution. autonomy —operationalised-as→ intervention autonomy intervention strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. training signal —causes→ road features training signal road features strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the network learns to detect road features with steering angle as its only training signal; causes is loose. Quotation withheld because this record is not cleared for redistribution. 98% — fraction of time the network drove autonomously (Holmdel to Atlantic Highlands) Quotation withheld because this record is not cleared for redistribution. 30 — system operating frame rate Quotation withheld because this record is not cleared for redistribution. 9 — network depth: 1 normalization + 5 convolutional + 3 fully connected Quotation withheld because this record is not cleared for redistribution. 10 — distance driven on the Garden State Parkway with zero interventions Quotation withheld because this record is not cleared for redistribution. A CNN learned to turn raw pixels from one front-facing camera straight into steering commands; the end-to-end approach proved surprisingly powerful. Quotation withheld because this record is not cleared for redistribution. The network learned useful road features using Quotation withheld, never explicitly supervised to detect them. Quotation withheld because this record is not cleared for redistribution. raw pixels raw pixels Quotation withheld because this record is not cleared for redistribution. steering angle steering angle Quotation withheld because this record is not cleared for redistribution. autonomy autonomy Quotation withheld because this record is not cleared for redistribution. autonomy autonomy Quotation withheld because this record is not cleared for redistribution. end-to-end learning end-to-end learning Quotation withheld because this record is not cleared for redistribution. standard-deviation standard-deviation Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://arxiv.org/abs/1604.07316.