End to End Learning for Self-Driving Cars

Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner · arXiv (NVIDIA technical report) · 2016 · robotics · record bojarski2016-self-driving
tag layer · index · concepts · authors · עברית

Where this record's text came from. faceted 2026-08 by reading the locally held text (pdftext); every evidence string verified as an exact substring of that text before merging. An NVIDIA technical report rather than a peer-reviewed paper, and it says so in effect ('preliminary results'), which is recorded as a limitation. The reported autonomy figures (98% on one route, 10 zero-intercept highway miles) come from a small, informally specified test set with no statistics, error bars or baseline comparison — a weakness the facets cannot express because the paper never states it. method is system-or-tool-building: the work builds a driving system and evaluates it in a purpose-built simulator and then on public roads. Two relations point at the same target (ALVINN) because the paper both credits it as inspiration and explicitly differentiates from it in the next sentence. 2026-08-13, 2026-08-14 and 2026-08-16 schema-0.3 passes: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node. Update, 15 September 2026: EXCEPTION, measured with the merge gate's own predicate (Paper.locate): 2 quote(s) on this record are not found as one exact string in the current staged text - limitations[9].evidence, limitations[9].for_which_use_evidence. The sentence above does not hold for them as written. Where a row records why - the quote is grounded in another copy of the paper, joins two passages with an ellipsis, or differs from the staged text by extraction noise - that note stays on the row. Update, 16 September 2026: EXCEPTION TO THE DATED PASS ABOVE, on this record only: this record carries 1 composite quote field - a quote assembled from two or more passages joined by an ellipsis. A COMPOSITE IS NOT AN EXACT CONTIGUOUS SUBSTRING of the staged text and is deliberately not anchored: each of its parts is verified separately at the offsets in evidence composite spans, and a part occurring more than once carries span null rather than a guessed offset. So the 'every quote verified as an exact substring' clause of the dated pass above DOES NOT HOLD for those fields, and never did - the parts were verified, the whole was not. Nothing is deleted, re-grounded or withdrawn: this writes down an exception that was always true of composites and had not been recorded, so a reader was told more than the pass had checked.

How to read this page. The article itself is not reproduced here — this paper is not cleared for redistribution, so what follows is the tag layer alone. The evidence sentences are withheld with the article. The table at the foot lists every tag the record carries; each row names the facet it belongs to, and the whose words column says whether the tag records the paper's own wording or the tagger's sentence about the paper. Every tag links to each other paper in the corpus carrying it. The machine-readable edition of the same thing is the TEI file beside this one, where the tags live in standOff, and the article is at its publisher via the link below. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.

Other fields define these differently the same word, defined another way in another discipline
deep learning 11 fields · autonomy 8 fields · training 3 fields · standard-deviation 3 fields · supervised learning 3 fields · convolutional neural network 2 fields · and 1 more →

Tags on this paper

Every tag the record carries, each linking to every other paper in the corpus that carries it. A tag appears here whether or not its words occur in the article — the spans below are only the ones a sentence in this paper licenses. Of the 9 concept tags on this paper, 5 are grounded in a quoted sentence.

convolutional neural network deep learning end-to-end learning autonomous vehicle supervised learning autonomy intervention road features training signal the steering command is stored as 1/r rather than r, which removes the singularity at straight-ahead and makes the label continuous through zero across left and right turns recovery is taught rather than encountered: the training set is augmented with images of the car already displaced from the lane centre, because human driving alone never shows the network a mistake off-centre viewpoints are synthesised under an approximation the paper states it knows to be wrong - everything below the horizon treated as flat ground, everything above it as infinitely far - accepted because the resulting distortions do not hurt training the label for a synthesised viewpoint is not the driver's command but the command that would return the car to the correct place within a fixed two-second horizon the training frames are filtered by the driver's activity label and then downsampled, because consecutive frames at full rate are near-duplicates the perturbations are drawn from a normal distribution deliberately twice as wide as the variation measured in human drivers autonomy is computed from a COUNTED event and an ASSUMED constant: each simulated intervention is charged a fixed six seconds of human takeover time the normalisation layer is inside the network but hard-coded, so it is not learned and can be changed with the architecture and run on the GPU convolutional steering training network camera road image features raw pixels steering angle standard-deviation standard deviation system-or-tool-building

Legend

relation to other work	`claim_relations`	how this paper stands to another: agrees, contradicts, extends, qualifies
claim type	`claim_type`	the shape of the claim: existence, mechanism, method, critique, prediction, null
concept edge	`concept_edges`	an asserted relation between two concepts, with the sentence asserting it
concept	`concepts`	an idea the paper works with, quoted where the paper uses it
content tag	`content_tags`	a finding, definition, quantity, statistic or variable named in the paper
control	`controls`	an experiment run to eliminate a rival explanation
limitation	`limitations`	what the paper says its evidence cannot support
mechanism	`mechanisms`	how the reported effect is said to come about
method	`method`	how the paper produced its evidence
phenomenon	`phenomenon`	the effect or regularity under study
sense	`senses`	a distinct use of a term, quoted where the paper fixes it
transfer	`transfers`	a method or concept carried in from another field

The paper

The full text is not reproduced here. This article is not cleared for redistribution, so the page carries the tag layer only. Its evidence sentences are withheld; the grounding status remains with each tag, and the article itself is at https://arxiv.org/abs/1604.07316. The reading view exists for the papers whose licence permits it.

Every tag on this page

claim_relations 4 · claim_type 1 · concept_edges 2 · concepts 5 · content_tags 12 · controls 1 · limitations 9 · mechanisms 8 · method 1 · phenomenon 1 · senses 3 · transfers 1

Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; none of 2 relations here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.

facet	tag	field	conf.	whose words
claim_type	method Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_type`		the paper's words
phenomenon	end-to-end learning of lane and road following Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`phenomenon`		the paper's words
method	system-or-tool-building Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: system-or-tool-building	`method`		the paper's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: the steering command is stored as 1/r rather than r, which removes the singularity at straight-ahead and makes the label continuous through zero across left and right turns	`mechanisms[0]`	0.9	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: recovery is taught rather than encountered: the training set is augmented with images of the car already displaced from the lane centre, because human driving alone never shows the network a mistake	`mechanisms[1]`	0.9	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: off-centre viewpoints are synthesised under an approximation the paper states it knows to be wrong - everything below the horizon treated as flat ground, everything above it as infinitely far - accepted because the resulting distortions do not hurt training	`mechanisms[2]`	0.9	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: the label for a synthesised viewpoint is not the driver's command but the command that would return the car to the correct place within a fixed two-second horizon	`mechanisms[3]`	0.9	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: the training frames are filtered by the driver's activity label and then downsampled, because consecutive frames at full rate are near-duplicates	`mechanisms[4]`	0.9	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: the perturbations are drawn from a normal distribution deliberately twice as wide as the variation measured in human drivers	`mechanisms[5]`	0.9	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: autonomy is computed from a COUNTED event and an ASSUMED constant: each simulated intervention is charged a fixed six seconds of human takeover time	`mechanisms[6]`	0.9	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: the normalisation layer is inside the network but hard-coded, so it is not learned and can be changed with the architecture and run on the GPU	`mechanisms[7]`	0.9	the tagger's words
limitations	more work is needed to improve robustness and to verify/visualize the network's internal processing Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the deployment readiness of the system: Quotation withheld	`limitations[0]`		the tagger's words
limitations	The viewpoint transformation used for augmentation is an approximation because the true 3D scene geometry is unknown. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: every augmented training example: PRECISE VIEWPOINT TRANSFORMATION REQUIRES 3D SCENE KNOWLEDGE WHICH WE DON'T HAVE	`limitations[1]`		the tagger's words
limitations	That approximation distorts objects standing above the ground plane. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the augmentation for anything but flat road: IT INTRODUCES DISTORTIONS FOR OBJECTS THAT STICK ABOVE THE GROUND, SUCH AS CARS, POLES, TREES, AND BUILDINGS - the distorted objects are the ones that matter	`limitations[2]`		the tagger's words
limitations	Larger augmentation magnitudes introduce undesirable artifacts into the training data. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the benefit of larger augmentation: ARTIFICIALLY AUGMENTING THE DATA DOES ADD UNDESIRABLE ARTIFACTS AS THE MAGNITUDE INCREASES	`limitations[3]`		the tagger's words
limitations	Robustness of the network, methods to verify it, and internal-state visualisation all remain open. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any safety or verification claim: MORE WORK IS NEEDED ... TO FIND METHODS TO VERIFY THE ROBUSTNESS, AND TO IMPROVE VISUALIZATION OF THE NETWORK-INTERNAL PROCESSING STEPS	`limitations[4]`		the tagger's words
limitations	The results are explicitly preliminary. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the standing of every result reported: THIS PAPER DESCRIBES PRELIMINARY RESULTS OF THIS NEW EFFORT	`limitations[5]`		the tagger's words
limitations	The reported autonomy percentage is built on an assumed six-second cost per intervention, which the authors state as an assumption rather than measure. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any autonomy percentage from this paper: it multiplies the intervention count by an assumed six seconds - the time the authors take a human to retake control, re-centre and restart self-steering - so the figure moves with that assumption	`limitations[6]`	1.0	the tagger's words
limitations	An intervention is a threshold crossing in simulation, not a human decision: the car counts as having failed when it passes one metre from the centre line. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the intervention count behind the autonomy figure: an intervention is triggered automatically when the simulated vehicle departs the centre line by more than one metre, so the number depends on that threshold	`limitations[7]`	1.0	the tagger's words
limitations	After every simulated intervention the car is placed back on the recorded ground truth, so the measure cannot observe how a failure would have developed. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: simulated autonomy as evidence of safety: after each intervention the vehicle is reset to the ground truth of the original video, so errors never compound and the metric counts how often the car left the lane rather than how badly	`limitations[8]`	1.0	the tagger's words
transfers	a convolutional network taking raw camera pixels, used in place of the perception-plus-control pipeline of robotics so that steering is learned end to end Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`transfers[0]`	0.85	the tagger's words
controls	manually calibrating the lane centre for each frame used by the simulator, rather than taking the human driver's position as the target Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. eliminates: the human drivers' own off-centre driving being learned as the correct steering target · outcome (stated by the tagger, no quotation): held	`controls[0]`	0.8	the tagger's words
concepts	convolutional neural network Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: convolutional neural network	`concepts[0]`		the paper's words
concepts	autonomy Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: autonomy	`concepts[5]`		the paper's words
concepts	intervention Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: intervention	`concepts[6]`		the paper's words
concepts	road features Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: road features	`concepts[7]`		the paper's words
concepts	training signal Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: training signal	`concepts[8]`		the paper's words
senses	autonomy - AS A STIPULATED FORMULA: one minus the intervention count times six seconds over elapsed time, as a percentage; the word names an arithmetic result, not an observed capability Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[0]`	1.0	the tagger's words
senses	a performance criterion - CHOSEN FOR LEGIBILITY, WHICH IS NOT THE SAME AS BEING RIGHT: the paper's argument for end-to-end training is that human-selected intermediate criteria are picked because a person can interpret them, and that this does not guarantee system performance Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[1]`	1.0	the tagger's words
senses	road features - LEARNED WITHOUT EVER BEING NAMED: the network detects them from the steering angle alone, and the authors state they never trained it to detect road outlines Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[2]`	1.0	the tagger's words
claim_relations	builds-on → Pomerleau ALVINN (1989) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[0]`		the tagger's words
claim_relations	builds-on → DARPA Autonomous Vehicle (DAVE) [5] Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[1]`		the tagger's words
claim_relations	builds-on → Pomerleau [6], the ALVINN system Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[2]`		the tagger's words
claim_relations	extends → Pomerleau [6], the ALVINN system Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[3]`		the tagger's words
concept_edges	autonomy —operationalised-as→ intervention Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 also in: autonomy · intervention	`concept_edges[0]`		the tagger's words
concept_edges	training signal —causes→ road features Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the network learns to detect road features with steering angle as its only training signal; causes is loose. also in: training signal · road features	`concept_edges[1]`		the tagger's words
content_tags	98% — fraction of time the network drove autonomously (Holmdel to Atlantic Highlands) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[0]`		the tagger's words
content_tags	30 — system operating frame rate Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[1]`		the tagger's words
content_tags	9 — network depth: 1 normalization + 5 convolutional + 3 fully connected Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[2]`		the tagger's words
content_tags	10 — distance driven on the Garden State Parkway with zero interventions Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[3]`		the tagger's words
content_tags	A CNN learned to turn raw pixels from one front-facing camera straight into steering commands; the end-to-end approach proved surprisingly powerful. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[0]`		the tagger's words
content_tags	The network learned useful road features using Quotation withheld, never explicitly supervised to detect them. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[1]`		the tagger's words
content_tags	raw pixels Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: raw pixels	`content_tags.variables[0]`		the tagger's words
content_tags	steering angle Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: steering angle	`content_tags.variables[1]`		the tagger's words
content_tags	autonomy Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: autonomy	`content_tags.variables[2]`		the tagger's words
content_tags	autonomy Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: autonomy	`content_tags.definitions[0]`		the tagger's words
content_tags	end-to-end learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: end-to-end learning	`content_tags.definitions[1]`		the tagger's words
content_tags	standard-deviation Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: standard-deviation	`content_tags.statistics[0]`		the tagger's words

Interpretive choices

Tags that could not be placed in the text:

limitations[9] — quotation withheld — the page has already reproduced as much of this paper as the rights budget allows
this quote's status: quote-not-in-current-staged-text · about this quote: marked quote-not-in-current-staged-text - the quote is a DECLARED COMPOSITE (the composite-quote note), so the whole string is not one span of the staged text; each of its two parts is verified there, exactly once. · what this limits: the trained network as a model of driving: only frames where the driver was staying in a lane were kept, the rest discarded, and road curves were then deliberately over-represented to remove a bias towards driving straight · NOT MACHINE-VOUCHED: not-in-our-copy - Both quotes on this row ('evidence', for which use evidence) are in NEITHER copy we hold - not in the text we tag against and not in the page we fetched ('fulltext.html') - checked with ligatures, hyphens, spacing and punctuation set aside. This body carries no measurable character damage at all. We do not know what it was taken from, and no class is claimed for it.

What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.