End to End Learning for Self-Driving Cars — TEI-tagged view

Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner · 2016 · robotics · arXiv:1604.07316 · full text at the source · TEI xml · tagged edition · index · concepts · authors
Where this record's text came from. faceted 2026-08 by reading the locally held text (pdftext); every evidence string verified as an exact substring of that text before merging. An NVIDIA technical report rather than a peer-reviewed paper, and it says so in effect ('preliminary results'), which is recorded as a limitation. The reported autonomy figures (98% on one route, 10 zero-intercept highway miles) come from a small, informally specified test set with no statistics, error bars or baseline comparison — a weakness the facets cannot express because the paper never states it. method is system-or-tool-building: the work builds a driving system and evaluates it in a purpose-built simulator and then on public roads. Two relations point at the same target (ALVINN) because the paper both credits it as inspiration and explicitly differentiates from it in the next sentence. 2026-08-13, 2026-08-14 and 2026-08-16 schema-0.3 passes: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node. Update, 15 September 2026: EXCEPTION, measured with the merge gate's own predicate (Paper.locate): 2 quote(s) on this record are not found as one exact string in the current staged text - limitations[9].evidence, limitations[9].for_which_use_evidence. The sentence above does not hold for them as written. Where a row records why - the quote is grounded in another copy of the paper, joins two passages with an ellipsis, or differs from the staged text by extraction noise - that note stays on the row. Update, 16 September 2026: EXCEPTION TO THE DATED PASS ABOVE, on this record only: this record carries 1 composite quote field - a quote assembled from two or more passages joined by an ellipsis. A COMPOSITE IS NOT AN EXACT CONTIGUOUS SUBSTRING of the staged text and is deliberately not anchored: each of its parts is verified separately at the offsets in evidence composite spans, and a part occurring more than once carries span null rather than a guessed offset. So the 'every quote verified as an exact substring' clause of the dated pass above DOES NOT HOLD for those fields, and never did - the parts were verified, the whole was not. Nothing is deleted, re-grounded or withdrawn: this writes down an exception that was always true of composites and had not been recorded, so a reader was told more than the pass had checked.
Other fields define these differently the same word, defined another way in another discipline
deep learning 11 fields · autonomy 8 fields · training 3 fields · standard-deviation 3 fields · supervised learning 3 fields · convolutional neural network 2 fields · and 1 more →
Tags on this paper every tag the record carries, whether or not the words occur in the text below — a number is how many papers in the corpus carry that tag; a tag with no number is unique to this paper
convolutional neural network11 deep learning42 end-to-end learning2 autonomous vehicle supervised learning15 autonomy17 intervention5 road features training signal system-or-tool-building173

Our tag layer for this paper

Cross-disciplinary analysis
content-tags & facets: phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text
phenomenonend-to-end learning of lane and road following— “We have empirically demonstrated that CNNs are able to learn the entire task of lane and road following without manual decomposition into road or lane marking detection, semantic abstraction, path planning, and control.”
claim-typemethod— “We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands.”
method — evidence— “Evaluating our networks is done in two steps, first in simulation, and then in on-road tests.”
transferfrom computer vision (convolutional neural networks): a convolutional network taking raw camera pixels, used in place of the perception-plus-control pipeline of robotics so that steering is learned end to end— “We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands.”transfers[0]
mechanismthe steering command is stored as 1/r rather than r, which removes the singularity at straight-ahead and makes the label continuous through zero across left and right turns— “In order to make our system independent of the car geometry, we represent the steering command as 1/r where r is the turning radius in meters. We use 1/r instead of r to prevent a singularity when driving straight (the turning radius for driving straight is infinity). 1/r smoothly transitions through zero from left turns (negative values) to…mechanisms[0]
mechanismrecovery is taught rather than encountered: the training set is augmented with images of the car already displaced from the lane centre, because human driving alone never shows the network a mistake— “Training with data from only the human driver is not sufficient. The network must learn how to recover from mistakes. Otherwise the car will slowly drift off the road. The training data is therefore augmented with additional images that show the car in different shifts from the center of the lane and rotations from the direction of the…mechanisms[1]
mechanismoff-centre viewpoints are synthesised under an approximation the paper states it knows to be wrong - everything below the horizon treated as flat ground, everything above it as infinitely far - accepted because the resulting distortions do not hurt training— “Precise viewpoint transformation requires 3D scene knowledge which we don’t have. We therefore approximate the transformation by assuming all points below the horizon are on flat ground and all points above the horizon are infinitely far away. This works fine for flat terrain but it introduces distortions for objects that stick above the…mechanisms[2]
mechanismthe label for a synthesised viewpoint is not the driver's command but the command that would return the car to the correct place within a fixed two-second horizon— “The steering label for transformed images is adjusted to one that would steer the vehicle back to the desired location and orientation in two seconds.”mechanisms[3]
mechanismthe training frames are filtered by the driver's activity label and then downsampled, because consecutive frames at full rate are near-duplicates— “To train a CNN to do lane following we only select data where the driver was staying in…mechanisms[4]
mechanismthe perturbations are drawn from a normal distribution deliberately twice as wide as the variation measured in human drivers
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
mechanisms[5]
mechanismautonomy is computed from a COUNTED event and an ASSUMED constant: each simulated intervention is charged a fixed six seconds of human takeover time
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
mechanisms[6]
mechanismthe normalisation layer is inside the network but hard-coded, so it is not learned and can be changed with the architecture and run on the GPU
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
mechanisms[7]
builds-on Pomerleau ALVINN (1989)
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
claim_relations[0]
builds-on DARPA Autonomous Vehicle (DAVE) [5]
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
claim_relations[1]
builds-on Pomerleau [6], the ALVINN system
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
claim_relations[2]
extends Pomerleau [6], the ALVINN system
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
claim_relations[3]
senseautonomy - AS A STIPULATED FORMULA: one minus the intervention count times six seconds over elapsed time, as a percentage; the word names an arithmetic result, not an observed capability
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
senses[0]
sensea performance criterion - CHOSEN FOR LEGIBILITY, WHICH IS NOT THE SAME AS BEING RIGHT: the paper's argument for end-to-end training is that human-selected intermediate criteria are picked because a person can interpret them, and that this does not guarantee system performance
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
senses[1]
senseroad features - LEARNED WITHOUT EVER BEING NAMED: the network detects them from the steering angle alone, and the authors state they never trained it to detect road outlines
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
senses[2]
controlmanually calibrating the lane centre for each frame used by the simulator, rather than taking the human driver's position as the target — held — rules out the human drivers' own off-centre driving being learned as the correct steering target
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
controls[0]
edgeautonomy —operationalised-as→ intervention
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
concept_edges[0]
audittraining signal —causes→ road features — Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the network learns to detect road features with steering angle as its only training signal; causes is loose.
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
concept_edges[1]
outcome (stated by the tagger, no quotation)control — heldcontrols[0]
for which uselimitation — the deployment readiness of the system: MORE WORK IS NEEDED TO IMPROVE THE ROBUSTNESS OF THE NETWORK
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[0]
for which uselimitation — every augmented training example: PRECISE VIEWPOINT TRANSFORMATION REQUIRES 3D SCENE KNOWLEDGE WHICH WE DON'T HAVE
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[1]
for which uselimitation — the augmentation for anything but flat road: IT INTRODUCES DISTORTIONS FOR OBJECTS THAT STICK ABOVE THE GROUND, SUCH AS CARS, POLES, TREES, AND BUILDINGS - the distorted objects are the ones that matter
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[2]
for which uselimitation — the benefit of larger augmentation: ARTIFICIALLY AUGMENTING THE DATA DOES ADD UNDESIRABLE ARTIFACTS AS THE MAGNITUDE INCREASES
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[3]
for which uselimitation — any safety or verification claim: MORE WORK IS NEEDED ... TO FIND METHODS TO VERIFY THE ROBUSTNESS, AND TO IMPROVE VISUALIZATION OF THE NETWORK-INTERNAL PROCESSING STEPS
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[4]
for which uselimitation — the standing of every result reported: THIS PAPER DESCRIBES PRELIMINARY RESULTS OF THIS NEW EFFORT
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[5]
for which uselimitation — any autonomy percentage from this paper: it multiplies the intervention count by an assumed six seconds - the time the authors take a human to retake control, re-centre and restart self-steering - so the figure moves with that assumption
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[6]
for which uselimitation — the intervention count behind the autonomy figure: an intervention is triggered automatically when the simulated vehicle departs the centre line by more than one metre, so the number depends on that threshold
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[7]
for which uselimitation — simulated autonomy as evidence of safety: after each intervention the vehicle is reset to the ground truth of the original video, so errors never compound and the metric counts how often the car left the lane rather than how badly
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[8]
evidence statuslimitation — quote-not-in-current-staged-text
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[9]
evidence status notelimitation — marked quote-not-in-current-staged-text - the quote is a DECLARED COMPOSITE (the composite-quote note), so the whole string is not one span of the staged text; each of its two parts is verified there, exactly once.
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[9]
for which uselimitation — the trained network as a model of driving: only frames where the driver was staying in a lane were kept, the rest discarded, and road curves were then deliberately over-represented to remove a bias towards driving straight
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[9]
not machine-vouchedlimitation — not-in-our-copy - Both quotes on this row ('evidence', for which use evidence) are in NEITHER copy we hold - not in the text we tag against and not in the page we fetched ('fulltext.html') - checked with ligatures, hyphens, spacing and punctuation set aside. This body carries no measurable character damage at all. We do not know what it was taken from, and no class is claimed for it.
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[9]
replicationuntested
limitationmore work is needed to improve robustness and to verify/visualize the network's internal processing
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[0]
limitationThe viewpoint transformation used for augmentation is an approximation because the true 3D scene geometry is unknown.
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[1]
limitationThat approximation distorts objects standing above the ground plane.
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[2]
limitationLarger augmentation magnitudes introduce undesirable artifacts into the training data.
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[3]
limitationRobustness of the network, methods to verify it, and internal-state visualisation all remain open.
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[4]
limitationThe results are explicitly preliminary.
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[5]
limitationThe reported autonomy percentage is built on an assumed six-second cost per intervention, which the authors state as an assumption rather than measure.
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[6]
limitationAn intervention is a threshold crossing in simulation, not a human decision: the car counts as having failed when it passes one metre from the centre line.
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[7]
limitationAfter every simulated intervention the car is placed back on the recorded ground truth, so the measure cannot observe how a failure would have developed.
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[8]
limitationThe training set is deliberately not the driving distribution: one driver activity is kept, the rest discarded, and curves are then over-represented on purpose.
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
limitations[9]
Content tags
content-observations: finding · quantity · variable · definition · key-term — grounded in the full text
findingA CNN learned to turn raw pixels from one front-facing camera straight into steering commands; the end-to-end approach proved surprisingly powerful.
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
content_tags.findings[0]
findingThe network learned useful road features using only the human steering angle as the training signal, never explicitly supervised to detect them.
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
content_tags.findings[1]
quantity98% — fraction of time the network drove autonomously (Holmdel to Atlantic Highlands)
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
content_tags.quantities[0]
quantity30 FPS — system operating frame rate
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
content_tags.quantities[1]
quantity9 layers — network depth: 1 normalization + 5 convolutional + 3 fully connected
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
content_tags.quantities[2]
quantity10 miles — distance driven on the Garden State Parkway with zero interventions
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
content_tags.quantities[3]
variableraw pixels (independent)
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
content_tags.variables[0]
variablesteering angle (dependent)
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
content_tags.variables[1]
variableautonomy (dependent)
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
content_tags.variables[2]
definitionautonomy
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
content_tags.definitions[0]
definitionend-to-end learning
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
content_tags.definitions[1]
key-termsconvolutional, steering, training, network, camera, road, image, autonomy, features
Concepts
terms this paper carries, each with the sentence it was cut from — the layer that joins one paper to another. Terms with no quote yet are on CONCEPTS.html.
conceptconvolutional neural network
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
concepts[0]
conceptautonomy
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
concepts[5]
conceptintervention
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
concepts[6]
conceptroad features
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
concepts[7]
concepttraining signal
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
concepts[8]
Statistics — cross-field join keys
field-invariant statistical terms, used in their statistical sense (gated on a number / symbol / test). A p-value or a standard deviation means the SAME thing in physics, biology, psychology and sociology — so these are the strongest cross-field join keys. Grounded verbatim.
standard-deviation SD/σ
Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher.
content_tags.statistics[0]

When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.

Tag layer © the Meta-Tagging Project, CC BY 4.0. The paper text is © its authors/publisher and is linked, not reproduced. Excerpts shown are short quotations for identification and verification. · back to index · concepts · authors · עברית
© 2026 Shir Sivroni · ORCID 0009-0000-2597-2824