End-to-End Training of Deep Visuomotor Policies

End-to-End Training of Deep Visuomotor Policies Sergey Levine Chelsea Finn Trevor Darrell Pieter Abbeel

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

End-to-End Training of Deep Visuomotor Policies 2016 JMLR / arXiv 1504.00702 levine2016-visuomotor levine2016-visuomotor.txt, the project's staged copy 1a9fa90cd063f795cc375970231c8b81eea577c3 arxiv-open robotics

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them method Quotation withheld because this record is not cleared for redistribution. visuomotor policy learning Quotation withheld because this record is not cleared for redistribution. system-or-tool-building system-or-tool-building Quotation withheld because this record is not cleared for redistribution. mechanisms guided policy search: convert reinforcement-learning policy search into supervised learning of a deep network, with a trajectory-optimizer acting as the teacher Quotation withheld because this record is not cleared for redistribution. training requires positioning objects at consistent positions what this limits: the training procedure as described: it requires Quotation withheld Quotation withheld because this record is not cleared for redistribution. The method does not generalise to dramatically different visual settings, especially under occluding distractors. what this limits: the generalisation claim: OUR CURRENT METHOD DOES NOT GENERALIZE TO DRAMATICALLY DIFFERENT SETTINGS, ESPECIALLY WHEN VISUAL DISTRACTO[rs occlude] Quotation withheld because this record is not cleared for redistribution. The approach requires the full system state to be observable during training, which restricts the range of applicable tasks. what this limits: the range of tasks the method can address: THE REQUIREMENT TO OBSERVE THE FULL STATE DURING TRAINING LIMITS THE TASKS TO WHICH THE METHOD CAN BE APPLIED Quotation withheld because this record is not cleared for redistribution. Tasks with freely moving objects would need heavier instrumentation such as motion capture. what this limits: any extension to freely moving objects: those TASKS ... REQUIRE MORE EXTENSIVE INSTRUMENTATION, SUCH AS MOTION CAPTURE Quotation withheld because this record is not cleared for redistribution. The spatial feature-point representation makes simplifying assumptions about feature presence and uniqueness. what this limits: every result produced through the visual encoder: THE FEATURE POINT REPRESENTATION ... ASSUMES THAT THE LEARNED FEATURES ARE PRESENT AT ALL TIMES, AND ONLY ONE INSTANCE OF EACH FEATURE IS EVER PRESENT IN THE IMAGE Quotation withheld because this record is not cleared for redistribution. The method needs the full state observable during training, and the authors say exactly when that is minor and when it is not. what this limits: applying this method to a new task: the full state must be observable during training, which the authors call minor when objects can be positioned consistently and not minor for tasks manipulating freely moving objects Quotation withheld because this record is not cleared for redistribution. The architecture comparison is not run on the target task but on a pretraining task, which the authors describe as a reasonable proxy. what this limits: the architecture comparison in this paper: it is measured on the pose-estimation pretraining task, described by the authors as a reasonable proxy for the visuomotor challenges Quotation withheld because this record is not cleared for redistribution. The training-time observability the method depends on is bought with a physical constraint on the world, stated plainly: providing the full state 'simply requires positioning objects in one of several known positions for each trial during training'. The assumption is not about the algorithm but about how the laboratory is arranged. Quotation withheld because this record is not cleared for redistribution. deep convolutional networks trained end-to-end from perception to control for robotic manipulation Quotation withheld because this record is not cleared for redistribution. two baselines that pre-train the vision layers for pose prediction instead of training end-to-end, one with the SAME architecture as the full policy eliminates: that the improvement comes from the network architecture rather than from training the vision and control layers jointly end-to-end · outcome: held Quotation withheld because this record is not cleared for redistribution. visuomotor policy visuomotor policy Quotation withheld because this record is not cleared for redistribution. guided policy search guided policy search Quotation withheld because this record is not cleared for redistribution. pose estimation pose estimation Quotation withheld because this record is not cleared for redistribution. supervised learning supervised learning Quotation withheld because this record is not cleared for redistribution. architecture quality - MEASURED ON A PROXY: accuracy on the pose-estimation pretraining task, chosen to isolate the architectures from other confounding factors Quotation withheld because this record is not cleared for redistribution. failure at insertion - defined by a threshold taken from the object itself: since the peg is 0.5 units long, a larger distance means the controller cannot insert Quotation withheld because this record is not cleared for redistribution. observable - split by phase, and the split is the method's central assumption: the full system state is observable AT TRAINING TIME BUT NOT AT TEST TIME, which is what lets a trajectory-centric RL algorithm supply supervision the deployed policy never gets Quotation withheld because this record is not cleared for redistribution. builds-on → convolutional neural networks (deep learning) Quotation withheld because this record is not cleared for redistribution. builds-on → reinforcement learning Quotation withheld because this record is not cleared for redistribution. builds-on → Bregman ADMM (BADMM) (Wang and Banerjee, 2014) Quotation withheld because this record is not cleared for redistribution. extends → Levine and Abbeel, 2014; Levine et al., 2015 (the authors' own conference papers) Quotation withheld because this record is not cleared for redistribution. extends → Mordatch and Todorov (2014) Quotation withheld because this record is not cleared for redistribution. guided policy search —operationalised-as→ supervised learning guided policy search supervised learning strength of the relation: 0.9 · note: THE METHOD'S WHOLE MOVE IS A CONVERSION BETWEEN PROBLEM CLASSES, stated as such: '…search TRANSFORMS policy search INTO A SUPERVISED LEARNING…' The hard problem is not solved; it is turned into an easier one whose labels another algorithm supplies. Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says guided policy search TRANSFORMS policy search into a supervised learning problem; operationalised-as is loose. Quotation withheld because this record is not cleared for redistribution. 92,000 — size of the CNN policy network Quotation withheld because this record is not cleared for redistribution. 7 — depth of the policy CNN Quotation withheld because this record is not cleared for redistribution. tens of minutes — real-world data needed to train a policy Quotation withheld because this record is not cleared for redistribution. 2 — prior papers this article extends Quotation withheld because this record is not cleared for redistribution. end-to-end training improves consistency and generalization about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. the policy maps raw camera images directly to motor torques Quotation withheld because this record is not cleared for redistribution. full system state is available at training but not at test time Quotation withheld because this record is not cleared for redistribution. raw image observations raw image observations Quotation withheld because this record is not cleared for redistribution. motor torques motor torques Quotation withheld because this record is not cleared for redistribution. cost function cost function Quotation withheld because this record is not cleared for redistribution. full state full state Quotation withheld because this record is not cleared for redistribution. guided policy search guided policy search Quotation withheld because this record is not cleared for redistribution. visuomotor policies visuomotor policies about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. policy search policy search Quotation withheld because this record is not cleared for redistribution. Bregman ADMM (BADMM) Bregman ADMM (BADMM) Quotation withheld because this record is not cleared for redistribution. mean mean about this quote: DAMAGED GLYPHS IN THIS QUOTE: (cid:107). These are raw PDF font glyph indices that leaked out in place of the characters they stand for - usually a mathematical operator. The quote is verbatim to the staged text and its anchor is sound; the STAGED TEXT is what is damaged. NOT repaired: the cid-to-character mapping is defined by the font embedded in that particular PDF, so guessing it would produce a clean and unverifiable quote. Repair needs the source PDF and is a re-staging job. Quotation withheld because this record is not cleared for redistribution. regression regression Quotation withheld because this record is not cleared for redistribution. standard-deviation standard-deviation Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://arxiv.org/abs/1504.00702.