Why Robotics? Learning, Planning, and Intelligence That Acts

Abstract

An autonomous-driving simulation planted the first seed of my interest in robotics. XSkill later made the field concrete by showing how a robot could discover, represent, and reuse skills. Agricultural robotics and multi-arm planning then sharpened the question I want to study: what turns successful behavior into a reusable skill? A robot must know when a skill applies, what it changes, how to combine it with other skills, and how to detect and recover from failure. Working through that question moved me from the structure around a learned model to the model itself, and to the place where it is tested hardest: the crossing from simulation to hardware.

The Idea of a Robot Learning a Skill

My interest in robotics began as a faint curiosity. As I read across machine learning, computer vision, language, and robotics, I kept returning to systems that turned representations into physical capabilities. A robot does more than label, predict, or generate. It acts, changes the world, observes the result, and decides what to do next.

The paper that made this attraction concrete for me was XSkill: Cross Embodiment Skill Discovery [1]. XSkill learns skill prototypes from unlabeled human and robot manipulation videos, transfers those representations to robot actions, and composes them for tasks specified by human prompt videos. What captured me was the possibility that a robot could discover a skill representation, ground it in action, and reuse it in another task.

What made this new to me was not the performance. Until then I had assumed, without examining it, that a robot's capability came from structure someone had written down: the states a person chose to represent, the actions a person enumerated, the cases a person anticipated. In XSkill the capability was the learned thing itself. Diffusion Policy [2] made the same point somewhere else in the stack: the model does not select from behaviors a person listed, it generates them. Both came out of the same group, and seeing that group's work alongside the general-purpose robot stacks now coming out of industry turned a reaction to one paper into an interest in a direction.

The paper also sharpened the question. A latent prototype may not correspond to a human-interpretable skill, and an unseen task may recombine familiar parts. XSkill generalized to some unseen transitions in simulation, while its real-world results remained sensitive to the diversity of robot teleoperation data [1]. The work showed that representation, transfer, and composition could be tested—and left “having a skill” open as a research problem.

That was when robot learning became concrete to me: the learned representation was grounded in a body, connected to action, and tested by what happened next.

What Engineering Projects Added

The autonomous-driving work took place in a simulation competition. Our system used hand-built search, rules, and PID control rather than a learned policy. The objective was not merely to reach a destination. The vehicle also had to remain safe, obey traffic rules, move at a reasonable speed, and avoid abrupt acceleration or braking. Even the sampling rate created a systems trade-off: processing fewer observations could reduce computation, but it also changed how frequently the controller could react.

In the driving simulator, even an ordinary intersection could include moving pedestrians and cross traffic. The challenge was not to plan once, but to keep deciding when to wait and when to proceed as the scene changed. It made me see planning and control as a loop, not a one-time calculation.

A later agricultural-robotics project added a different lesson. Capturing plants with a robot arm, reconstructing their geometry, and preparing measurements for downstream use meant that the state was never simply handed to the system in a clean symbolic form. Camera viewpoints, reconstruction error, plant structure, and the meaning of the final measurement were coupled. Perception did not sit before planning as a finished input; it shaped what the system could know and therefore what it could do.

Together, these projects showed me that a module matters through the loop in which it operates. A planner depends on observations. A controller depends on the rate and quality of updates. A reconstruction matters only in relation to the decision it supports. In robotics, an apparently local design choice travels through the rest of the system.

When an Explicit Plan Changed My Question

At Rutgers, I worked as a research intern on a multi-arm tabletop object-rearrangement project. The professor posed the high-level problem. I independently developed the formulation and model, wrote the pseudocode and implementation, explored algorithmic approaches, and obtained systematic but preliminary simulation results.

The high-level problem was easy to state. Several objects had initial and goal positions, and some goals could be occupied by other objects. Multiple arms had to move the objects into the target arrangement. Once I tried to formulate the problem precisely, however, “move every object to its goal” separated into several interacting questions. Which dependencies determine the ordering? When is temporary space needed? How should a plan trade execution time against action count? When does keeping one arm occupied simplify one move but delay the rest?

No single objective captured every relevant sense of “better”; reducing the trade-offs to one scalar would itself require a judgment about their relative importance. A plan that used less temporary space could require a longer schedule. Optimizing action count did not automatically optimize execution time. Decisions that looked good for one arm could constrain what another arm could do. Explicit modeling did not remove these trade-offs, but it forced them into view. I could inspect the assumptions, trace why a decision was made, and ask which objective or constraint was actually responsible.

The project produced systematic but preliminary simulation results. It changed the question I carried away. I had treated “uses learning” as a sign that a method was more intelligent or advanced. The project showed me that explicit structure can create capability when a problem has identifiable objects, dependencies, resources, and goals.

For a well-specified problem, explicit models and classical algorithms can provide stable, efficient, inspectable performance and generalize across instances without retraining. Learning becomes valuable when perception, dynamics, or variation resist hand-built models. The project led me to a more useful question: what must a learned skill expose so an explicit planner can use it?

From Task Success to a Reusable Skill

I once dismissed machine learning as “only statistics.” That was wrong, and it is worth being specific about how. I had treated “statistical” as a way of saying “not really reasoning,” as though fitting a distribution were a lesser activity than writing a rule. But statistical estimation is how modern data-driven methods learn, and a rule I write by hand is also an estimate—it is just one fitted from my own limited sample of the world, with no error bar and no way to notice when it stops holding. The question worth asking is not whether a method is statistical. It is what a particular success demonstrates.

Success on familiar tasks can demonstrate an effective policy or representation. A reusable skill must do more: apply to new objects, compose with other capabilities, adapt from limited experience, and identify unsafe or unreliable execution before it begins. More data may improve these properties, but data volume is not the test.

High-quality demonstrations provide valuable supervision. I want to know how robots can adapt from a small, carefully selected set and remain useful when later data are noisy or heterogeneous. That requires sample-efficient learning and explicit tests for corrupted, biased, or unsafe examples.

Rearrangement makes that last case concrete. An arm needs somewhere to put an object while it clears the way, so it moves it to free space. Every step of that is locally correct: the grasp succeeds, the placement is collision-free, the object ends up somewhere valid. The plan fails several actions later, because the free space it chose was another object's goal, and nothing in the executed motion carries the information that would have said so. A system that only checks whether each action succeeded cannot see this. A system that knows what each skill is expected to change can.

The options framework represents a temporally extended action through an initiation set, a policy, and a termination condition [3]. Classical planning operators specify when an action applies and what effects it produces, while task-and-motion planning connects that structure to geometric feasibility [4]. Together, these formalisms show why a long-horizon planner needs more than a callable policy: it needs an interface that exposes applicability, effects, and termination.

Success on a familiar task and a skill that can be reused are not the same claim.

The Interface Between Learning and Planning

The useful boundary lies in the interface between learning and planning, not in the labels of research communities. Learning handles structure that resists hand specification: representations from high-dimensional observations, object properties, affordances, complex dynamics, behavior from demonstrations, and adaptation to variation. Explicit models and planners expose goals, hard constraints, dependencies, resource conflicts, task structure, and feasibility for inspection.

Long-horizon manipulation systems have combined engineered and learned perception with task-and-motion planning so that RGB-D estimates support multi-step action [5]. This work shows why “learning or planning” is the wrong system-level question. The interface between them is the more interesting problem.

A planner needs more from a learned skill than a name: applicability conditions, predicted effects, cost, duration, and calibrated uncertainty. A learned skill needs more from a planner than a command: a goal, context, constraints, and a definition of completion. Execution must report what happened so the system can continue, adjust, substitute another capability, or replan.

None of this has to be exact: preconditions can be learned, effects probabilistic, explicit models imperfect.

Where This Leaves Me

Three things have held my attention through all of this: what a robot can learn from contact rather than from cases someone wrote down in advance; where the line between a learned model and the structure around it actually falls; and what a robot has to carry with it to finish something long.

Why Robotics

Why robotics? Because robotics turns ideas about intelligence into decisions and actions. The physical world does not accept an attractive representation, an elegant plan, or a high validation score as a substitute for a working loop. It returns consequences.

There is a more specific version of this. Most of the work I want to do trains in simulation and then has to survive contact with hardware. It is tempting to treat that gap as an engineering tax, something to be minimized so that the real result can be reported. I have come to think close to the opposite. The gap is where the assumptions inside a learned model become visible: what it silently depended on, what it never had to represent, what it cannot recover from. That makes it an object worth studying rather than an obstacle to clear.

I want the test to be a robot doing something with consequences: first in simulation, then on hardware.

References

1. M. Xu, Z. Xu, C. Chi, M. Veloso, and S. Song, “XSkill: Cross Embodiment Skill Discovery,” in Proc. 7th Conf. Robot Learn. (CoRL), ser. Proc. Mach. Learn. Res., vol. 229, pp. 3536–3555, Nov. 2023.
2. C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, “Diffusion Policy: Visuomotor Policy Learning via Action Diffusion,” in Proc. Robot.: Sci. Syst. (RSS), Jul. 2023.
3. R. S. Sutton, D. Precup, and S. Singh, “Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning,” Artif. Intell., vol. 112, nos. 1–2, pp. 181–211, Aug. 1999.
4. C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Pérez, “Integrated Task and Motion Planning,” Annu. Rev. Control Robot. Auton. Syst., vol. 4, pp. 265–293, May 2021.
5. A. Curtis, X. Fang, L. P. Kaelbling, T. Lozano-Pérez, and C. R. Garrett, “Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordances,” in Proc. 2022 IEEE Int. Conf. Robot. Autom. (ICRA), pp. 1940–1946, May 2022.