Robot Learning / Physical AI
Robot Learning is where artificial intelligence stops predicting the world and starts physically interacting with it.
A language model can make a wrong prediction and generate a strange sentence.
A robot can make a wrong prediction and:
drop a glass,
crash into a wall,
damage a motor,
fall down a staircase,
collide with a person,
destroy an expensive component.
That single difference changes almost everything.
Robot learning combines ideas from:
machine learning,
deep learning,
reinforcement learning,
computer vision,
control theory,
dynamics,
optimization,
robotics,
simulation,
probability,
and increasingly, large multimodal models.
This family of ideas is often called Physical AI or Embodied AI.
The central question is simple:
How do we build machines that perceive the physical world, understand what is happening, decide what should happen next, and execute actions successfully under real-world uncertainty?
This article develops that idea from the ground up.
1. The Core Mental Model
Almost every intelligent robot can be understood as a loop.
Environment
↓
Sensors
↓
Perception
↓
Internal Representation
↓
Decision / Policy
↓
Action
↓
Robot Motors
↓
Environment changes
↓
Sensors again
For example, imagine a robot picking up a coffee mug.
Its camera observes:
table
mug
chair
human hand
The robot estimates:
mug center = (x, y, z)
mug orientation = ...
robot arm position = ...
obstacles = ...
It then decides:
move arm toward mug
open gripper
align gripper
close gripper
lift
The robot executes the movement.
But perhaps the mug moves slightly.
The robot sees the new state and adjusts.
So intelligent robotics is fundamentally a closed-loop problem.
You repeatedly perform:
observe → estimate → decide → act → observe again
The learning question is:
Which parts of this loop should be learned from data instead of programmed manually?
That question leads us into robot learning.
2. Classical Robotics vs Robot Learning
Traditional robotics relies heavily on manually constructed models.
An engineer might explicitly program:
Detect object
↓
Estimate object pose
↓
Calculate inverse kinematics
↓
Generate trajectory
↓
Run trajectory controller
↓
Close gripper
Every component has carefully designed mathematics.
This approach is extremely powerful.
Industrial robotics was built largely this way.
But it can struggle when environments become messy.
Imagine a household robot.
Objects have different:
shapes,
textures,
weights,
lighting conditions,
positions,
materials.
People move unpredictably.
Furniture changes.
Objects may be partially hidden.
The same manually engineered rules cannot easily cover every possible situation.
Robot learning introduces another possibility.
Instead of programming every behavior explicitly, we allow robots to learn patterns from data and experience.
Conceptually:
Traditional Robotics
World
↓
Hand-designed perception
↓
Hand-designed rules
↓
Hand-designed controller
↓
Action
versus:
Robot Learning
World
↓
Learned perception
↓
Learned representation
↓
Learned policy
↓
Action
Real systems often combine both.
That hybrid approach will become very important later.
3. What Exactly Does a Robot Learn?
A robot can learn many different things.
It may learn:
Perception
image → object locations
State estimation
camera + sensors → robot/world state
Representation
raw observations → useful latent features
Dynamics
current state + action → predicted next state
Policy
state → action
Reward model
state/action → how desirable is this?
Value function
state → expected future reward
Grasp strategy
object geometry → gripper pose
Locomotion
body state + terrain → motor commands
Manipulation
visual scene + task → arm actions
Modern robotics research increasingly tries to learn several of these components together.
4. Imitation Learning
Suppose you want a robot arm to place dishes into a dishwasher.
One possibility is to manually program every movement.
Another is:
Show the robot how a human does it.
This is imitation learning.
The robot learns behavior from demonstrations generated by an expert.
The expert might be:
a human,
another robot controller,
a planner,
a teleoperation system,
or a highly optimized simulation policy.
The dataset might look like:
observation₁ → expert action₁
observation₂ → expert action₂
observation₃ → expert action₃
...
For robotics:
camera image
robot joint positions
gripper state
↓
expert action
The robot learns:
π(a | s)
where:
π= policys= statea= action
Meaning:
Given this state, what action would the expert take?
5. Learning From Demonstrations
Learning from Demonstrations, often abbreviated LfD, is the broader idea of robots learning behaviors from example demonstrations.
Imagine teaching a robot to pour water.
A human teleoperates the robot.
The robot records:
camera frames
joint positions
joint velocities
gripper state
force sensor readings
actions
One demonstration might contain thousands of time steps.
For example:
t = 0
observe cup
action: move right
t = 1
observe cup
action: move forward
t = 2
gripper near cup
action: close gripper
t = 3
cup held
action: lift
...
After collecting many demonstrations, a model learns the mapping.
LfD is especially useful when:
the correct behavior is difficult to specify mathematically,
humans can easily demonstrate the task,
reward engineering would be difficult.
6. Behavioral Cloning
The simplest form of imitation learning is Behavioral Cloning.
It turns robotics into supervised learning.
Suppose your dataset is:
(state, expert_action)
You train:
policy(state) → predicted_action
If the expert action is continuous:
a = [joint_velocity_1,
joint_velocity_2,
joint_velocity_3,
...]
you might minimize mean squared error:
L = ||a_predicted - a_expert||²
For discrete actions:
forward
backward
left
right
stop
you might use cross-entropy.
Conceptually:
for observation, expert_action in dataset:
predicted_action = policy(observation)
loss = action_loss(
predicted_action,
expert_action
)
loss.backward()
optimizer.step()
Simple.
Powerful.
But there is a serious problem.
7. Distribution Shift in Behavioral Cloning
Imagine the expert always drives perfectly in the center of a road.
The training dataset contains mostly:
car centered
car centered
car centered
car centered
The learned policy makes a small mistake.
Now the vehicle is slightly left of center.
But the training dataset rarely contained situations like:
car far left
The robot doesn't know how to recover.
It makes another mistake.
Now it is even farther left.
The errors compound.
This problem is called distribution shift or covariate shift.
Training data contains:
states visited by expert
but deployment contains:
states visited by learned policy
Those distributions may differ.
This is one reason robotics is harder than ordinary supervised learning.
Your model influences the future data it receives.
8. Reinforcement Learning
Imitation learning says:
Copy the expert.
Reinforcement learning says:
Try actions and learn which behaviors produce good outcomes.
A reinforcement-learning problem usually contains:
Agent
Environment
State
Action
Reward
At time t:
state sₜ
↓
policy chooses action aₜ
↓
environment changes
↓
reward rₜ
↓
new state sₜ₊₁
The goal is to maximize cumulative reward:
G = r₀ + γr₁ + γ²r₂ + γ³r₃ + ...
where γ is the discount factor.
9. Example: Teaching a Robot to Walk
Suppose a quadruped must learn locomotion.
Possible reward:
reward =
+ forward_velocity
- falling_penalty
- excessive_energy
- unstable_body_motion
The robot experiments with motor commands.
Initially:
step
fall
step
fall
twist
fall
Eventually useful behaviors emerge:
balance
move legs rhythmically
stabilize torso
walk
run
recover
Rather than explicitly programming the gait, optimization discovers it.
This has made reinforcement learning extremely important for:
locomotion,
dexterous manipulation,
drone control,
robotic hands,
dynamic motion,
complex coordination.
10. The Markov Decision Process
Reinforcement learning is usually formalized using a Markov Decision Process, or MDP.
An MDP contains:
S = states
A = actions
P = transition dynamics
R = reward function
γ = discount factor
The dynamics describe:
P(s' | s, a)
Meaning:
If I perform action
ain states, how likely am I to reach states'?
The policy is:
π(a | s)
A value function estimates:
V(s)
which means:
How good is this state considering future rewards?
Or:
Q(s, a)
meaning:
How good is taking action
afrom states?
These concepts are foundational even when modern neural networks hide much of the mathematics.
11. Why Reinforcement Learning Is Difficult in Robotics
RL works beautifully in games because simulation is cheap.
A game agent may collect:
billions of interactions
A physical robot cannot.
Imagine training a robot by crashing it ten million times.
You would quickly run out of robots.
Real-world RL has problems:
data is expensive
hardware wears out
experiments are slow
resetting environments is difficult
unsafe exploration is unacceptable
Therefore robot learning often uses:
simulation
+
imitation learning
+
offline datasets
+
carefully constrained RL
rather than blindly exploring in the physical world.
12. Visuomotor Policies
A visuomotor policy maps visual information directly to motor behavior.
For example:
camera image
↓
neural network
↓
robot action
More realistically:
RGB image
depth image
robot joint state
gripper state
↓
vision encoder
↓
representation
↓
policy network
↓
joint commands
Example:
Input:
image of table
robot joint positions
Output:
[0.03, -0.08, 0.12, ..., gripper_close]
The network learns the relationship between visual geometry and movement.
This avoids manually separating the system into:
object detector
pose estimator
planner
controller
although hybrid systems often still use these components.
13. End-to-End Robot Learning
The extreme version is an end-to-end policy:
pixels → actions
For example:
camera
↓
Transformer
↓
motor commands
This is conceptually attractive because the entire system can optimize for the final task.
But it introduces challenges:
enormous data requirements,
difficult debugging,
limited interpretability,
safety concerns,
unpredictable generalization.
If the robot fails, it becomes harder to identify whether the problem came from:
perception?
representation?
reasoning?
policy?
control?
Therefore production systems frequently combine learned and classical components.
14. Vision-Language Models
A Vision-Language Model, or VLM, understands both images and language.
Conceptually:
image + text → semantic understanding
For example:
Image:
robot sees a kitchen
Prompt:
"Where is the red mug?"
Model:
"The red mug is on the table next to the kettle."
Or:
"What object should I use to clean the table?"
→ "The cloth near the sink."
VLMs give robots high-level semantic understanding that traditional perception systems often lack.
Traditional computer vision might produce:
object_17
bounding_box = ...
A VLM can reason:
That is a mug.
It probably contains a drink.
The user asked me to bring their coffee.
Therefore this object is relevant.
This semantic reasoning is extremely valuable.
15. Vision-Language-Action Models
A Vision-Language-Action model, or VLA, goes one step further.
Instead of:
vision + language → text
we have:
vision + language → robot actions
Example:
Camera:
kitchen scene
Instruction:
"Put the apple into the bowl."
Robot state:
arm position
gripper state
The model outputs something representing:
move toward apple
open gripper
align
close gripper
lift
move toward bowl
release
A simplified architecture might be:
Images ──────────────┐
↓
Language ──────→ Transformer → Action Decoder → Robot
↑
Robot State ─────────┘
This idea is important because robotics begins to resemble modern foundation-model training.
Instead of building one policy for:
pick apple
another for:
pick mug
another for:
open drawer
we train one large policy across enormous collections of tasks.
16. Why VLA Models Matter
Traditional robot policies are often narrow.
A policy trained to:
pick red cube
may fail at:
pick blue mug
VLA systems aim for much broader generalization.
The goal is:
many robots
+
many environments
+
many objects
+
many instructions
+
many demonstrations
↓
general robot foundation model
Similar to how language models learned general linguistic patterns from huge datasets, robot foundation models attempt to learn broad physical patterns.
The long-term dream is something like:
"Please clean the kitchen."
The robot interprets the goal, understands objects and relationships, plans subtasks and executes them.
That is dramatically harder than generating text.
17. Embodied AI
Embodied AI refers to intelligence that exists inside an agent that perceives and interacts with an environment.
The important word is embodied.
A language model typically receives tokens.
An embodied agent receives:
images
depth
sound
joint positions
forces
touch
motion
temperature
spatial relationships
And its actions change what it will observe next.
For example:
Robot cannot see behind box.
It can physically move.
move left
Now it can see behind the box.
Its action actively changes its information.
This is sometimes called active perception.
Intelligence is no longer:
input → output
It becomes:
observe
think
act
observe consequences
adapt
18. The Physical AI Loop
A useful architecture for Physical AI is:
┌───────────────┐
│ Perception │
└───────┬───────┘
↓
┌───────────────┐
│ Representation│
└───────┬───────┘
↓
┌───────────────┐
│ World Model │
└───────┬───────┘
↓
┌───────────────┐
│ Planner/Policy│
└───────┬───────┘
↓
┌───────────────┐
│ Controller │
└───────┬───────┘
↓
Robot
↓
Environment
↓
Sensors
└────────→ back to perception
Different robot-learning systems learn different boxes.
Future systems may learn nearly the entire stack.
19. World Models
One of the most powerful ideas in modern AI is the world model.
A world model learns how the environment behaves.
In simple terms:
If I do this, what will happen next?
Suppose:
current state = sₜ
action = aₜ
The world model predicts:
sₜ₊₁ = f(sₜ, aₜ)
For example:
robot pushes box right
Prediction:
box moves right 20 cm
robot arm ends here
Or:
robot releases object too early
Prediction:
object falls
A sufficiently capable world model allows the robot to mentally test actions before executing them.
20. Learned Dynamics
Traditional robotics derives dynamics from physics.
For example:
M(q)q̈ + C(q,q̇)q̇ + g(q) = τ
where roughly:
q= joint configuration,q̇= joint velocity,q̈= joint acceleration,M= mass/inertia effects,C= velocity-dependent effects,g= gravity,τ= applied torque.
These equations are powerful.
But reality contains effects that are difficult to model perfectly:
friction
flexibility
motor delay
surface variation
cables
contact
unknown payloads
wear
Instead, we can learn dynamics:
neural_network(
current_state,
action
)
→ next_state
This is called learned dynamics.
21. Model-Based Reinforcement Learning
If the robot learns a model of the environment, it can use that model to plan.
This is model-based reinforcement learning.
Compare:
Model-free RL
state → policy → action
The policy learns directly from experience.
Model-based RL
state
↓
world model
↓
simulate possible futures
↓
choose good action
For example:
Option A:
push cup forward
→ simulated cup falls
Option B:
grasp cup first
→ simulated cup remains stable
choose B
The major advantage is sample efficiency.
Physical experience is expensive.
Imagined experience inside a model is cheap.
22. Representation Learning for Robotics
Raw sensor data is enormous.
An image might contain:
1920 × 1080 × 3
pixel values.
The robot does not need to reason directly over every pixel.
Instead, neural networks learn useful representations.
Example:
image
↓
vision encoder
↓
latent representation
The representation might implicitly encode:
objects
depth
shape
pose
material
affordances
spatial relationships
This is called representation learning.
A good representation makes downstream control easier.
23. What Is an Affordance?
An important robotics concept is affordance.
An affordance describes what actions an object supports.
Example:
A chair affords:
sitting
moving
grasping
A handle affords:
pulling
A button affords:
pressing
A cup handle affords:
grasping
A robot doesn't merely need to recognize:
"This is a drawer."
It needs to understand:
"The handle is the useful location for opening this drawer."
Robot representations increasingly try to capture these action-relevant concepts.
24. Grasp Learning
Grasping looks easy because humans have spent their entire lives mastering it.
For robots, it is difficult.
Given an object, the robot must determine:
where to grasp
how to orient the gripper
how much force to apply
whether the grasp will remain stable
A grasp model may estimate:
P(success | image, grasp_pose)
The robot proposes many candidate grasps:
G₁
G₂
G₃
...
Gₙ
The model evaluates them:
G₁ → 0.13
G₂ → 0.92
G₃ → 0.48
Choose:
G₂
Modern grasping systems may use:
RGB images,
depth maps,
point clouds,
tactile sensing,
learned 3D representations.
25. Locomotion Learning
Locomotion means generating movement for robots such as:
quadrupeds,
humanoids,
bipeds,
drones,
legged platforms.
Walking requires coordinating many joints while maintaining balance.
Inputs might include:
joint angles
joint velocities
IMU readings
foot contacts
terrain observations
desired velocity
Output:
joint targets
or
joint torques
A learned policy might be:
π(
robot_state,
desired_velocity,
terrain
)
→ motor commands
Reinforcement learning has become particularly successful here because physics simulators can generate huge amounts of synthetic locomotion experience.
26. Manipulation Learning
Manipulation means changing the state of objects.
Examples:
pick
place
push
pull
rotate
pour
fold
insert
open
close
assemble
Manipulation is difficult because of contact.
Before contact:
robot moves through free space
After contact:
robot + object interact physically
Tiny errors can produce drastically different outcomes.
Consider inserting a plug into a socket.
A 2 mm alignment error may mean:
success
versus:
failure
This makes manipulation one of the most challenging areas of robot learning.
27. Tactile Learning
Humans do not manipulate objects using vision alone.
Imagine tying your shoes while looking elsewhere.
Your fingers provide information about:
pressure
slippage
contact
texture
shape
force
Robots can use tactile sensors in similar ways.
A tactile policy may detect:
object slipping
and respond:
increase grip force
Or detect:
connector slightly misaligned
and perform:
small corrective motion
Tactile learning becomes especially important when:
vision is occluded
which happens constantly during manipulation.
The robot's own hand often blocks its camera.
28. Multimodal Sensor Fusion
Real robots use many sensors.
Typical modalities include:
RGB camera
depth camera
LiDAR
radar
IMU
joint encoders
force/torque sensors
tactile sensors
microphones
GPS
Each sensor has strengths and weaknesses.
For example:
Camera
Excellent semantic information.
But sensitive to:
lighting
occlusion
motion blur
LiDAR
Excellent geometry.
But weaker semantic information.
IMU
Excellent short-term motion information.
But accumulates drift.
Tactile sensors
Excellent contact information.
But only once physical contact occurs.
Multimodal sensor fusion combines these signals.
Conceptually:
camera ────────┐
LiDAR ─────────┤
IMU ───────────┤
touch ─────────┼→ Fusion Model → State Representation
joint state ───┘
This provides a richer estimate of reality.
29. Early Fusion vs Late Fusion
There are many fusion architectures.
Early fusion
Combine features early:
camera features
+
LiDAR features
+
robot state
↓
single network
Late fusion
Process separately:
Camera → vision model ──────┐
│
LiDAR → geometry model ─────┼→ fusion
│
IMU → motion model ─────────┘
Cross-attention fusion
Modern Transformers can allow modalities to attend to one another.
vision tokens
robot tokens
language tokens
tactile tokens
↓
multimodal Transformer
This architecture is increasingly common in foundation-model robotics.
30. The Reality Gap
Training robots directly in the real world is expensive.
So we often train in simulation.
A simulated robot can perform:
millions
or billions
of interactions without:
breaking hardware
hurting anyone
requiring human resets
But there is a problem.
Simulation is not reality.
This difference is called the reality gap.
Your simulator may model:
friction = 0.7
while the actual floor behaves like:
friction = 0.61
The simulated motor may react instantly.
The real motor has delay.
The simulated camera may be perfectly clean.
The real camera has:
noise
blur
exposure changes
lens distortion
A policy that performs beautifully in simulation may completely fail on the real robot.
31. Sim-to-Real
Sim-to-real is the process of training in simulation and transferring the learned behavior to physical hardware.
Conceptually:
Simulation
↓
Train policy
↓
Transfer
↓
Real robot
The fundamental problem is:
P_simulation ≠ P_real_world
The observations and dynamics are slightly different.
Several techniques help solve this.
32. Domain Randomization
One clever strategy is domain randomization.
Instead of making simulation perfectly realistic, make it wildly variable.
During training randomize:
lighting
textures
camera position
object size
friction
mass
motor strength
latency
sensor noise
gravity
For example:
friction = random.uniform(0.4, 1.2)
object_mass = random.uniform(0.5, 2.0)
camera_noise = random.uniform(0.0, 0.05)
motor_strength = random.uniform(0.8, 1.2)
The policy experiences thousands of different worlds.
Eventually the real world becomes:
just another variation.
Conceptually:
One perfect simulator
❌
Thousands of imperfect randomized simulators
✅
This is a powerful idea.
33. Domain Adaptation
Another approach is domain adaptation.
Suppose we train using:
synthetic images
but deploy using:
real camera images
Domain adaptation tries to make representations work across both domains.
For example:
synthetic image
↓
encoder
↓
representation
and:
real image
↓
same encoder
↓
similar representation
The model learns to ignore differences irrelevant to the task.
Such as:
lighting style
background texture
camera noise
while preserving important features:
object geometry
position
orientation
34. Domain Randomization vs Domain Adaptation
They are related but different.
Domain randomization says:
Train across so much variation that the real environment becomes familiar.
Domain adaptation says:
Explicitly reduce the difference between training and deployment domains.
You will often see both used together.
35. System Identification
Before controlling a robot well, we often need to understand its physical parameters.
This process is called system identification.
Suppose we don't know the precise:
mass
friction
motor constants
joint damping
center of mass
We perform experiments.
Example:
apply motor torque = 2 Nm
observe acceleration
From the response we estimate the hidden physical parameters.
Conceptually:
actions + measured responses
↓
parameter estimation
↓
approximate robot model
For a simple system:
F = ma
if we know F and measure a, we can estimate:
m = F / a
Real robots involve much more sophisticated versions of the same idea.
36. Why System Identification Matters for Learning
Suppose your simulator assumes the robot weighs:
40 kg
but the real robot weighs:
43 kg
That difference changes:
acceleration
balance
motor requirements
contact forces
System identification helps align simulation with reality.
A common pipeline is:
real robot data
↓
system identification
↓
improved simulator
↓
train policy
↓
deploy
Sometimes this process is repeated.
37. Residual Learning
Classical robotics models are often already very good.
Instead of replacing them, we can learn only what the model misses.
This is residual learning.
Suppose a classical controller outputs:
u_classical
A neural network learns a correction:
u_residual
Final action:
u = u_classical + u_residual
For example:
classical controller:
"apply 10 Nm"
learned residual:
"+0.7 Nm because friction is higher than expected"
actual command:
10.7 Nm
This is one of the most practical patterns in Physical AI.
The classical controller handles known physics.
The neural network handles unknown complexity.
38. Learning + Classical Control Hybrids
Pure deep-learning control is exciting.
But production robotics often benefits enormously from classical control.
Consider:
High-level learned policy
↓
desired end-effector position
↓
motion planner
↓
trajectory
↓
PID / MPC controller
↓
motors
The neural network decides:
where should the robot move?
Classical control handles:
how should motors precisely execute that motion?
This separation is powerful.
39. Why Classical Controllers Are Still Important
Control systems often run at very high frequencies.
For example:
100 Hz
500 Hz
1 kHz
A large multimodal Transformer may run at:
5 Hz
10 Hz
20 Hz
You do not want a giant AI model controlling every motor torque directly unless the architecture is designed specifically for that.
Instead:
AI policy at 10 Hz
↓
desired motion
↓
controller at 1,000 Hz
↓
motor commands
This gives you both:
intelligence
+
stability
40. Hierarchical Robot Architecture
A sophisticated robot may contain several control levels.
Language / Goal
"Clean the kitchen"
↓
Task Planner
"Pick up plate"
↓
Skill Policy
"Reach toward plate"
↓
Motion Planner
trajectory
↓
Controller
joint commands
↓
Motors
Different levels operate at different timescales.
For example:
Task planning: every few seconds
VLA policy: 5–20 Hz
Motion planning: 10–100 Hz
Joint controller: 500–2000 Hz
Understanding these layers is extremely important for an AI architect working on robotics.
41. Safety Constraints
Robot learning cannot optimize only:
complete the task
It must also obey:
don't hit humans
don't exceed motor limits
don't fall
don't enter forbidden regions
don't apply dangerous forces
don't damage itself
These are safety constraints.
A generic constrained optimization problem might be:
maximize:
expected reward
subject to:
collision_probability < threshold
joint_torque < limit
velocity < safe_limit
This leads to areas such as:
safe reinforcement learning,
constrained RL,
control barrier functions,
safety filters,
constrained optimization,
runtime safety monitors.
42. Safety Filters
One practical architecture is:
Learned Policy
↓
Proposed Action
↓
Safety Filter
↓
Safe Action
↓
Robot
Suppose the model proposes:
move arm quickly toward target
The safety layer detects:
human hand in trajectory
It replaces the command with:
STOP
The learned model therefore does not have unrestricted authority over hardware.
This idea is likely to remain extremely important in real-world autonomous systems.
43. Hard vs Soft Constraints
A useful distinction:
Soft constraint
Violating it is undesirable.
Example:
use less energy
Represent it as a penalty:
reward -= energy_cost
Hard constraint
Violation is unacceptable.
Example:
never exceed maximum motor torque
This should generally not be represented only by:
"hopefully the reward teaches the model not to do it."
The system architecture should enforce it.
Safety-critical robotics should never depend entirely on the learned policy behaving nicely.
44. Uncertainty-Aware Policies
Machine-learning models can be wrong.
The dangerous case is when they are:
wrong
+
confident
A good robot should understand uncertainty.
Suppose a perception system estimates:
Object A = mug
confidence = 99%
Object B = medicine bottle
confidence = 52%
The robot might autonomously grasp the mug.
But for the uncertain bottle it could:
move camera for better view
ask human
avoid action
This is uncertainty-aware decision making.
45. Types of Uncertainty
Two important forms appear often.
Aleatoric uncertainty
Uncertainty inherent in observations.
Example:
dark room
motion blur
sensor noise
object partially hidden
Even a perfect model cannot completely remove it.
Epistemic uncertainty
Uncertainty because the model lacks knowledge.
Example:
The robot sees an object type never encountered during training.
training distribution:
cups
plates
books
deployment:
unknown industrial valve
The robot should realize:
"I don't know."
That is extremely valuable.
46. Out-of-Distribution Detection
Suppose the robot was trained indoors.
Suddenly it operates:
outside during heavy snow
The sensory input may be far outside the training distribution.
A robust system should detect this.
Conceptually:
observation
↓
OOD detector
↓
normal?
├── yes → continue
└── no → slow down / stop / ask for help
For robotics, refusing to act can be a feature.
47. Active Perception
Uncertainty can sometimes be reduced through action.
Imagine the robot sees:
partially hidden object
Instead of immediately guessing:
"probably a mug"
it can move.
move camera 20 cm left
New observation:
full object visible
Now confidence increases.
This is active perception.
The robot acts not only to change the world, but also to gather information.
That idea tightly connects perception, planning and control.
48. Putting Everything Together
Imagine a household robot receives:
"Bring me the blue mug from the kitchen."
Let's trace the system.
Step 1 — Language understanding
A language/VLM system interprets:
object = blue mug
location = kitchen
goal = deliver object to user
Step 2 — Navigation
Robot navigates toward the kitchen.
SLAM
+
localization
+
path planning
+
obstacle avoidance
may handle this.
Step 3 — Visual understanding
Camera observes several objects.
VLM identifies:
blue mug on counter
Step 4 — Representation
The system estimates:
object position
orientation
reachable region
grasp affordances
Step 5 — Grasp policy
A learned grasp model predicts:
best gripper pose
Step 6 — Motion planning
Classical planner generates:
collision-free arm trajectory
Step 7 — Controller
Low-level controller tracks the trajectory.
Step 8 — Tactile feedback
Gripper detects:
slipping
and increases grip force.
Step 9 — Uncertainty handling
If grasp confidence is low:
reposition camera
or:
attempt a different grasp
Step 10 — World-model reasoning
Robot predicts whether lifting will collide with nearby objects.
Step 11 — Return navigation
Robot navigates back.
Step 12 — Safe delivery
Human hand approaches.
Safety system reduces speed.
Finally:
release mug
This seemingly simple task may involve almost every concept discussed in this article.
49. A Modern Physical AI Stack
A useful mental model for future robots is:
┌─────────────────────────────────────┐
│ Language / Goal Understanding │
│ LLM / VLM │
└──────────────────┬──────────────────┘
↓
┌─────────────────────────────────────┐
│ Task Planning │
│ reasoning / planning / memory │
└──────────────────┬──────────────────┘
↓
┌─────────────────────────────────────┐
│ Multimodal World Representation │
│ vision + depth + touch + state │
└──────────────────┬──────────────────┘
↓
┌─────────────────────────────────────┐
│ World Model │
│ predict consequences │
└──────────────────┬──────────────────┘
↓
┌─────────────────────────────────────┐
│ VLA / Learned Skill Policy │
│ observation + goal → actions │
└──────────────────┬──────────────────┘
↓
┌─────────────────────────────────────┐
│ Motion Planning / Control │
│ MPC / trajectory / PID │
└──────────────────┬──────────────────┘
↓
┌─────────────────────────────────────┐
│ Safety Layer │
└──────────────────┬──────────────────┘
↓
Robot
↓
World
This is not the only architecture.
But it gives you an excellent conceptual map.
50. The Data Problem in Robot Learning
Modern AI progress has been heavily driven by data.
For language models, humanity already produced enormous amounts of text.
For vision models, the internet contains billions of images.
But the internet does not contain billions of datasets like:
camera frame
+
robot joint positions
+
joint velocities
+
forces
+
action
+
next camera frame
Robot interaction data is expensive.
Someone has to physically execute or simulate the action.
Therefore one of Physical AI's biggest bottlenecks is:
How do we obtain enough diverse, high-quality robot experience?
Possible sources include:
human teleoperation
simulation
autonomous exploration
existing robot fleets
synthetic data
internet video
human motion capture
cross-robot datasets
51. Robot Data Is Temporally Structured
Another important difference from ordinary image datasets:
Robot data is sequential.
You cannot randomly treat each frame independently.
frame₁ → action₁
frame₂ → action₂
frame₃ → action₃
Action at time t influences observation at time t+1.
Therefore robotics models must reason about:
history
velocity
causality
temporal dependencies
This is why:
recurrent networks,
Transformers,
diffusion policies,
world models,
are useful for robotics.
52. Action Chunking
Predicting a single tiny motor action at a time can be inefficient.
Instead, policies can predict sequences.
For example:
Input:
current observation
Output:
next 20 actions
Conceptually:
observation
↓
policy
↓
[aₜ, aₜ₊₁, aₜ₊₂ ... aₜ₊₁₉]
This is sometimes called action chunking.
It helps the model capture temporally coherent behavior.
Instead of:
move 1 mm
rethink
move 1 mm
rethink
the system can generate a meaningful movement segment.
53. Diffusion Policies
Diffusion models are not only for images.
They can also model robot trajectories.
A diffusion policy can learn a distribution over possible action sequences.
Instead of predicting:
one exact action
it can represent:
many valid ways to accomplish the task
This matters because robotics is highly multimodal.
Imagine grasping a mug.
You could grasp:
the handle
the side
the rim
Multiple solutions are valid.
A simple regression model might average them and output an invalid motion.
A generative policy can represent multiple possibilities.
54. Offline Reinforcement Learning
Remember the problem:
Real-world exploration is expensive and dangerous.
Suppose we already have a large dataset:
(s, a, r, s')
generated by previous robots or humans.
Can we learn a better policy without collecting new data?
That is offline reinforcement learning.
Instead of:
agent ↔ environment
during training, we have:
fixed dataset
↓
RL algorithm
↓
policy
This is highly attractive for Physical AI because robot fleets could continuously accumulate valuable datasets.
55. Imitation Learning + Reinforcement Learning
These approaches do not need to compete.
A common strategy is:
Step 1:
Human demonstrations
↓
Step 2:
Behavioral cloning
↓
Step 3:
Reasonable initial policy
↓
Step 4:
Reinforcement learning
↓
Step 5:
Improve beyond demonstrations
This solves a major RL problem.
Instead of starting with:
randomly flailing robot
the robot starts with something competent.
Then reinforcement learning fine-tunes the behavior.
56. Robot Learning Is Really About Generalization
Suppose the robot learns:
pick red cup
but only succeeds when:
same cup
same table
same lighting
same camera
same position
That is not very useful intelligence.
We want generalization across:
objects
positions
lighting
rooms
robots
tasks
languages
users
physical conditions
The deeper goal of Physical AI is not merely:
learn a trajectory
but:
Learn reusable concepts about physical interaction.
Concepts like:
graspability
support
containment
collision
balance
slipping
pushing
pulling
opening
closing
stacking
pouring
57. Physical Intelligence Requires Causality
Language models can learn many statistical correlations.
Robotics forces us closer to causal understanding.
Suppose the robot observes:
cup falls
It should understand:
I released the cup.
Therefore the cup fell.
Or:
I pushed the box.
The box moved.
The agent is actively intervening on the world.
In causal terminology:
do(action)
→ observe consequence
Physical interaction generates unusually rich causal learning opportunities.
This is one reason embodied intelligence may become important for building more general forms of AI.
58. The World as a Training Environment
A robot continuously receives a learning signal from reality.
It predicts:
"If I push this, it should move."
Then pushes.
Reality answers:
correct
or:
wrong
The physical world becomes a teacher.
This creates a learning loop:
predict
↓
act
↓
observe
↓
prediction error
↓
update internal model
This resembles how humans learn enormous amounts about physics before ever studying equations.
59. Robot Learning and Classical Robotics Are Converging
It is tempting to frame the future as:
Deep Learning
VS
Classical Robotics
That is probably the wrong framing.
The strongest systems increasingly combine:
learned perception
+
learned semantic reasoning
+
learned policies
+
classical geometry
+
optimization
+
planning
+
control theory
+
safety constraints
For example:
VLM identifies target
↓
learned policy proposes grasp
↓
motion planner checks collision
↓
inverse kinematics computes joints
↓
MPC executes motion
↓
force controller handles contact
The result is far more reliable than blindly replacing every component with one neural network.
60. What Should Be Learned?
A senior robotics architect constantly asks:
Which components actually benefit from learning?
Good candidates include situations with:
complex perception
unknown dynamics
high-dimensional data
difficult hand-engineered rules
semantic reasoning
large behavioral variability
Classical methods remain attractive where:
physics is known
constraints are strict
correctness matters
analytical solutions exist
real-time guarantees matter
Example:
Use learning to answer:
"Where should I grasp this strange object?"
Use deterministic safety logic to enforce:
"Never exceed this joint limit."
That division of responsibility is extremely important.
61. A Practical Hybrid Manipulation Example
Imagine a robot must pick objects from a warehouse bin.
A good architecture might be:
Camera + Depth
↓
Neural perception model
↓
Object segmentation
↓
Learned grasp predictor
↓
Candidate grasp poses
↓
Collision checker
↓
Motion planner
↓
Trajectory
↓
Classical controller
↓
Robot arm
↓
Tactile verification
Learning handles uncertainty.
Classical robotics handles geometric and safety-critical structure.
This pattern is extremely common.
62. Scaling Physical AI
Why are foundation models exciting?
Because scaling produced surprising capabilities in language and vision.
Researchers now ask:
Can a similar scaling effect occur in robotics?
Imagine training on:
1 robot
1 task
10,000 demonstrations
versus:
100 robot types
100,000 tasks
billions of trajectories
images
language
video
touch
actions
The second system might begin learning reusable physical concepts.
Potentially:
door
becomes associated with:
handle
hinge
pulling
opening
obstacle
passage
rather than simply being a visual category.
63. Cross-Embodiment Learning
Different robots have different bodies.
A humanoid has:
arms
hands
legs
A mobile manipulator has:
wheels
arm
gripper
A drone has:
propellers
camera
Could they share knowledge?
This is the cross-embodiment problem.
The high-level skill:
open drawer
may be shared.
But low-level actions differ.
One possible architecture separates:
task representation
↓
general policy
↓
embodiment-specific action decoder
This could allow large robot foundation models to learn across heterogeneous hardware.
64. Foundation Models for Robotics
A future general robot model may learn from:
text
images
videos
robot trajectories
3D scenes
human demonstrations
simulations
tactile signals
Its internal representations might connect:
language concepts
visual concepts
physical concepts
actions
consequences
For example:
"fragile"
would not merely be a dictionary concept.
It would influence physical behavior:
reduce grip force
move more slowly
avoid collisions
That is a deeper form of semantic grounding.
65. Grounding
This brings us to an important AI concept: grounding.
A language model may understand the word:
heavy
through language relationships.
A physical robot can experience:
heavy object
→ requires more force
→ accelerates differently
→ affects balance
The symbol becomes connected to physical experience.
Similarly:
slippery
hot
fragile
soft
heavy
sharp
unstable
can become grounded in sensory and motor experience.
This is one reason Physical AI is intellectually fascinating.
66. A Robot's Internal State
Real robots rarely receive the true state of the environment.
They receive observations.
There is an important distinction:
State:
true condition of the world
Observation:
what sensors tell us about the world
The robot might not know:
object behind wall
exact friction
human intention
hidden obstacle
Therefore many robotics problems are better described as Partially Observable Markov Decision Processes, or POMDPs.
The robot maintains a belief:
b(s)
a probability distribution over possible states.
This connects directly with uncertainty-aware robotics.
67. Memory Matters
Suppose the robot briefly sees a ball roll behind a couch.
Current camera image:
ball invisible
A purely reactive model might behave as if the ball disappeared.
An intelligent robot remembers:
ball probably behind couch
Therefore advanced robot policies often need memory.
Possible mechanisms:
RNN hidden states
Transformers
scene memory
3D maps
object memory
world models
Long-horizon physical tasks require remembering what happened earlier.
68. Long-Horizon Tasks
Picking up an object may take seconds.
But consider:
"Make me breakfast."
The robot may need to:
navigate to kitchen
locate ingredients
open refrigerator
retrieve eggs
retrieve pan
turn on stove
cook
plate food
clean workspace
deliver meal
One small failure can ruin the entire task.
Long-horizon robotics therefore requires:
planning
memory
skill composition
error recovery
world models
uncertainty management
This goes far beyond one neural network mapping an image to a motor command.
69. Skill Libraries
A useful approach is to teach reusable skills:
navigate(location)
pick(object)
place(object, location)
open(drawer)
close(drawer)
pour(container_a, container_b)
A high-level planner composes them.
Example:
Goal:
"Put the milk in the refrigerator."
Plan:
navigate(milk)
pick(milk)
navigate(fridge)
open(fridge)
place(milk, shelf)
close(fridge)
The skills themselves may be learned policies.
The planner may be symbolic, search-based or powered by an LLM/VLM.
70. Error Recovery
Real robotics requires recovery.
Suppose:
grasp fails
A brittle robot stops.
A robust robot reasons:
Why did the grasp fail?
Object slipped.
Try:
different grasp angle
+
higher grip force
Robotics should therefore be designed around:
attempt
observe result
detect failure
replan
retry
rather than assuming perfect execution.
This separates demonstrations from deployed autonomous systems.
71. Closed-Loop vs Open-Loop Policies
This distinction is fundamental.
Open-loop
observe once
↓
generate entire trajectory
↓
execute blindly
If the environment changes, the robot does not adapt.
Closed-loop
observe
act
observe
correct
act
observe
correct
Physical systems strongly benefit from closed-loop control.
Even when a policy predicts action chunks, observations should periodically update the plan.
Reality is too unpredictable for blind execution.
72. Contact-Rich Robotics
Some tasks are particularly difficult because they involve repeated contact.
Examples:
assembling components
inserting connectors
turning screws
opening tight lids
cutting food
folding clothes
tying knots
These tasks depend on tiny forces and geometric details.
Vision alone may not be enough.
Systems often need:
vision
+
force sensing
+
tactile sensing
+
high-frequency control
This is an active frontier of robot learning.
73. Why Manipulation Is Harder Than It Looks
Take a simple instruction:
"Pick up the shirt."
A shirt has:
no fixed shape
many possible folds
self-occlusion
complex friction
many grasp points
The number of possible configurations is enormous.
Rigid-object manipulation is difficult.
Deformable-object manipulation is even harder.
Examples:
clothes
cables
food
bags
rope
fabric
This remains an important frontier.
74. Model Predictive Control and Learned Models
A powerful hybrid combines Model Predictive Control (MPC) with learned dynamics.
At every step:
1. observe current state
2. predict many future trajectories
3. evaluate them
4. execute only first action
5. observe again
6. repeat
Suppose a learned dynamics model predicts:
sₜ₊₁ = fθ(sₜ, aₜ)
MPC can search possible action sequences:
A₁
A₂
A₃
...
simulate their consequences and choose the best.
This combines:
machine learning
+
optimization
+
feedback control
very elegantly.
75. Safety Through Redundancy
Safety-critical autonomous systems rarely rely on one mechanism.
You may have:
learned collision prediction
+
geometric collision checking
+
force limits
+
emergency stop
+
human proximity detector
+
hardware current limits
Multiple layers protect the system.
This is called defense in depth.
If one AI component fails, the robot should not immediately become dangerous.
76. Reality Is the Ultimate Evaluation Dataset
A robot model can score highly in simulation benchmarks and still fail in reality.
Therefore evaluation must consider:
task success rate
collision rate
grasp success
energy consumption
completion time
recovery ability
robustness
safety violations
generalization
But also evaluate under variation.
For example:
new object
new room
new lighting
new camera angle
new payload
new surface
new user instruction
Otherwise you may simply be measuring memorization.
77. Important Evaluation Questions
When testing a physical AI system, ask:
Success
Does the robot complete the task?
Robustness
Does it still succeed under noise?
Generalization
Can it handle unseen scenarios?
Recovery
What happens after a mistake?
Safety
Can failures become dangerous?
Latency
Does the system react quickly enough?
Computational requirements
Can the model actually run onboard?
Energy
How much power does inference consume?
Reliability
Will it work 10 times?
100 times?
10,000 times?
A 90% success rate sounds impressive.
But for a daily household task:
90% × 20 actions
can lead to poor end-to-end reliability.
Reliability compounds.
78. The Robotics Reliability Problem
Suppose a task requires ten sequential subtasks.
Each succeeds 95% of the time.
Overall success is approximately:
0.95¹⁰ ≈ 60%
That surprises many people.
Individual skills can look excellent while the overall robot feels unreliable.
For long-horizon autonomy, we need:
high individual reliability
+
failure detection
+
recovery
+
replanning
This is why robotics systems engineering matters as much as model quality.
79. Physical AI Is Not Just “Put an LLM on a Robot”
This misconception is worth removing.
An LLM can help with:
language
task planning
semantic reasoning
instruction understanding
tool selection
But a physical robot additionally requires:
geometry
state estimation
kinematics
dynamics
control
latency management
sensor fusion
collision avoidance
real-time execution
hardware interfaces
safety
You cannot replace all of robotics with:
prompt = "Walk forward without falling"
Physical intelligence requires understanding the entire stack.
80. Robot Learning Is Multi-Timescale Intelligence
Humans operate across many timescales.
You decide:
"I want coffee."
over seconds.
Your arm plans:
reach toward mug
over hundreds of milliseconds.
Your muscles stabilize motion on much shorter timescales.
Robots are similar.
Strategic reasoning seconds–minutes
Task planning seconds
Skill policy 100–500 ms
Motion control 10–100 ms
Motor control <10 ms
Different algorithms belong at different levels.
A senior AI/robotics architect must understand this.
81. Robot Learning Training Pipeline
A realistic project might follow:
1. Define task
2. Build robot/simulation environment
3. Define observations
4. Define action space
5. Collect demonstrations
6. Train behavioral cloning policy
7. Evaluate in simulation
8. Add domain randomization
9. Transfer to hardware
10. Collect failure cases
11. Fine-tune
12. Add reinforcement learning
13. Add safety layers
14. Evaluate generalization
15. Deploy
16. Collect fleet data
17. Retrain
This resembles the data flywheel of internet AI systems, except collecting physical data is much harder.
82. Observations and Actions Matter Enormously
Before choosing a neural architecture, define the interface.
Observation space
What does the policy see?
RGB?
depth?
point cloud?
joint position?
velocity?
force?
language?
history?
Action space
What does the policy control?
joint torque?
joint velocity?
joint position?
end-effector pose?
skill token?
These choices dramatically change the learning problem.
For example:
policy → joint torques
gives enormous flexibility but requires the policy to learn low-level dynamics.
Whereas:
policy → end-effector target
allows classical controllers to solve the lower-level problem.
83. Choosing the Right Abstraction Level
An important architectural principle is:
Learn at the highest level where learning provides value, and delegate well-understood deterministic behavior to reliable algorithms when appropriate.
For example:
Instead of learning:
motor currents
for a warehouse arm, learn:
desired grasp pose
and let traditional robotics handle motion.
But for highly dynamic quadruped locomotion, directly learning joint commands might make sense.
There is no universally correct abstraction.
It depends on the task.
84. Where This Field Is Going
Physical AI appears to be moving toward several converging ideas.
Large robot foundation models
One model across many tasks and robots.
Vision-language-action models
Language-conditioned policies that act directly.
Massive heterogeneous datasets
Combining experience from many robots.
Simulation at enormous scale
Generating physical interaction data cheaply.
Learned world models
Allowing robots to predict and reason about consequences.
Better multimodal sensing
Vision + touch + force + audio + proprioception.
Human demonstrations
Making it easier to teach robots new behaviors.
Hybrid architectures
Combining foundation models with traditional planning/control.
Continual learning
Robots improving from experience after deployment.
Stronger safety architectures
Separating intelligent decision making from physical safety guarantees.
These ideas are likely to shape the next generation of autonomous machines.
85. The Most Important Mental Shift
When studying ordinary AI, it is tempting to think:
input → model → output
Physical AI requires a different mental model:
┌─────────────┐
│ Observe │
└──────┬──────┘
↓
┌─────────────┐
│ Understand │
└──────┬──────┘
↓
┌─────────────┐
│ Predict │
└──────┬──────┘
↓
┌─────────────┐
│ Decide │
└──────┬──────┘
↓
┌─────────────┐
│ Act │
└──────┬──────┘
↓
Physical World
↓
Observe
The system exists inside a loop.
Every action changes the next input.
Every mistake changes the future state.
Every uncertainty matters.
That is the fundamental difference.
86. The Concepts as One Connected Map
You can now connect all the topics.
ROBOT LEARNING
│
┌────────────────┴────────────────┐
│ │
How to Learn? What to Learn?
│ │
┌───────┴────────┐ ┌──────────┼───────────┐
│ │ │ │ │
Imitation RL Representation Dynamics Policy
Learning │
│ │
Behavioral Model-Free RL
Cloning Model-Based RL
│ │
Demonstrations World Models
Perception side:
Vision
│
├── VLM
│
├── Multimodal Fusion
│
├── Representation Learning
│
└── Tactile Learning
Action side:
Policies
│
├── Visuomotor Policies
├── VLA Models
├── Grasp Learning
├── Manipulation Learning
└── Locomotion Learning
Reality-transfer side:
Simulation
│
├── Sim-to-Real
├── Domain Randomization
├── Domain Adaptation
└── System Identification
Reliability side:
Real Robot
│
├── Classical Control
├── Residual Learning
├── Safety Constraints
└── Uncertainty-Aware Policies
These are not isolated topics.
They are pieces of one architecture.
87. What an AI Architect Should Understand Deeply
If your ultimate specialization is Robot Learning / Physical AI, you do not need to become the world's leading researcher in every subfield.
But you should understand the interfaces between them.
You should be able to reason about:
What sensors should we use?
What representation should we learn?
What should the policy observe?
What action space should it control?
Should we use imitation learning or RL?
Can the task be trained in simulation?
How large is the reality gap?
Should dynamics be modeled analytically or learned?
Where should classical control remain?
How do we detect uncertainty?
What happens when the model fails?
How do we enforce safety?
How will we collect more robot data?
How do we evaluate generalization?
How do we recover after errors?
Which decisions happen at 1 Hz, 10 Hz, 100 Hz and 1 kHz?
That is architectural thinking.
88. Final Perspective
The progression of modern AI can roughly be imagined as:
Perception
"What is this?"
↓
Generation
"What should I say?"
↓
Reasoning
"What should I do?"
↓
Agents
"What tools should I use?"
↓
Physical AI
"How do I change the physical world?"
Physical AI is the point where intelligence becomes grounded in reality.
A robot must understand:
space
motion
force
contact
friction
uncertainty
cause and effect
objects
people
goals
time
risk
It must combine semantic intelligence with physical competence.
That is why the future of robotics will probably not belong purely to:
roboticists
or purely to:
AI researchers.
It will belong to engineers who understand both.
They will know how to combine:
deep learning
+
foundation models
+
reinforcement learning
+
computer vision
+
geometry
+
simulation
+
dynamics
+
planning
+
control
+
real hardware
into one coherent autonomous system.
And that is the essence of Robot Learning / Physical AI.
The ultimate objective is not simply to create machines that can predict the next token.
It is to create machines that can:
perceive the world,
understand the world,
predict the world,
interact with the world,
learn from the result,
and become better at acting within it.
That is when AI stops merely describing reality and begins learning how to operate inside reality.