Robotics System Integration
A robot is not a collection of independent software modules.
It is a physical machine where:
software runs on computers,
computers communicate over networks and buses,
sensors measure the world,
estimators construct an internal belief,
planners decide what should happen,
controllers compute commands,
firmware translates commands into electrical actions,
motor drivers move actuators,
mechanics convert actuator force into motion,
power electronics keep everything alive,
and safety systems must be able to stop the entire chain.
This is why robotics system integration is one of the hardest parts of robotics engineering.
You can understand:
ROS
SLAM
YOLO
control theory
Linux
embedded programming
individually and still struggle when the complete robot refuses to move.
A real robot might fail because:
planner generated no path
controller rejected the path
TF transform is missing
odometry is wrong
DDS messages are not reaching another machine
ROS node crashed
serial packet was corrupted
MCU watchdog triggered
motor controller entered fault state
encoder cable disconnected
battery voltage dropped
wheel is mechanically jammed
emergency stop is active
All of these can produce the same user-visible symptom:
The robot doesn't move.
That is why strong robotics engineers learn to reason through the entire causal chain.
1. What robotics system integration actually means
System integration means making many subsystems work together as one reliable machine.
A simplified autonomous mobile robot contains:
AUTONOMOUS ROBOT
Sensors
│
▼
Drivers
│
▼
ROS Middleware
│
▼
Perception
│
▼
Localization
│
▼
Planning
│
▼
Control
│
▼
Motor Commands
│
▼
MCU / Firmware
│
▼
Motor Driver
│
▼
Motors
│
▼
Mechanical Motion
But even this picture is incomplete.
The real robot also needs:
Linux
networking
power
time synchronization
coordinate transforms
telemetry
logging
watchdogs
fault handling
safety
mechanical calibration
So the actual system looks more like:
┌────────────────────┐
│ AI / Robot │
│ Software │
└─────────┬──────────┘
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
Perception Localization Planning
│ │ │
└──────────────┼──────────────┘
▼
Control
│
▼
ROS / DDS
│
▼
Linux / IPC
│
▼
Network
│
▼
MCU / Firmware
│
▼
Motor Drivers
│
▼
Motors
│
▼
Mechanical Robot
▲
│
Sensors
│
└─────────────── feedback
Meanwhile:
Power ─────────────────────► everything
Safety ────────────────────► everything
Telemetry ─────────────────► operators
Time synchronization ──────► distributed system
System integration is the discipline of making all of this behave correctly together.
2. The most important mindset: think in causal chains
Suppose you type:
ros2 topic pub /cmd_vel geometry_msgs/msg/Twist ...
and the robot does not move.
A beginner may think:
/cmd_velis broken.
A systems engineer asks:
Was the command actually published?
Did the controller subscribe?
Did DDS deliver it?
Did the controller accept it?
Did the hardware interface receive it?
Did the serial packet reach the MCU?
Did firmware decode it?
Did firmware enable the motor driver?
Did the motor driver produce current?
Did the motor produce torque?
Did the wheel mechanically rotate?
Did the encoder report movement?
Did odometry update?
This is a causal chain.
A useful abstraction is:
Intent
↓
Command
↓
Transport
↓
Interpretation
↓
Actuation
↓
Physical effect
↓
Measurement
↓
Feedback
Whenever something fails, find the first point where expected cause stops producing expected effect.
That is the core of robotics debugging.
3. A robot is a cyber-physical system
Normal software mostly manipulates information.
Robotics software manipulates the physical world.
This makes robots:
cyber-physical systems
The cyber side includes:
algorithms
software
networks
operating systems
firmware
The physical side includes:
motors
gearboxes
wheels
arms
batteries
mass
friction
inertia
temperature
mechanical tolerances
These interact continuously.
For example:
software commands wheel = 1 m/s
does not guarantee:
wheel moves at 1 m/s
because the actual result depends on:
battery voltage
motor torque
load
friction
terrain
gearbox
motor controller
PID tuning
wheel diameter
slippage
Robotics engineering therefore requires reasoning across both software and physics.
4. The compute layer
Most modern robots contain multiple computers.
For example:
Main Computer
NVIDIA Jetson / x86 computer
Microcontroller
STM32 / ESP32 / Teensy
Motor Controller
Dedicated embedded controller
Sensor processors
Inside cameras, LiDARs, IMUs, etc.
Each has a different role.
A typical architecture:
HIGH-LEVEL COMPUTE
Jetson / x86 computer
│
│ ROS / Ethernet / USB
▼
Microcontroller
│
│ CAN / PWM / UART
▼
Motor Controllers
│
▼
Motors
The high-level computer might run:
perception
SLAM
Nav2
AI models
planning
logging
UI
The microcontroller might run:
motor control
encoder sampling
safety checks
watchdog
battery monitoring
real-time loops
This division is extremely important.
5. Why you should not run everything on Linux
Linux is powerful, but ordinary Linux is not perfectly deterministic.
Suppose a motor-control loop must run every:
1 millisecond
Linux may occasionally delay a process because of:
scheduler activity
disk I/O
interrupts
other processes
memory pressure
For navigation planning, this may be acceptable.
For a high-frequency motor-control loop, it may not be.
That is why many robots divide responsibilities:
Linux computer
↓
high-level decisions
MCU
↓
hard or near-real-time control
For example:
Linux:
"Drive at 0.6 m/s"
MCU:
compute PWM every 1 ms
read encoders
run PID
check current
check watchdog
This separation creates a more robust system.
6. Linux is part of the robot
Many robotics engineers think of Linux merely as the operating system running ROS.
In reality, Linux itself becomes an important subsystem.
You should understand:
processes
threads
CPU usage
memory
permissions
devices
USB
serial ports
network interfaces
systemd
logs
kernel drivers
udev
real-time scheduling
Imagine a LiDAR normally appears as:
/dev/ttyUSB0
After reboot it becomes:
/dev/ttyUSB1
Your ROS driver now fails.
This is not a SLAM problem.
It is a Linux device-management problem.
A solution may involve:
udev rules
to create a stable device name:
/dev/lidar
This tiny detail can determine whether a production robot starts reliably.
7. Processes and nodes
ROS nodes are normal operating-system processes or components running inside processes.
For example:
lidar_driver
camera_driver
slam_toolbox
nav2_controller
robot_state_publisher
motor_bridge
A node may fail because:
process crashed
dependency missing
parameter invalid
device unavailable
permission denied
port already open
out of memory
GPU error
Therefore debugging ROS sometimes requires ordinary Linux tools:
ps
top
htop
dmesg
journalctl
lsusb
lspci
ip
ss
A good robotics engineer does not stop at:
ros2 node list
They can descend into the operating system when necessary.
8. ROS as the robot's software nervous system
ROS is not the robot.
ROS is the middleware and software framework connecting robot components.
Typical ROS communication patterns include:
Topics
Services
Actions
Parameters
TF
Think of ROS approximately as the robot's information nervous system.
Example:
LiDAR driver
│
└── /scan
│
▼
SLAM / localization
│
└── /map
│
▼
planner
│
└── path
│
▼
controller
│
└── /cmd_vel
│
▼
base driver
Understanding integration means understanding not just each node but the flow between them.
9. Topics
Topics are generally used for continuous streams of information.
Examples:
/camera/image_raw
/scan
/odom
/imu/data
/cmd_vel
/joint_states
A publisher sends messages.
Subscribers receive them.
For example:
LiDAR driver
│
└── publishes /scan
│
├── SLAM node
└── obstacle detector
A topic may exist but still not work correctly.
Questions include:
Is anyone publishing?
Is anyone subscribing?
What is the frequency?
What is the message type?
What is the QoS?
Are timestamps correct?
Are values sensible?
Useful tools:
ros2 topic list
ros2 topic info /scan
ros2 topic echo /scan
ros2 topic hz /scan
10. Services
Services are request-response interactions.
Example:
Client:
"Reset odometry."
Server:
"Done."
They are appropriate for discrete operations.
Examples:
reset
calibrate
enable motor
save map
change operating mode
Services are generally not appropriate for high-rate continuous control.
You would not normally send every motor command through a service.
11. Actions
Actions are useful for operations that:
take time
provide feedback
can be cancelled
Example:
Navigate robot to pose.
The request might take 20 seconds.
During execution:
feedback:
distance remaining = 4.2 m
And the task can be cancelled.
ROS 2 navigation uses actions heavily.
12. Parameters
Parameters configure nodes.
Examples:
maximum velocity
controller gains
map resolution
sensor frame
planner frequency
robot radius
A parameter mistake can look like an algorithm failure.
Suppose:
max_vel_x = 0.0
Everything else may work.
Planner finds a path.
Controller runs.
But robot never moves.
Never underestimate configuration.
13. ROS graph thinking
Imagine your navigation stack as a graph:
LiDAR
│
/scan
▼
Localization
│
/pose
▼
Planner
│
/path
▼
Controller
│
/cmd_vel
▼
Motor Interface
When debugging, traverse the graph.
Ask at each edge:
Does this message exist?
Is its rate correct?
Is its content correct?
Is the next node receiving it?
This turns a huge robot into a sequence of smaller verifiable relationships.
14. DDS
ROS 2 uses DDS-style middleware underneath much of its communication.
DDS stands for:
Data Distribution Service
You usually interact with ROS 2, not DDS directly.
But DDS affects:
discovery
message transport
reliability
QoS
network behavior
multicast
latency
For example:
Laptop sees ROS nodes.
Robot does not see laptop nodes.
The problem may not be ROS application code.
It may be:
firewall
multicast
different ROS_DOMAIN_ID
network isolation
DDS configuration
Wi-Fi behavior
This is why robotics software engineers need networking knowledge.
15. ROS 2 QoS
QoS means:
Quality of Service
It controls how messages should behave.
Important policies include:
reliability
durability
history
depth
deadline
lifespan
A famous integration problem is:
publisher exists
subscriber exists
but no messages arrive
because their QoS policies are incompatible.
For example, sensor streams often use:
best effort
rather than:
reliable
because receiving the newest scan is more important than retransmitting an old lost scan.
QoS is not merely theoretical.
It directly affects whether distributed robot components communicate.
16. Networking
Modern robots are distributed systems.
A robot may contain:
Jetson
MCU gateway
operator laptop
remote server
camera computer
LiDAR
PLC
connected through:
Ethernet
Wi-Fi
CAN
USB
serial
Therefore you should understand:
IP addresses
subnets
ports
TCP
UDP
multicast
DNS
routing
packet loss
latency
bandwidth
17. TCP vs UDP
TCP provides:
reliable
ordered
connection-oriented
communication.
If a packet is lost, TCP retransmits it.
This is useful for things like:
configuration
file transfer
web APIs
UDP is:
connectionless
lower-overhead
not guaranteed delivery
For real-time sensor streams, losing one packet can sometimes be better than waiting for an old packet.
Robotics systems therefore use both styles depending on the problem.
18. Latency matters differently in robotics
Suppose your camera perception system produces:
30 FPS
That sounds good.
But imagine the pipeline is:
Camera capture 30 ms
Network transfer 20 ms
AI inference 80 ms
Postprocessing 20 ms
Planner reaction 40 ms
Total latency:
190 ms
At:
2 m/s
the robot travels approximately:
2×0.19=0.38m2 \times 0.19 = 0.38m
before reacting.
That is 38 centimeters.
Therefore:
high FPS
does not automatically mean:
low control latency
Robotics engineers must reason about end-to-end latency.
19. Bandwidth
Sensors generate large amounts of data.
Example:
RGB camera
1920 × 1080
30 FPS
If uncompressed RGB uses 3 bytes per pixel:
1920×1080×3×301920 \times 1080 \times 3 \times 30
which is about:
186 MB/s
before protocol overhead.
Add:
multiple cameras
depth
LiDAR
telemetry
and network bandwidth becomes significant.
This is why robots sometimes use:
compression
hardware encoding
dedicated Ethernet
sensor-side processing
20. Time synchronization
Distributed robots depend heavily on time.
Imagine:
camera frame timestamp = 10.000 s
IMU timestamp = 10.050 s
LiDAR timestamp = 9.970 s
If clocks are wrong, sensor fusion can become inaccurate.
For mobile robots, a small timestamp error combined with motion can create significant geometric error.
Systems may use:
NTP
PTP
hardware timestamps
sensor synchronization
Time is a hidden integration dependency.
If sensor fusion behaves strangely, timestamps should always be inspected.
21. Microcontrollers
Microcontrollers run close to hardware.
Common examples include:
STM32
ESP32
Teensy
AVR
RP2040
An MCU may be responsible for:
reading encoders
generating PWM
reading battery voltage
monitoring current
running PID control
reading bump switches
executing emergency-stop logic
communicating with the main computer
Unlike your Linux computer, the MCU operates directly with:
GPIO
ADC
timers
interrupts
PWM
CAN
UART
SPI
I2C
This layer is where software meets electronics.
22. Firmware
Firmware is the software running on embedded controllers.
A typical mobile-base firmware loop might look conceptually like:
while (1) {
read_encoders();
compute_velocity();
check_watchdog();
check_estop();
run_pid();
set_motor_pwm();
publish_status();
}
This loop might run at:
100 Hz
500 Hz
1 kHz
depending on the system.
Firmware must often be:
predictable
fault-tolerant
low-latency
safe
A mistake at this layer can affect physical hardware immediately.
23. Real-time control
Imagine a control loop intended to run every:
10 ms
Ideal timing:
0 ms
10 ms
20 ms
30 ms
40 ms
Poor timing:
0 ms
12 ms
18 ms
36 ms
41 ms
The second version has:
jitter
Control systems are sensitive to timing.
This is why:
motor control
joint servo control
high-frequency stabilization
are often performed on microcontrollers or real-time processors.
24. Watchdogs
A watchdog detects when a subsystem stops behaving correctly.
Example:
Linux normally sends:
velocity command every 50 ms
Firmware tracks the last command time.
If no command arrives for:
500 ms
firmware executes:
motor command = 0
This protects against:
ROS crash
network loss
Linux freeze
process failure
A robot should not continue driving forever because the controlling computer disappeared.
This is one of the most important practical safety patterns.
25. Communication between Linux and MCU
Common interfaces include:
UART
USB serial
CAN
Ethernet
SPI
Suppose Linux wants:
linear velocity = 0.5 m/s
angular velocity = 0.2 rad/s
It might encode a packet:
HEADER
COMMAND_ID
LINEAR_VELOCITY
ANGULAR_VELOCITY
CHECKSUM
The MCU parses it.
Potential problems include:
wrong baud rate
corrupted bytes
packet framing
endianness
integer scaling
checksum failure
buffer overflow
stale packets
protocol mismatch
So a failure below ROS may appear exactly like a ROS failure above it.
26. CAN bus
CAN is extremely common in robotics and automotive systems.
It is designed for reliable communication between embedded devices.
Devices may include:
motor controllers
battery management system
steering controller
safety PLC
sensor modules
A CAN network might look like:
Main Controller
│
──────CAN BUS────────────────────
│ │ │
Motor L Motor R BMS
CAN messages have identifiers.
For example:
0x101 → left motor command
0x102 → right motor command
0x201 → left encoder feedback
Common integration issues include:
wrong bitrate
incorrect termination
duplicate IDs
bus-off state
wiring polarity
grounding issues
message scaling errors
27. Electrical buses are physical systems too
A communication bus can fail because of software.
It can also fail because of electronics.
For example, CAN often requires approximately:
120 Ω termination
at both ends of the bus.
If termination is wrong, the electrical signal itself may be degraded.
Similarly, serial communication problems can arise from:
ground mismatch
noise
long cables
voltage-level mismatch
EMI
poor connectors
A robotics engineer occasionally needs an oscilloscope, not another log statement.
28. Sensors
Sensors are how robots observe internal and external state.
Common sensors include:
encoders
IMUs
LiDAR
cameras
depth cameras
ultrasonic sensors
GPS/GNSS
force sensors
current sensors
temperature sensors
limit switches
Every sensor has:
noise
bias
latency
resolution
range
sampling rate
failure modes
Real sensors never produce perfect truth.
29. Sensor drivers
A sensor driver converts device-specific communication into data the robotics software understands.
Example:
LiDAR hardware
│
Ethernet packets
▼
LiDAR driver
│
sensor_msgs/LaserScan
▼
ROS
The driver must handle:
device initialization
packet parsing
timestamps
coordinate frames
configuration
reconnection
error handling
A broken driver can produce perfectly valid-looking ROS messages containing incorrect data.
Always inspect actual values.
30. Never trust a sensor merely because it publishes
Suppose:
ros2 topic hz /scan
shows:
10 Hz
You might think the LiDAR is fine.
But maybe every range value is:
0
or:
inf
Or perhaps the scan is rotated 180 degrees.
Therefore sensor validation includes:
Is data arriving?
Is frequency correct?
Are values physically plausible?
Is the frame correct?
Are timestamps correct?
Is orientation correct?
Visualization tools such as RViz are invaluable here.
31. Calibration
Sensors often require calibration.
Examples:
camera intrinsic calibration
camera-to-robot extrinsic calibration
IMU bias calibration
wheel radius calibration
wheelbase calibration
joint zero calibration
LiDAR extrinsic calibration
A system can be algorithmically correct and still perform poorly because calibration is wrong.
Suppose actual wheel radius is:
0.10 m
but software assumes:
0.11 m
Your odometry will systematically overestimate distance.
Over several meters, the localization error becomes significant.
32. Coordinate frames
Robotics depends heavily on coordinate systems.
Common frames include:
map
odom
base_link
laser
camera_link
imu_link
Suppose the LiDAR reports an obstacle at:
x = 2 m
But:
2 meters relative to what?
The answer is its coordinate frame.
A point only becomes meaningful when you know:
frame
position
orientation
timestamp
33. TF
ROS uses TF to manage coordinate transformations.
A typical mobile robot transform tree:
map
│
▼
odom
│
▼
base_link
├── laser
├── camera_link
└── imu_link
This means software can ask:
Where was the LiDAR relative to the map at time T?
TF combines transformations through the tree.
For example:
map → odom
odom → base_link
base_link → laser
to derive:
map → laser
34. Why TF causes so many robotics problems
Suppose LiDAR data exists.
SLAM runs.
Planner runs.
But navigation fails because:
No transform from laser to base_link
or:
map → odom missing
or:
transform timestamp too old
Common TF failures include:
missing frame
wrong parent
wrong child
incorrect static transform
frame-name mismatch
timestamp problems
transform loop
multiple publishers for same transform
TF is one of the first things experienced ROS engineers check.
35. Static vs dynamic transforms
Some relationships never change.
Example:
camera mounted 20 cm above base
This can use a:
static transform
Other relationships change continuously.
Example:
odom → base_link
because the robot moves.
This is a:
dynamic transform
Mixing these incorrectly leads to strange behavior.
36. Encoders
Wheel encoders measure motor or wheel rotation.
Suppose an encoder produces:
2048 ticks/revolution
If the wheel circumference is:
2πr2\pi r
and:
r = 0.1 m
then one full rotation travels approximately:
2π(0.1)≈0.628m2\pi(0.1) \approx 0.628m
Each tick corresponds to approximately:
0.628/2048≈0.000307m0.628/2048 \approx 0.000307m
or:
0.307 mm
Encoder counts help estimate wheel movement.
37. Encoder failure examples
If the left encoder wire disconnects:
left wheel physically moves
left encoder reports zero
Odometry may conclude the robot is rotating.
The navigation stack may then behave unpredictably.
This illustrates an important principle:
Higher software layers assume lower layers tell the truth.
When those assumptions break, high-level behavior can look irrational.
38. Odometry
Odometry estimates robot motion from local sensors.
For a differential-drive robot, wheel encoders are often used.
If both wheels move equally:
robot moves forward
If:
left wheel faster than right
the robot turns.
Odometry typically estimates:
x
y
heading
linear velocity
angular velocity
and publishes:
odom → base_link
39. Odometry drifts
Odometry is not globally accurate.
Errors come from:
wheel slip
wheel diameter error
encoder noise
uneven terrain
mechanical compliance
integration error
The error accumulates over time.
Therefore:
odometry
provides useful short-term local motion, while:
SLAM / localization
corrects long-term drift.
40. Localization
Localization answers:
Where am I?
The robot may combine:
wheel odometry
IMU
LiDAR
camera
GPS
map
A localization system produces an estimate such as:
x = 3.2 m
y = 5.8 m
yaw = 1.1 rad
along with uncertainty.
Localization is foundational.
Planning a path from the wrong location produces the wrong plan.
41. State estimation
Localization is often part of a broader:
state estimation
problem.
The robot's state may include:
position
orientation
velocity
acceleration
joint states
sensor bias
Algorithms include:
Kalman Filter
Extended Kalman Filter
Unscented Kalman Filter
particle filter
factor graph optimization
In production robots, state estimation often becomes the bridge between noisy physical sensors and deterministic planning software.
42. Sensor fusion
No sensor is perfect.
Wheel encoders are good for:
short-term motion
but drift.
IMU is good for:
angular velocity
acceleration
but suffers from bias and integration drift.
GPS provides:
global position
but may be noisy or unavailable indoors.
LiDAR localization provides:
map-relative position
but can fail in feature-poor environments.
Sensor fusion combines their strengths.
Conceptually:
Encoder ─┐
IMU ─────┼──► State Estimator ─► Robot State
GPS ─────┤
LiDAR ───┘
43. Perception
Perception answers questions about the environment.
Examples:
Where are obstacles?
Where are people?
Which object is the target?
Where is the road?
Where are graspable objects?
Perception may use:
classical computer vision
deep learning
LiDAR processing
3D geometry
sensor fusion
Examples:
YOLO
semantic segmentation
depth estimation
point-cloud clustering
object tracking
Perception converts raw sensor data into useful environmental understanding.
44. Perception is not enough
Suppose YOLO detects:
person
bounding box = [420, 220, 670, 700]
That tells you where the person appears in the image.
The robot may still need:
3D position
distance
velocity
world-frame position
confidence
identity
So real integration often looks like:
Camera
↓
Object Detector
↓
2D Detection
↓
Depth / geometry
↓
3D Object Position
↓
TF transformation
↓
Map-frame target
↓
Planner
AI output is only one stage in a robotics pipeline.
45. AI inside robotics
AI models may perform:
object detection
segmentation
speech recognition
language understanding
grasp prediction
policy learning
world modeling
semantic mapping
But AI does not remove systems engineering.
In fact it often adds new dependencies:
GPU
model weights
preprocessing
postprocessing
latency
memory
confidence thresholds
fallback logic
An AI system that takes:
500 ms
to detect an obstacle may be mathematically impressive and operationally useless for a fast-moving robot.
46. Planning
Planning answers:
What should the robot do?
For navigation, this may include:
global planning
local planning
trajectory planning
behavior planning
A global planner may produce:
start
↓
waypoints
↓
goal
while avoiding known obstacles.
The plan is typically expressed in a coordinate frame such as:
map
47. Global vs local planning
A global planner reasons over a larger map.
Example:
Go through hallway A,
turn left,
then enter room B.
A local planner or controller handles nearby conditions:
person temporarily blocking path
chair moved
robot slightly off trajectory
Conceptually:
Global Planner
│
▼
Long-range path
│
▼
Local Controller
│
▼
Immediate motion
48. Control
The planner says:
Follow this trajectory.
The controller determines:
What command should I issue right now?
For a mobile robot:
linear velocity
angular velocity
For a robotic arm:
joint position
joint velocity
joint torque
Control closes the loop between desired behavior and measured behavior.
49. Closed-loop control
Imagine commanding:
wheel speed = 10 rad/s
If you simply apply fixed motor voltage, wheel speed may vary with:
battery
load
friction
slope
Closed-loop control measures actual speed.
Desired Speed
│
▼
Controller
│
▼
Motor
│
▼
Actual Speed
│
└──────── feedback
The controller computes:
error=desired−measurederror = desired - measured
and changes the motor command accordingly.
50. PID control
A common controller is:
PID
where:
u(t)=KPe(t)+KI∫e(t)dt+KDde(t)dtu(t) = K_P e(t) + K_I \int e(t)dt + K_D \frac{de(t)}{dt}
The terms are:
P: react to current error
I: react to accumulated error
D: react to rate of error change
For wheel velocity:
desired = 10 rad/s
measured = 8 rad/s
error = 2 rad/s
PID uses this error to increase motor effort.
Poorly tuned PID can cause:
oscillation
overshoot
slow response
instability
51. Control frequency
Control loops operate at specific frequencies.
Example:
high-level navigation: 10–50 Hz
wheel velocity control: 100–1000 Hz
joint torque control: 1 kHz+
These numbers vary by robot.
The important idea is:
Different layers operate at different timescales.
You do not need a global path planner at 1000 Hz.
You may absolutely need torque control at 1000 Hz.
52. Motors
Common robot motors include:
DC motors
BLDC motors
stepper motors
servo motors
AC motors
A motor converts electrical energy into mechanical torque.
Software may request:
velocity
position
torque
but ultimately the motor driver controls:
voltage
current
commutation
PWM
depending on motor type.
53. Motor drivers
The computer cannot usually power motors directly.
A motor may require:
10 A
30 A
100 A
while a microcontroller GPIO pin can provide only tiny current.
Therefore:
MCU
↓
Motor Driver
↓
Motor
The motor driver handles power electronics.
It may provide:
current control
PWM
commutation
overcurrent protection
temperature protection
fault reporting
54. BLDC motor control
Brushless DC motors require electronic commutation.
The controller must energize motor phases appropriately based on rotor position.
Advanced motor controllers often use:
FOC
Field-Oriented Control.
FOC transforms motor currents into rotating coordinate frames and independently controls torque-producing and flux-producing components.
You do not need to implement FOC to integrate every robot, but understanding that motor controllers contain their own control systems is important.
55. Torque, speed, and gearing
A motor's raw characteristics may not match the robot.
A gearbox trades:
speed
for:
torque
For example:
motor speed = 3000 RPM
gear ratio = 30:1
approximately:
output speed = 100 RPM
while torque increases roughly proportionally, minus losses.
Mechanical drivetrain selection directly affects what your control software can achieve.
56. Mechanical systems matter to software
Suppose your controller oscillates.
You might blame PID tuning.
But the true cause could be:
wheel loose on shaft
gearbox backlash
flexible mounting
uneven wheel diameter
damaged bearing
mechanical resonance
Physical systems contain:
friction
backlash
compliance
inertia
resonance
wear
Software models are approximations.
A robotics engineer learns to inspect hardware when software explanations stop making sense.
57. Backlash
Backlash is mechanical play between components.
For example, in a gearbox:
motor turns slightly
output does not move yet
because gear teeth have clearance.
When direction reverses, the controller may command movement but see no immediate response.
This can cause:
position error
oscillation
poor precision
No amount of ROS debugging fixes worn gears.
58. Friction
Friction creates another integration issue.
Suppose small motor commands produce:
no movement
until command exceeds a threshold.
Then suddenly:
robot jumps
This can be caused by:
static friction
The controller may need compensation.
Mechanical behavior therefore affects control design.
59. Power
Power is one of the most underestimated parts of robot integration.
A robot may contain:
battery
DC/DC converters
motor power rail
5V rail
12V rail
24V rail
computer power supply
sensor power
A simplified architecture:
Battery
│
├──► Motor Driver ─► Motors
│
├──► 12V Converter ─► Sensors
│
└──► 5V Converter ─► MCU
If power is unstable, every software layer above it becomes unstable.
60. Voltage sag
Suppose the battery is nominally:
24 V
When motors accelerate heavily, current spikes.
Internal battery resistance may cause voltage to fall:
24 V
↓
20 V
↓
17 V
The onboard computer may reboot.
From the software perspective:
ROS nodes suddenly disappear
But the real cause is electrical.
This is:
voltage sag
A systems engineer checks:
battery voltage under load
current draw
converter limits
wiring resistance
61. Grounding
Electronic systems require proper grounding.
Poor grounding can create:
sensor noise
communication errors
MCU resets
encoder corruption
random faults
Motors are particularly noisy electrical devices.
High-current switching produces electromagnetic interference.
Routing:
motor power
signal wires
encoder wires
communication lines
incorrectly can cause intermittent problems that look like software bugs.
62. Battery management systems
Many battery packs include a:
BMS
Battery Management System.
The BMS monitors:
cell voltages
current
temperature
state of charge
overcurrent
undervoltage
overvoltage
It may disconnect power if unsafe conditions occur.
If the robot suddenly shuts down under heavy acceleration, the BMS may be protecting the battery.
63. Power budgeting
Every subsystem consumes power.
Example:
Jetson 30 W
LiDAR 15 W
2 cameras 10 W
MCU 2 W
motors average 80 W
motors peak 400 W
Average:
137 W
but peak may exceed:
450 W
The power system must support peak demand, not merely average demand.
64. Telemetry
Telemetry means observing robot state while it operates.
Useful telemetry includes:
CPU temperature
GPU temperature
battery voltage
battery current
motor current
motor temperature
encoder velocity
network latency
localization confidence
planner state
control command
fault codes
disk usage
ROS node health
A production robot without telemetry is extremely difficult to debug.
65. Logging
Logs should help reconstruct:
What happened?
Good logs include:
timestamp
component
severity
event
context
error code
For example:
14:31:22.150 MOTOR_DRIVER ERROR
Left motor overcurrent detected: 34.2 A
is far more useful than:
ERROR
When debugging intermittent failures, logs are often your only evidence.
66. Metrics
Logs describe events.
Metrics describe quantities over time.
Examples:
battery voltage
motor current
GPU utilization
planner latency
network packet loss
localization covariance
control error
A graph may reveal:
motor current rises
battery voltage falls
computer resets
Now the causal chain becomes obvious.
67. Tracing
Tracing follows one operation across multiple components.
Suppose navigation becomes slow.
A trace could show:
Camera capture 30 ms
Detection 120 ms
Tracking 20 ms
Planner 25 ms
Controller 5 ms
Now you know AI inference dominates latency.
Distributed tracing is increasingly useful as robots become complex software systems.
68. Health monitoring
A robot should continuously know whether subsystems are healthy.
For example:
LiDAR:
healthy
Camera:
healthy
Localization:
degraded
Left encoder:
failed
Battery:
low
Motor controller:
healthy
Then higher-level software can respond intelligently.
Instead of continuing navigation with broken localization, the robot may:
stop
relocalize
notify operator
return home
69. Safety
Safety must not be an afterthought.
Robots move physical hardware.
A software mistake can damage:
robot
environment
equipment
people
Safety is therefore layered.
70. Emergency stop
An emergency stop should usually be capable of stopping hazardous motion even if high-level software is broken.
Bad architecture:
E-stop button
↓
ROS topic
↓
Python node
↓
motor command = 0
If Linux freezes, this may fail.
Safer systems often use a hardwired path:
E-stop
↓
Safety Relay / Controller
↓
Motor Enable
High-level software may also receive E-stop state, but it should not necessarily be the only mechanism.
71. Hardware vs software safety
Software safety might include:
speed limit
geofence
collision avoidance
watchdog
trajectory validation
Hardware safety might include:
emergency stop
relay
fuse
circuit breaker
mechanical brake
safety-rated controller
Strong safety systems use multiple independent layers.
This is:
defense in depth
72. Fail-safe behavior
A fail-safe system asks:
If this component fails, what should the robot do?
Examples:
command communication lost
→ stop
localization lost
→ stop or enter recovery
battery critically low
→ stop or dock
motor overtemperature
→ reduce power or stop
encoder failure
→ disable affected actuator
Fail-safe design means failure leads toward a safer state.
73. Fault detection
A robot should detect unreasonable states.
Example:
motor command = forward
encoder velocity = zero
motor current = very high
Possible interpretation:
mechanical jam
Another case:
motor command = zero
encoder velocity = high
Possible interpretation:
encoder fault
robot pushed externally
control failure
Cross-checking multiple signals enables fault detection.
74. Plausibility checks
Sensors should be checked against physical expectations.
Examples:
battery voltage cannot be -30 V
wheel velocity cannot jump from 0 to 500 m/s
IMU acceleration should be physically plausible
encoder count change has maximum rate
temperature cannot rise 100°C in one millisecond
These checks catch:
bad packets
overflow
unit errors
sensor faults
75. Units
Unit mistakes are legendary engineering failures.
Robotics commonly uses:
meters
radians
seconds
newtons
newton-meters
volts
amps
But hardware may provide:
millimeters
degrees
encoder ticks
RPM
milliseconds
raw ADC counts
Suppose firmware sends:
wheel velocity = 1000
Does that mean:
1000 RPM?
1000 encoder ticks/s?
1.000 rad/s?
1000 mm/s?
Interfaces should define units explicitly.
76. Coordinate conventions
Another class of bugs involves orientation conventions.
Common conventions include:
x forward
y left
z up
But a sensor may use:
x right
y down
z forward
Cameras often use different coordinate conventions from mobile robots.
A sign error can turn:
turn left
into:
turn right
Always document coordinate frames.
77. Degrees vs radians
Suppose an API expects:
radians
and you send:
90
thinking degrees.
You intended:
90∘=1.5708rad90^\circ = 1.5708 rad
but instead gave:
90 rad
which represents more than 14 full rotations.
Many robotics bugs are embarrassingly simple interface mismatches.
78. Data types and overflow
Embedded systems commonly use fixed-width integers.
Suppose encoder count uses:
int16_t
Maximum:
32767
Then:
32766
32767
-32768
-32767
because the integer wraps.
If the Linux software does not handle rollover, odometry may suddenly report a huge motion.
Understanding data representation matters.
79. Endianness
Suppose MCU sends:
0x12345678
Bytes may appear as:
12 34 56 78
or:
78 56 34 12
depending on protocol ordering.
If sender and receiver disagree, numerical values become nonsense.
This is:
endianness
Another classic integration bug.
80. Configuration management
A real robot may depend on hundreds of configuration values:
IP addresses
ROS parameters
PID gains
camera calibration
wheel radius
CAN IDs
serial baud rates
AI thresholds
planner parameters
safety limits
These should not exist as random hardcoded values scattered across repositories.
Good systems centralize configuration and track versions.
You should be able to answer:
Which configuration was running when this failure happened?
81. Hardware revisions
Robots evolve.
Example:
Robot v1
wheel radius = 100 mm
Robot v2
wheel radius = 105 mm
If both run the same incorrect configuration, one robot's odometry may be wrong.
Production robotics often requires:
hardware revision tracking
serial-number configuration
calibration database
Software must know which physical machine it is controlling.
82. Startup sequencing
A robot cannot always start everything simultaneously.
Example dependency chain:
network ready
↓
hardware interfaces ready
↓
sensors ready
↓
TF available
↓
localization ready
↓
planner ready
↓
navigation enabled
If navigation starts before localization has initialized, it may fail.
Production systems therefore use:
lifecycle management
health checks
dependency orchestration
83. ROS lifecycle nodes
ROS 2 supports managed lifecycle states such as:
unconfigured
inactive
active
finalized
A navigation node may exist but remain:
inactive
Then it publishes nothing.
This produces another classic symptom:
Everything looks launched, but navigation doesn't work.
The node is alive but not operational.
Lifecycle state must be inspected.
84. Recovery behavior
Real robots encounter unexpected situations.
Examples:
path blocked
localization lost
sensor temporarily unavailable
wheel stuck
network reconnecting
A robust system should attempt recovery.
For navigation:
path blocked
↓
replan
↓
still blocked
↓
rotate / clear local costmap
↓
retry
↓
fail safely
The robot should not simply crash because one assumption became false.
85. State machines
Complex robot behavior is often represented as a state machine.
Example:
IDLE
↓
LOCALIZING
↓
NAVIGATING
↓
DOCKING
↓
CHARGING
Transitions might occur because:
goal received
battery low
navigation failed
dock detected
E-stop pressed
State machines make behavior explicit and easier to reason about.
86. Behavior trees
Behavior trees are common in modern robotics, especially navigation.
A simplified navigation behavior tree might be:
NavigateToGoal
│
├── ComputePath
│
├── FollowPath
│
└── Recovery
├── ClearCostmap
└── Spin
They provide structured:
fallback
retry
sequence
condition
behavior.
Nav2 uses behavior-tree concepts extensively.
87. Deterministic vs probabilistic systems
Many low-level components are approximately deterministic.
Example:
command motor PWM = 40%
But perception and localization are probabilistic.
Example:
object detected with confidence = 0.82
or:
pose covariance = ...
Integration must handle uncertainty.
Do not treat:
AI confidence 0.51
like an unquestionable boolean fact.
Higher-level logic should account for confidence and failure modes.
88. Graceful degradation
Suppose one camera fails.
Should the robot immediately shut down?
Maybe.
Maybe not.
A well-designed system might support:
normal mode
degraded mode
safe-stop mode
Example:
rear camera fails
↓
disable reverse autonomous motion
↓
continue limited forward operation
This is graceful degradation.
It is common in mature autonomous systems.
89. The full navigation causal chain
Suppose the user asks:
Move to the kitchen.
A simplified causal chain is:
Goal
↓
Navigation action
↓
Localization
↓
Global planner
↓
Path
↓
Local controller
↓
/cmd_vel
↓
Base hardware interface
↓
Serial/CAN command
↓
MCU
↓
Motor controller
↓
Motor current
↓
Motor torque
↓
Wheel rotation
↓
Robot motion
↓
Encoders / IMU
↓
Odometry
↓
Localization update
↓
Controller feedback
This loop repeats continuously.
If the robot doesn't move, inspect the chain.
90. Debugging example: planner problem
Suppose:
goal received
but:
no /cmd_vel
First check:
Was a path generated?
No.
Then the issue is above the controller.
Potential causes:
goal outside map
occupancy map blocked
planner inactive
costmap invalid
localization missing
TF unavailable
You do not need to inspect motor drivers yet.
This is causal debugging.
91. Debugging example: controller problem
Suppose:
valid path exists
but:
/cmd_vel = 0
Now investigate:
controller status
controller parameters
local costmap
speed limits
robot footprint
goal tolerances
obstacle detection
Again, firmware is probably not yet the relevant layer.
92. Debugging example: hardware interface problem
Suppose:
/cmd_vel = 0.5 m/s
is being published.
But robot does not move.
Next question:
Is the hardware interface receiving it?
Check:
subscriber
logs
command conversion
If it receives the command:
Is it sending the expected command to the MCU?
Inspect:
serial packets
CAN frames
hardware logs
93. Debugging example: MCU problem
Suppose valid CAN messages reach the MCU.
But motor command remains zero.
Potential causes:
E-stop active
watchdog fault
firmware state machine not enabled
command rejected
checksum error
safety interlock
motor fault
Now ROS debugging would be wasted effort.
You have already localized the fault beneath ROS.
94. Debugging example: motor driver problem
Suppose MCU outputs the correct motor command.
Motor driver still produces no motor current.
Inspect:
enable pin
fault line
supply voltage
overcurrent fault
temperature fault
driver configuration
At this point an oscilloscope or multimeter may be more useful than RViz.
95. Debugging example: mechanical problem
Suppose:
motor current is high
motor shaft turns
wheel does not
Possible causes:
loose coupling
broken gearbox
sheared key
wheel disconnected
This is mechanical.
The software stack may be completely correct.
96. Debugging example: undervoltage
Suppose the robot moves normally slowly.
During acceleration:
motors start
computer reboots
Possible chain:
high acceleration
↓
motor current spike
↓
battery voltage sag
↓
DC/DC converter input too low
↓
Jetson brownout
↓
Linux reboots
↓
ROS disappears
A beginner may say:
ROS crashes when the motor moves.
A systems engineer says:
Let's measure the power rail.
97. Debugging example: TF
Suppose:
LiDAR publishes
map exists
planner active
controller active
but Nav2 complains:
Could not transform base_link to map
Then the navigation stack cannot determine where the robot is.
Potential missing chain:
map
↓
odom
↓
base_link
Maybe:
AMCL is inactive
odometry node stopped
frame IDs don't match
timestamps are stale
The robot may physically work perfectly while the software coordinate graph is broken.
98. Debugging example: DDS/network
Suppose:
robot computer sees /scan
operator laptop does not
Both run ROS 2.
Check:
same ROS_DOMAIN_ID?
same subnet?
firewall?
multicast enabled?
Wi-Fi client isolation?
DDS implementation?
This is a distributed-systems problem.
Not a LiDAR problem.
99. The debugging ladder
A useful debugging ladder is:
1. Physical
2. Electrical
3. Firmware
4. Communication bus
5. Operating system
6. Middleware
7. ROS graph
8. State estimation
9. Planning
10. Control
11. Behavior / AI
You do not always need to debug bottom-up.
Instead use observations to determine where the causal chain breaks.
100. Binary search for robotics systems
Imagine this chain:
Planner
↓
Controller
↓
ROS topic
↓
Driver
↓
CAN
↓
MCU
↓
Motor
Instead of checking everything randomly, test a midpoint.
Question:
Is /cmd_vel correct?
If no:
problem is above /cmd_vel
If yes:
problem is below /cmd_vel
Then check:
Does CAN contain the correct command?
If yes:
problem is below CAN
You are essentially performing binary search over the causal chain.
This is far faster than random debugging.
101. Verify inputs and outputs at every boundary
Every subsystem has:
input
processing
output
For example:
Controller
Input:
path
pose
costmap
Output:
/cmd_vel
When debugging:
Are inputs correct?
Is output correct?
If inputs are wrong:
upstream problem
If inputs are correct and output wrong:
component problem
If output correct but next system sees wrong data:
interface problem
This pattern works almost everywhere.
102. Interfaces are where many bugs live
A component may work perfectly in isolation.
Integration fails at boundaries.
Examples:
meters vs millimeters
degrees vs radians
little-endian vs big-endian
frame A vs frame B
timestamp system A vs B
ROS QoS mismatch
CAN scaling mismatch
wrong message version
This is why good interfaces are:
explicit
typed
versioned
documented
validated
103. Hardware-in-the-loop testing
Some robot behavior can be tested without the full machine.
For example:
real controller computer
real firmware
simulated motor feedback
This is:
Hardware-in-the-Loop
or:
HIL
HIL testing can validate:
control interfaces
fault handling
communication
timing
before risking physical hardware.
104. Simulation
Simulation lets us test:
navigation
perception
planning
control
without real hardware.
Tools include:
Gazebo
Isaac Sim
Webots
MuJoCo
But simulation has limits.
Real robots introduce:
sensor noise
latency
wheel slip
network drops
lighting
mechanical tolerances
power issues
This creates the:
sim-to-real gap
105. Mocking
You can replace a subsystem with a fake implementation.
For example:
fake LiDAR publisher
fake motor controller
fake localization source
Then test other components independently.
Suppose navigation works with fake odometry but fails with real odometry.
You have localized the integration issue.
Mocks are useful far beyond normal application software.
106. Replay
ROS bag files allow recorded sensor streams to be replayed.
Example:
real robot run
↓
record:
/scan
/imu
/odom
/camera
Later:
play recording
↓
test SLAM
perception
localization
This makes difficult field failures reproducible.
Reproducibility is one of the most valuable debugging advantages you can create.
107. Field debugging
Robots often fail only in real environments.
Examples:
direct sunlight breaks depth camera
metal structure interferes with compass
Wi-Fi drops in warehouse corner
rain affects LiDAR
floor causes wheel slip
battery behaves differently when cold
Strong robotics teams collect:
logs
bags
metrics
fault snapshots
during field operation.
Without evidence, intermittent failures are extremely difficult to diagnose.
108. Reproducibility
When a robot fails, record:
software commit
firmware version
configuration
hardware revision
map
AI model version
ROS bag
system logs
battery state
time
environment
Then you can reconstruct the system.
If someone says:
It failed yesterday.
but cannot tell you what was running, debugging becomes much harder.
109. Versioning the whole robot
A production robot has multiple versions:
application software
ROS packages
firmware
kernel
drivers
AI model
calibration
configuration
hardware
A meaningful robot build might be:
robot software: 2.7.1
firmware: 1.9.4
AI model: detector-v12
calibration: robot-042-cal-2026-08
hardware: rev-C
System integration includes controlling this complexity.
110. Updates
Updating one part can break another.
Example:
firmware v2
changes CAN velocity scaling from:
mm/s
to:
0.1 mm/s
Old ROS software still interprets it as mm/s.
Robot speed becomes 10× wrong.
Therefore interfaces need:
compatibility
version checks
migration plans
111. Deployment
A robot should boot into a known operational state.
Production deployment may require:
systemd services
Docker
ROS launch files
environment variables
device permissions
GPU runtime
network configuration
health checks
restart policies
The question is not merely:
Can I launch it manually?
The real question is:
Can the robot start reliably 1000 times without an engineer SSHing into it?
112. Automatic restart
Some components should restart after transient failure.
For example:
camera driver crashes
↓
supervisor restarts driver
↓
camera reconnects
But automatic restart must be designed carefully.
If a critical component crashes repeatedly:
restart
crash
restart
crash
the robot should probably enter a degraded or safe state rather than pretending everything is fine.
113. Observability
A mature robot should make its internal state visible.
You should be able to answer:
What mode is the robot in?
Where does it think it is?
What is it trying to do?
What command is it sending?
Why is it stopped?
Which sensor is unhealthy?
Which fault is active?
What was the last planner failure?
Why did safety trigger?
This is:
observability
Robotics systems without observability are painful to maintain.
114. Explainability for robot behavior
Imagine the robot is stationary.
An operator should ideally see:
Navigation state:
BLOCKED
Reason:
Front obstacle within safety zone.
Planner:
Valid path available.
Controller:
Velocity suppressed by safety layer.
instead of:
Robot stopped.
This dramatically improves debugging and operations.
115. Safety layers can override autonomy
Suppose AI says:
move forward
Planner says:
move forward
Controller says:
0.8 m/s
But safety laser detects a person.
Final command becomes:
0 m/s
That is correct.
A control architecture may look like:
Planner
↓
Controller
↓
Safety Filter
↓
Motor Interface
Therefore seeing:
controller output = 0.8
does not guarantee the motors receive 0.8.
Always know where command arbitration occurs.
116. Command arbitration
Robots often have multiple possible command sources:
autonomy
teleoperation
docking controller
safety controller
manual maintenance mode
Only one should control actuators according to defined priorities.
Example:
Safety stop highest priority
Teleoperation
Autonomy
Idle lowest priority
Without clear arbitration, two components may fight over /cmd_vel.
117. Multiple publishers
Suppose both:
Nav2
and:
teleop_keyboard
publish to:
/cmd_vel
The robot may receive alternating commands.
Symptoms:
jerking
random stopping
unpredictable movement
Check:
ros2 topic info /cmd_vel
and see how many publishers exist.
This simple check solves many confusing problems.
118. Saturation
Actuators have limits.
Suppose controller requests:
20 m/s
but robot maximum is:
1.5 m/s
The command is saturated.
Similarly:
motor current
torque
steering angle
acceleration
all have physical limits.
Controllers must account for saturation.
Otherwise integral windup or unstable behavior can occur.
119. Rate limiting
Even valid commands may change too quickly.
Example:
0 m/s
→
2 m/s
instantaneously.
The robot cannot physically produce infinite acceleration.
A rate limiter may constrain:
maximum acceleration
maximum jerk
to protect:
mechanics
payload
motors
passengers
traction
120. Jerk
Jerk is the rate of change of acceleration.
jerk=dadtjerk = \frac{d a}{dt}
High jerk produces sudden motion.
For mobile robots carrying fragile equipment or people, smooth jerk-limited trajectories may be important.
Physical comfort and mechanical stress become software constraints.
121. Thermal systems
Compute and motors generate heat.
A GPU may throttle when hot.
A motor may reduce performance.
A motor driver may shut down.
A battery may become unsafe.
Therefore robots monitor:
CPU temperature
GPU temperature
motor temperature
driver temperature
battery temperature
A performance degradation after 20 minutes may be thermal, not algorithmic.
122. Thermal throttling
Suppose AI inference normally takes:
40 ms
After long operation:
90 ms
Maybe the GPU has reached its thermal limit and reduced clock speed.
That increased latency can then affect:
perception
planning
control
A thermal issue becomes an autonomy issue.
This is system integration.
123. Resource contention
Multiple processes may compete for:
CPU
GPU
memory
network
disk
Example:
SLAM
object detection
video encoding
logging
all run on the same Jetson.
The detector consumes the GPU.
Video encoder consumes memory bandwidth.
SLAM misses deadlines.
Localization becomes unstable.
You cannot optimize each node independently.
You must optimize the whole resource budget.
124. CPU affinity and priorities
For time-sensitive systems, engineers sometimes control:
CPU affinity
thread priority
real-time scheduling
Example:
control loop
may be pinned to a dedicated CPU core.
This reduces interference from:
logging
UI
background processes
Not every robot needs this, but high-performance robots often do.
125. The robot as layered architecture
A useful mental model is:
Layer 8: Mission / AI
Layer 7: Behavior
Layer 6: Planning
Layer 5: Localization / Perception
Layer 4: Control
Layer 3: ROS / Middleware
Layer 2: Drivers / Firmware
Layer 1: Electronics / Power
Layer 0: Mechanics / Physics
Higher layers depend on lower layers.
If:
Layer 1 fails
Layer 8 cannot save you.
This architecture helps decide where to investigate.
126. Debug from evidence, not assumptions
Bad debugging:
Robot doesn't move.
"Nav2 must be broken."
Better:
Robot doesn't move.
Is planner output present?
Yes.
Is /cmd_vel nonzero?
Yes.
Does motor bridge receive it?
Yes.
Does MCU report command?
Yes.
Does driver report motor current?
No.
Now inspect driver enable/power.
Every step eliminates possibilities.
127. Observe both command and feedback
Whenever possible, inspect:
commanded state
and:
measured state
Example:
commanded wheel speed = 5 rad/s
measured wheel speed = 0 rad/s
This immediately tells you the problem lies after command generation.
If instead:
commanded wheel speed = 0
the problem lies upstream.
Command-vs-feedback comparison is one of the most powerful debugging techniques in robotics.
128. Golden signals for robots
For each major actuator, record something like:
desired command
command after safety
command sent to hardware
hardware acknowledgement
measured state
fault status
For example:
desired linear velocity: 0.8 m/s
safe linear velocity: 0.8 m/s
sent wheel RPM: 120 RPM
motor driver enabled: yes
measured wheel RPM: 0 RPM
motor current: 35 A
Immediately:
high current + no wheel speed
suggests a mechanical blockage.
Good telemetry shortens debugging dramatically.
129. A practical "robot doesn't move" checklist
Start with obvious physical conditions:
Is emergency stop released?
Is robot powered?
Is battery voltage healthy?
Are motor drivers enabled?
Is robot mechanically free?
Then high-level autonomy:
Was a goal accepted?
Is localization valid?
Is planner producing a path?
Is controller active?
Then commands:
Is /cmd_vel being published?
Is it nonzero?
Is another node overriding it?
Then hardware interface:
Does driver receive the command?
Is correct wheel command computed?
Then communication:
Are serial/CAN packets being sent?
Does MCU acknowledge them?
Then embedded control:
Is watchdog satisfied?
Is E-stop state clear?
Does firmware enable motors?
Then power stage:
Does driver receive supply voltage?
Does it produce motor current?
Are fault pins active?
Then mechanics:
Does motor shaft turn?
Does gearbox turn?
Does wheel turn?
Is anything jammed?
Then feedback:
Do encoders update?
Does odometry change?
Does TF change?
This is the full causal chain.
130. A practical sensor debugging checklist
Suppose a sensor is "not working."
Ask:
Is it physically powered?
Does Linux detect the device?
Does the interface exist?
Can the raw device communicate?
Does the driver start?
Does the ROS topic exist?
Is the expected frequency present?
Are data values sensible?
Are timestamps correct?
Is frame_id correct?
Does TF contain the sensor transform?
Does downstream software subscribe?
Does downstream QoS match?
Again, go through layers.
131. A practical localization debugging checklist
If localization is poor:
Is map correct?
Is initial pose reasonable?
Are LiDAR scans correct?
Is odometry correct?
Is IMU correct?
Are frames correct?
Are timestamps synchronized?
Is map→odom being published?
Is odom→base_link being published?
Is wheel calibration correct?
Is there excessive wheel slip?
Does environment contain enough features?
Do not immediately retune the localization algorithm before validating its inputs.
132. A practical perception debugging checklist
If AI detection is poor:
Is camera image correct?
Is exposure appropriate?
Is image orientation correct?
Is preprocessing correct?
Is model correct?
Is model loaded correctly?
Is inference using expected precision?
Are confidence thresholds sensible?
Is postprocessing correct?
Is coordinate conversion correct?
Is latency acceptable?
Often the "AI problem" lies before or after the neural network.
133. A practical networking checklist
If distributed ROS nodes cannot communicate:
Can machines ping each other?
Are they on the same subnet?
Are firewalls blocking traffic?
Is ROS_DOMAIN_ID the same?
Does DDS discovery work?
Is multicast allowed?
Is Wi-Fi client isolation enabled?
Are QoS settings compatible?
Are interfaces bound correctly?
Is clock synchronization reasonable?
Networking skills are essential for multi-computer robots.
134. A practical power checklist
If the robot randomly resets:
Check battery voltage idle.
Check voltage under motor load.
Check DC/DC output.
Check peak current.
Check connectors.
Check ground.
Check fuse / breaker.
Check BMS events.
Check thermal state.
Check kernel logs for brownout-like USB/device disconnects.
A power problem can manifest as hundreds of unrelated-looking software errors.
135. Build test points into the architecture
Good systems expose intermediate signals.
Bad:
AI
↓
mysterious system
↓
motor
Better:
AI target
↓
planned goal
↓
trajectory
↓
controller command
↓
safety command
↓
wheel command
↓
CAN message
↓
motor state
You should be able to inspect every important boundary.
Observability should be designed, not added after failure.
136. Integration tests
Unit tests verify individual components.
Integration tests verify components together.
Examples:
Does controller convert path into velocity?
Does ROS driver correctly encode CAN commands?
Does firmware correctly decode command packets?
Does E-stop override autonomous command?
Does encoder feedback reach odometry?
End-to-end tests might verify:
send navigation goal
↓
robot reaches location
All levels are important.
137. Fault injection
A powerful testing technique is deliberately causing failures.
Examples:
disconnect LiDAR
drop CAN packets
kill localization node
simulate low battery
delay camera frames
disable one encoder
drop network connection
Then observe:
Does the robot fail safely and recover correctly?
This is much better than discovering failure behavior for the first time in production.
138. Safety invariants
A safety invariant is something that must always remain true.
Examples:
Robot must not move while E-stop is active.
Robot must not exceed 1 m/s in pedestrian zone.
Motor current must not exceed safe threshold for too long.
Autonomy cannot control motors in maintenance mode.
Architecture should enforce these invariants at the correct layer.
139. Don't put every safety rule in AI
Suppose an AI planner is instructed:
Never drive into people.
That is not enough.
Safety-critical constraints should often exist in lower, deterministic layers too.
Example:
AI planner
↓
motion controller
↓
independent safety scanner
↓
motor command
If the AI fails, the safety layer still stops the robot.
140. Human-machine interface
Operators need to understand and control the robot.
Useful interfaces include:
start
stop
pause
manual mode
autonomous mode
fault reset
battery status
current mission
robot location
fault reason
Poor UI can create operational errors even if engineering is correct.
System integration includes the human using the system.
141. Modes
Real robots often have multiple modes:
OFF
BOOTING
MANUAL
AUTONOMOUS
MAINTENANCE
FAULT
ESTOP
CHARGING
Allowed commands depend on mode.
Example:
AUTONOMOUS
→ navigation commands accepted
ESTOP
→ all movement rejected
MAINTENANCE
→ low-speed manual control only
Mode management prevents unsafe state combinations.
142. Interlocks
An interlock prevents unsafe actions unless conditions are satisfied.
Example:
motor_enable =
estop_released
AND
safety_system_ok
AND
battery_ok
AND
control_watchdog_ok
If any condition becomes false:
motor_enable = false
Interlocks are foundational in industrial robotics.
143. The hidden state problem
Suppose a motor command is ignored.
Maybe the motor controller is in:
FAULT
state.
You repeatedly publish commands.
Nothing happens.
Unless you inspect controller state, the problem appears mysterious.
Robotic subsystems often have internal state machines.
Always ask:
What state is this component currently in?
not merely:
What command did I send?
144. Latched faults
Some faults remain active even after the original condition disappears.
Example:
overcurrent occurs
↓
driver enters fault
↓
current becomes normal
↓
driver remains disabled
It may require:
explicit reset
power cycle
This is a:
latched fault
Knowing this prevents endless command debugging.
145. Initialization
Hardware often requires initialization sequences.
Example:
power motor driver
wait 500 ms
configure mode
clear faults
enable drive
set control mode
begin commands
Sending commands before initialization finishes may do nothing.
Startup behavior is part of the interface contract.
146. Race conditions
Distributed systems can start in unpredictable orders.
Suppose:
controller starts
before:
TF exists
and exits.
On another boot TF starts first, so everything works.
Now the robot behaves:
sometimes works
sometimes fails
This is often a startup race condition.
Lifecycle coordination and readiness checks reduce these problems.
147. Heartbeats
Subsystems may send periodic:
heartbeat
messages.
Example:
motor controller heartbeat every 100 ms
If heartbeat disappears:
controller unavailable
Higher-level software can stop motion or raise a fault.
Heartbeats are simple but powerful health signals.
148. Sequence numbers
Messages may carry:
sequence number
Example:
1001
1002
1003
1005
Missing:
1004
indicates packet loss.
Sequence numbers help detect:
lost messages
duplicates
reordering
especially in low-level communication protocols.
149. Checksums and CRC
Communication packets can become corrupted.
A checksum or CRC allows the receiver to detect corruption.
Packet:
HEADER
DATA
CRC
Receiver computes CRC again.
If values differ:
discard packet
Without error detection, one corrupted byte could potentially become a dangerous motor command.
150. Isolation of failure domains
A good architecture limits how far failures spread.
Example:
camera crashes
should not necessarily crash:
motor controller
Similarly:
AI model OOM
should not disable emergency stop.
This is fault isolation.
Subsystem boundaries should protect critical functionality.
151. Safety-critical vs noncritical compute
It is useful to distinguish:
safety-critical
from:
mission-critical
and:
noncritical
For example:
E-stop input:
safety-critical
motor watchdog:
safety-critical
navigation:
mission-critical
object labeling UI:
noncritical
These should not all have identical architecture or failure behavior.
152. AI failures should degrade gracefully
Suppose object detector crashes.
A weak design:
entire robot process crashes
A stronger design:
perception marked unhealthy
↓
robot enters restricted motion or stops
↓
operator notified
AI should be treated as a fallible subsystem.
153. The system boundary matters
When someone says:
The robot works.
Ask:
Works under what conditions?
Which floor?
Which lighting?
Which network?
Which payload?
Which battery level?
Which temperature?
Which speed?
Which map?
System performance is always defined relative to an operating envelope.
154. Operating envelope
An autonomous robot may be designed for:
indoor use
0–40°C
flat flooring
maximum slope 5°
maximum payload 50 kg
dry conditions
speed ≤ 1.5 m/s
Outside this envelope, correct operation may not be guaranteed.
Defining this explicitly is professional engineering.
155. Performance budgets
System requirements should be divided into budgets.
Example total response latency:
maximum = 150 ms
Allocate:
sensor acquisition 20 ms
perception 50 ms
planning 30 ms
control 10 ms
communication 10 ms
margin 30 ms
Now each team knows its limit.
The same idea applies to:
power
compute
network bandwidth
memory
mass
thermal load
156. System integration is about budgets
Every robot has finite:
watts
kilograms
CPU cores
GPU memory
network bandwidth
battery energy
physical volume
money
latency
Improving one subsystem may consume budget needed elsewhere.
Example:
bigger AI model
↓
more GPU power
↓
more heat
↓
bigger cooling
↓
more power consumption
↓
shorter battery life
Engineering is always trade-offs across the whole system.
157. Compute-power coupling
Suppose upgrading perception from:
10 W accelerator
to:
60 W accelerator
improves detection.
But now:
battery life decreases
cooling requirement increases
DC/DC converter changes
robot mass increases
A software decision has mechanical and electrical consequences.
That is why system engineers think across disciplines.
158. Mechanical-compute coupling
Suppose you replace a sensor with a heavier LiDAR mounted high on the robot.
Now:
center of gravity rises
which can affect:
stability
maximum acceleration
turning behavior
A perception upgrade may require control and mechanical changes.
159. Sensor-placement coupling
A mathematically excellent sensor can perform badly if mounted poorly.
Examples:
LiDAR blocked by robot body
camera vibrating
IMU far from assumed frame
GPS antenna near interference source
depth camera facing reflective surfaces
Sensor integration includes mechanical placement.
160. Cable management is engineering
Loose cables can cause:
intermittent encoder loss
USB disconnect
mechanical interference
EMI
connector fatigue
Professional robots use:
strain relief
locking connectors
shielding
proper routing
service loops
A robot that works on a bench but fails after two weeks may have a connector problem.
161. Environmental robustness
Real environments contain:
dust
vibration
temperature changes
lighting variation
water
metal
electromagnetic noise
network congestion
people
A laboratory prototype often assumes ideal conditions.
A production robot must survive nonideal conditions.
System integration is where this gap becomes visible.
162. Reliability
Reliability asks:
How consistently does the robot perform correctly over time?
A robot that succeeds:
9 times out of 10
may be impressive as a research demo.
A production robot performing:
1000 missions/day
would fail about:
100 times/day
at 90% reliability.
Production robotics requires extremely different thinking from demonstrations.
163. Mean time between failures
Reliability engineering uses concepts such as:
MTBF
Mean Time Between Failures.
If a subsystem fails every 20 hours on average, that may be unacceptable for autonomous deployment.
Failures include:
software crashes
mechanical wear
sensor disconnects
network faults
thermal shutdowns
System integration eventually becomes reliability engineering.
164. Redundancy
Critical systems may include redundant sensing or computation.
Examples:
dual encoders
multiple cameras
LiDAR + camera
dual communication paths
redundant safety channels
If one source fails, another can maintain safe operation or provide cross-checking.
Redundancy is expensive but valuable for high-assurance systems.
165. Diverse redundancy
Two identical sensors may fail from the same cause.
Example:
two cameras
can both fail in darkness.
Using:
camera + LiDAR
provides different failure characteristics.
This is sometimes called:
diverse redundancy
because failures are less correlated.
166. Debugging intermittent failures
Intermittent bugs are the hardest.
Examples:
once every 3 hours
only when turning sharply
only after heating up
only in one room
only when battery below 30%
Approach:
log continuously
correlate multiple signals
identify conditions
reproduce
narrow the causal chain
Never rely only on memory:
It looked like the motor stopped first.
Measure it.
167. Correlation does not prove cause
Suppose:
camera freezes
at the same time:
robot stops
You might conclude:
Camera freeze caused the stop.
But maybe both were caused by:
USB power rail failure
or:
computer overload
The systems engineer searches for common upstream causes.
168. Common-cause failures
One physical event may break many subsystems.
Example:
12V rail drops
and causes:
camera disconnect
LiDAR reset
network switch reboot
Three software failures appear simultaneously.
The true failure is one shared infrastructure component.
Think about dependencies.
169. Dependency graphs
A robot can be represented as a dependency graph.
Example:
Battery
├── Computer
│ ├── ROS
│ │ ├── Localization
│ │ └── Navigation
│ └── AI
│
└── Motor Driver
└── Motors
If battery fails:
everything downstream fails
Dependency graphs help reason about common failure modes.
170. FMEA thinking
A common engineering approach is:
Failure Modes and Effects Analysis
or:
FMEA
For each component ask:
How can it fail?
What happens if it fails?
How do we detect it?
How dangerous is it?
How do we recover?
Example:
Component:
wheel encoder
Failure:
signal stuck at zero
Effect:
bad odometry
Detection:
commanded wheel motion + zero encoder velocity
Mitigation:
stop navigation and raise fault
This mindset produces robust robots.
171. Fault trees
Another useful tool is a:
fault tree
Start with failure:
Robot does not move
Possible branches:
Robot does not move
│
├── No command generated
│ ├── planner failed
│ └── controller inactive
│
├── Command not delivered
│ ├── DDS issue
│ ├── driver crashed
│ └── CAN failure
│
├── Actuation unavailable
│ ├── E-stop
│ ├── driver fault
│ └── power loss
│
└── Mechanical failure
├── jam
├── gearbox failure
└── wheel disconnected
This organizes debugging systematically.
172. Build observability around fault trees
If the top-level question is:
Why isn't the robot moving?
telemetry should let you inspect every major branch quickly:
planner output?
controller command?
safety output?
motor enable?
CAN health?
driver fault?
motor current?
wheel speed?
Good observability is designed around expected failure modes.
173. Don't overfit debugging to the previous failure
Suppose last week's problem was:
TF
Today the robot does not move.
Do not immediately spend two hours on TF.
Symptoms can have many causes.
Always re-establish evidence.
Experienced engineers use prior knowledge to prioritize possibilities, not to replace diagnosis.
174. Layer isolation
When debugging, temporarily bypass layers.
For example:
Normal:
Nav2
↓
Controller
↓
/cmd_vel
↓
Motor
Test:
manual /cmd_vel
↓
Motor
If manual control works:
hardware chain is probably healthy
Now focus above it.
Similarly, directly command the MCU if necessary.
Layer isolation is incredibly useful.
175. Progressive integration
Do not build the entire robot and then test everything together.
Better:
1. Power system
2. MCU
3. Motor driver
4. Manual motor control
5. Encoders
6. Odometry
7. LiDAR
8. TF
9. Localization
10. Navigation
11. AI
12. Advanced behavior
At each stage, verify stable operation.
This reduces debugging complexity dramatically.
176. Bring-up
The first process of making new robot hardware functional is often called:
bring-up
Typical bring-up sequence:
check power rails
flash firmware
verify communication
spin motors safely
read encoders
validate sensor devices
establish TF
publish robot state
test teleoperation
test odometry
test localization
test autonomy
Bring-up is a core robotics integration skill.
177. Bench testing before floor testing
Before letting the robot drive:
raise wheels off ground
and test:
motor direction
encoder direction
E-stop
watchdog
velocity scaling
If:
positive command
causes:
left wheel forward
right wheel backward
you want to discover that on the bench, not at full speed.
178. Direction conventions
For differential drive:
positive left wheel velocity
positive right wheel velocity
should correspond consistently with the robot's coordinate convention.
If one encoder is reversed, odometry becomes wrong.
A classic bring-up test is:
push robot forward manually
and verify:
both encoder counts increase consistently
179. Unit tests for hardware conversion
Suppose converting linear/angular velocity into wheel angular velocity:
vL=v−ωL/2rv_L = \frac{v - \omega L/2}{r}vR=v+ωL/2rv_R = \frac{v + \omega L/2}{r}
where:
v = linear velocity
ω = angular velocity
L = wheel separation
r = wheel radius
This conversion can be unit-tested without hardware.
For:
v = 1 m/s
ω = 0
expect:
left = right
For:
v = 0
ω > 0
expect opposite wheel directions.
Test math before field testing.
180. Differential-drive kinematics
For wheel linear velocities:
vLv_L
and:
vRv_R
robot linear velocity is approximately:
v=vR+vL2v = \frac{v_R + v_L}{2}
and angular velocity:
ω=vR−vLL\omega = \frac{v_R - v_L}{L}
This connects:
wheel motion
to:
robot motion
Incorrect:
wheel radius
wheel separation
direction signs
directly corrupt odometry and control.
181. Manipulator integration
For robot arms, the causal chain is similar:
Task Planner
↓
Motion Planner
↓
Joint Trajectory
↓
Joint Controller
↓
Fieldbus
↓
Servo Drive
↓
Motor
↓
Gearbox
↓
Joint
↓
Encoder
↓
Controller feedback
Failures can occur at every level.
System integration principles generalize beyond mobile robots.
182. Joint states
Robot arms commonly publish:
position
velocity
effort
for each joint.
If joint ordering is wrong:
joint_1 value interpreted as joint_2
kinematics become nonsense.
Again:
Interface correctness matters as much as algorithm correctness.
183. URDF
URDF describes robot structure.
It defines things such as:
links
joints
geometry
visuals
collision shapes
inertia
transforms
A wrong URDF can affect:
TF
visualization
collision checking
kinematics
planning
simulation
So the robot model is part of integration.
184. Collision geometry
The visual model and collision model may differ.
Visual mesh:
detailed
pretty
large
Collision model:
simpler
computationally efficient
If collision geometry is wrong, planners may:
think robot can fit where it cannot
or:
refuse valid paths
Planning problems can originate in robot geometry.
185. Inertia
Dynamics depend on:
mass
center of mass
inertia tensor
In simulation, incorrect inertia can cause:
unstable motion
unrealistic acceleration
oscillation
For torque-controlled robots, these parameters matter even more.
186. Simulation must match hardware interfaces
An effective simulator should preserve the same high-level interfaces.
Real:
/cmd_vel
↓
hardware
↓
/odom
Simulation:
/cmd_vel
↓
simulated robot
↓
/odom
Then higher-level software can operate unchanged.
This reduces the gap between simulation and real deployment.
187. Abstraction layers
Good architecture separates:
what
from:
how
For example, navigation says:
desired velocity = 0.5 m/s
It should not need to know:
PWM duty cycle
CAN packet layout
motor phase current
The hardware abstraction layer handles that.
This enables replacing hardware without rewriting the entire autonomy stack.
188. Hardware abstraction
A good base interface might expose:
set_wheel_velocity()
read_wheel_velocity()
read_faults()
enable()
disable()
Underneath:
Robot A → CAN motor driver
Robot B → UART motor driver
Higher-level control does not care.
Abstraction reduces coupling.
189. But abstractions leak
No abstraction is perfect.
Suppose one motor controller supports:
velocity command at 100 Hz
and another:
torque command at 1 kHz
These differences affect higher-level design.
Senior engineers understand both:
the abstraction
and:
what lies beneath it
This is especially important in robotics.
190. Real-time vs best-effort data
Not all robot information has equal timing importance.
High priority:
E-stop
motor control
joint state
IMU
Lower priority:
UI thumbnails
debug logs
map visualization
A saturated network should not allow:
HD debugging video
to delay:
critical motor feedback
System design should respect communication priorities.
191. Safety network separation
Industrial systems sometimes use separate:
safety network
or dedicated hardwired safety circuits independent of normal autonomy traffic.
Why?
Because the system controlling safety should not fail merely because:
ROS traffic overloaded Ethernet
Safety independence is a key architectural principle.
192. Resource monitoring
A robot should monitor:
CPU
RAM
GPU RAM
disk
network
temperature
power
Example:
disk reaches 100%
↓
ROS bag recording fails
↓
logger crashes
↓
another process misbehaves
Resource exhaustion should be detected before catastrophic failure.
193. Disk exhaustion
Logging can consume enormous storage.
Suppose cameras generate:
20 MB/s
That becomes:
20×3600=72GB/hour20 \times 3600 = 72GB/hour
A 1 TB disk can fill quickly.
Production logging often uses:
rotation
retention
compression
event-triggered recording
194. Event buffers
Instead of permanently storing every high-bandwidth stream, robots may keep a rolling buffer.
Example:
last 30 seconds continuously retained
When a fault occurs:
save previous 30 seconds
+
next 30 seconds
This gives excellent debugging evidence without storing everything forever.
195. Clock ordering in logs
Logs from multiple computers are only useful together when their clocks are synchronized.
Otherwise:
computer A:
fault at 12:00:01
computer B:
sensor disconnect at 11:59:54
You cannot confidently determine event ordering.
Distributed robotics depends heavily on consistent time.
196. Remote diagnostics
Production robots may be far from engineers.
Remote diagnostics should expose enough information to answer:
Is robot online?
Battery?
Temperatures?
Current fault?
Software version?
Sensor health?
Last mission?
Network health?
But remote access must also be secure.
Robotics systems are networked physical machines.
Cybersecurity matters.
197. Security
A compromised robot can create physical consequences.
Important concerns include:
authentication
encrypted communication
secure updates
restricted SSH
network segmentation
credential management
signed firmware
A motor command interface should not be openly accessible to arbitrary devices on the network.
Security is part of system safety.
198. OTA updates
Robots deployed in fleets often require:
Over-the-Air updates
OTA systems should handle:
download
verification
installation
reboot
health check
rollback
If update fails, robot should recover to a known-good version.
A failed software update should not permanently brick a field robot.
199. Fleet consistency
Suppose 100 robots exist.
If they each have random software and calibration states, debugging becomes impossible.
Fleet systems should know:
which version is on which robot
which hardware revision
which configuration
which calibration
Production robotics eventually becomes fleet engineering.
200. The senior integration mindset
A senior robotics engineer does not think:
I am a ROS engineer.
or:
I am an AI engineer.
They think:
I am responsible for the behavior of the machine.
If navigation fails because:
battery sagged
that is still relevant.
If perception fails because:
camera mount vibrates
that is relevant.
If controller oscillates because:
gearbox backlash
that is relevant.
If ROS communication fails because:
multicast is blocked
that is relevant.
System boundaries should not become thinking boundaries.
201. The causal-chain method
Whenever something goes wrong:
Step 1 — Define the symptom precisely
Bad:
Robot broken.
Better:
Robot receives navigation goal,
planner creates path,
but wheel velocity remains zero.
Step 2 — Identify the causal chain
goal
→ planner
→ controller
→ command
→ driver
→ MCU
→ motor
Step 3 — Find the last known-good point
/cmd_vel is correct
Therefore everything above it probably works.
Step 4 — Find the first bad point
CAN command is zero
Now investigate the conversion between:
/cmd_vel
and:
CAN
Step 5 — Fix the root cause
Not the symptom.
202. Never change five things at once
Suppose navigation fails.
You:
retune controller
change planner
modify TF
change motor firmware
change wheel radius
Then robot works.
What fixed it?
You do not know.
Engineering debugging should be:
hypothesis
↓
controlled change
↓
measurement
↓
conclusion
Change one meaningful variable at a time when possible.
203. Instrument before optimizing
If you cannot see:
motor current
wheel command
wheel speed
add telemetry first.
If you cannot inspect:
planner time
AI latency
network latency
add measurement first.
Blind optimization creates folklore.
Instrumentation creates engineering knowledge.
204. Understand nominal behavior first
You cannot diagnose abnormal behavior if you do not know normal behavior.
Record a healthy run:
normal battery voltage
normal current
normal CPU
normal latency
normal encoder rates
normal TF
normal localization covariance
Then compare failure runs against the baseline.
This dramatically improves debugging.
205. Golden run
A useful concept is a:
golden run
A known-good recording of:
logs
ROS bag
metrics
configuration
from a successful mission.
When something breaks after an update:
compare failing run
vs
golden run
Differences often reveal the problem.
206. The real robotics debugging question
When the robot does something wrong, do not ask only:
Which component is broken?
Ask:
Which assumption in the causal chain became false?
Examples:
planner assumes localization is valid
controller assumes odometry is accurate
firmware assumes commands arrive periodically
driver assumes power voltage is stable
software assumes encoder direction is correct
Failures often expose broken assumptions.
207. Integration contracts
Every subsystem should define a contract.
Example motor controller contract:
Input:
wheel velocity in rad/s
Range:
-20 to +20 rad/s
Update rate:
minimum 20 Hz
Timeout:
500 ms
On timeout:
stop motors
Feedback:
wheel velocity at 100 Hz
Fault behavior:
latched until reset
Clear contracts dramatically reduce integration ambiguity.
208. Error semantics
Suppose a function returns:
false
What does that mean?
temporary failure?
invalid command?
hardware disconnected?
safety stop?
timeout?
Interfaces should expose meaningful error states.
Example:
OK
TIMEOUT
ESTOP_ACTIVE
OVERCURRENT
ENCODER_FAULT
COMMUNICATION_LOST
Good error semantics improve automation and debugging.
209. Handling stale data
A topic may still contain the most recent message even though the sensor has stopped.
Example:
last localization pose:
3 seconds old
If controller treats it as current, behavior may be unsafe.
Always distinguish:
value
from:
fresh value
Freshness limits are essential in robotics.
210. Timestamp validation
Suppose current time:
100.0 s
last sensor message:
98.1 s
Age:
1.9 s
If maximum acceptable age is:
0.2 s
then data must be considered invalid.
This should be explicit, not implicit.
211. Confidence and health are separate
A perception system may return:
object confidence = 0.95
while the camera stream itself is:
2 seconds stale
The detection is confidently wrong for the current moment.
So systems often need both:
semantic confidence
and:
system health/freshness
These are different concepts.
212. Robot systems operate in loops
A robot is not:
sense
then plan
then act
finished
It continuously repeats:
sense
↓
estimate
↓
plan
↓
control
↓
act
↓
sense again
This is a closed-loop autonomous system.
Everything is continuously correcting based on new information.
213. End-to-end loop frequency
If perception updates at:
30 Hz
localization at:
50 Hz
planner at:
5 Hz
controller at:
20 Hz
the effective reaction loop depends on how these components interact.
The slowest or highest-latency stage may dominate certain behaviors.
System-level timing matters more than isolated node frequency.
214. Queue buildup
Suppose camera produces:
30 FPS
but AI processes:
10 FPS
If every frame is queued:
frame 1
frame 2
frame 3
...
latency grows continuously.
After several seconds, AI may be processing old images.
In robotics, often:
latest frame
is more valuable than:
every frame
Queue strategy matters.
215. Backpressure
When downstream processing cannot keep up with upstream data, the system needs a policy.
Options include:
drop old messages
drop new messages
reduce sensor rate
slow producer
buffer temporarily
This is:
backpressure
Without it, memory or latency can explode.
216. Freshness vs completeness
For navigation:
newest LiDAR scan
is often more useful than receiving every scan.
For financial transactions, losing a message is unacceptable.
Different systems prioritize differently.
Robotics frequently values:
freshness
over:
perfect delivery
for high-rate state streams.
This influences DDS QoS and architecture.
217. The control loop cannot outrun perception truth
Suppose controller updates:
100 Hz
but localization updates:
5 Hz
The controller executes 20 cycles using essentially the same pose estimate.
This may still work through interpolation/odometry, but illustrates an important principle:
Fast control does not compensate for slow or stale state information.
The complete loop matters.
218. Uncertainty should propagate
Suppose localization confidence becomes poor.
Planner should not behave as though pose remains perfectly known.
A mature system may:
reduce speed
increase safety margins
attempt relocalization
stop
This turns estimator uncertainty into behavior.
System integration is partly about propagating meaning across layers.
219. Safety margins depend on uncertainty
Suppose normal localization uncertainty is:
±2 cm
but temporary uncertainty grows to:
±30 cm
Driving within:
10 cm
of obstacles is no longer safe.
Therefore safety margins may need to depend on state-estimation quality.
This is system-level autonomous reasoning.
220. Real robots need graceful stopping
Stopping is not always:
velocity = 0 immediately
At high speeds this may be mechanically impossible.
There may be:
normal stop
controlled emergency stop
power-cut emergency stop
Different stop categories balance:
stability
hardware safety
human safety
Safety architecture should define these explicitly.
221. Braking distance
Suppose robot travels:
2 m/s
and can decelerate at:
1 m/s²
Ignoring reaction delay:
d=v22ad = \frac{v^2}{2a}d=42=2md = \frac{4}{2} = 2m
It requires roughly:
2 meters
to stop.
Add sensor and processing delay and the required distance increases.
Software safety must respect physics.
222. Reaction distance
If total detection/control latency is:
200 ms
at:
2 m/s
the robot travels:
2×0.2=0.4m2 \times 0.2 = 0.4m
before deceleration even begins.
Total stopping distance becomes:
reaction distance
+
braking distance
This is why end-to-end latency is a safety property.
223. Safety speed depends on sensing
A robot with sensing range:
2 m
cannot safely drive arbitrarily fast.
Safe speed depends on:
sensor range
latency
deceleration
uncertainty
environment
Performance is constrained by the entire system.
224. Multi-rate systems
Robots commonly have many loop rates:
motor current control 10 kHz
velocity control 1 kHz
odometry 100 Hz
local controller 20 Hz
global planner 1 Hz
mission planner 0.1 Hz
These layers interact.
Understanding multi-rate systems helps avoid:
aliasing
stale commands
unnecessary computation
timing mismatch
225. Command hierarchy
A clean hierarchy might be:
Mission:
"Go to Room 302"
Behavior:
"Navigate"
Planner:
"Follow this path"
Controller:
"v = 0.6, ω = 0.2"
Wheel kinematics:
Left = 8 rad/s
Right = 12 rad/s
Motor control:
PWM / current commands
Each layer transforms high-level intent into increasingly physical commands.
Debugging can follow the same hierarchy downward.
226. Feedback hierarchy
Feedback flows upward:
motor current
↓
wheel speed
↓
odometry
↓
robot pose
↓
navigation progress
↓
mission status
The robot is really two chains:
command chain downward
feedback chain upward
Correct autonomy depends on both.
227. The complete robot loop
A more complete picture is:
HIGH-LEVEL INTENT
│
▼
Mission
│
▼
Behavior
│
▼
Planning
│
▼
Control
│
▼
Hardware Interface
│
▼
MCU
│
▼
Motor Driver
│
▼
Motor
│
▼
Mechanics
│
▼
World
│
▼
Sensors
│
▼
Drivers
│
▼
State Estimation
│
└──────────────► feedback
Around everything:
power
safety
time
network
telemetry
configuration
That is the robot.
228. Why senior robotics engineers look "broad"
A senior robotics engineer may discuss:
TF
then suddenly ask:
What's the motor rail voltage?
Then:
Is CAN terminated correctly?
Then:
What frame is that detection in?
Then:
What's the controller frequency?
This breadth is not lack of specialization.
It is necessary because robot behavior emerges from interactions between domains.
229. You don't need to be the world's best expert in every layer
System integration does not mean mastering:
power electronics
control theory
deep learning
mechanical design
Linux kernel engineering
networking
at equal depth.
It means being able to understand:
what each layer does
what assumptions it makes
what its interfaces are
how it fails
how to inspect it
when to call a specialist
This is the appropriate kind of breadth.
230. The T-shaped robotics engineer
A useful career model is:
Broad systems knowledge
────────────────────────────────────────
Linux ROS networking sensors firmware
control AI power mechanics safety
│
│
│
Deep specialization
│
│
You may specialize deeply in:
perception
control
planning
embedded
AI
while retaining enough systems knowledge to integrate the complete robot.
This combination is extremely valuable.
231. What "MASTER" means here
Mastering robotics system integration does not mean memorizing every CAN register or ROS command.
It means you can walk into a system where:
The robot doesn't move.
and methodically determine whether the problem is:
mission logic
planner
controller
TF
localization
ROS lifecycle
DDS
networking
driver
Linux
serial
CAN
firmware
watchdog
motor driver
power
encoder
mechanical drivetrain
safety interlock
without randomly guessing.
You know how to reduce uncertainty until the failing link becomes visible.
232. The most important debugging habit
Before touching anything, write:
Expected:
A should cause B.
Observed:
A happened, but B did not.
Example:
Expected:
nonzero /cmd_vel
should produce
nonzero CAN wheel command.
Observed:
nonzero /cmd_vel exists,
CAN wheel command remains zero.
Now the problem is constrained to:
cmd_vel → hardware bridge → CAN encoding
This one habit transforms debugging quality.
233. The second most important debugging habit
Ask:
What evidence proves this?
Someone says:
The controller is sending commands.
Evidence?
topic echo?
log?
packet capture?
oscilloscope?
Someone says:
The motor driver is fine.
Evidence?
fault register?
current measurement?
driver telemetry?
Systems debugging must be evidence-based.
234. The third most important debugging habit
Distinguish:
commanded
from:
received
from:
executed
Example:
ROS commanded 1 m/s
MCU received 1 m/s
driver accepted command
wheel actually moved 0.8 m/s
These are four different facts.
Never collapse them into:
The motor was commanded.
235. The fourth most important debugging habit
Check what changed.
If system worked yesterday:
software update?
firmware update?
configuration?
battery?
hardware?
cable?
network?
map?
sensor calibration?
environment?
Change analysis often provides the fastest hypothesis.
236. The fifth most important debugging habit
Know when to go down one layer.
If ROS topic is correct:
inspect driver
If driver output is correct:
inspect bus
If bus is correct:
inspect firmware
If firmware is correct:
inspect electronics
If electronics are correct:
inspect mechanics
Do not stay inside your favorite abstraction when evidence points elsewhere.
237. The core integration map to remember
When you come back to this blog months later, remember this:
INTENT
↓
BEHAVIOR
↓
PLANNING
↓
CONTROL
↓
ROS
↓
DRIVER
↓
COMMUNICATION BUS
↓
MCU / FIRMWARE
↓
MOTOR DRIVER
↓
MOTOR
↓
MECHANICS
↓
WORLD
↓
SENSORS
↓
DRIVERS
↓
LOCALIZATION / PERCEPTION
↓
FEEDBACK TO CONTROL
And surrounding all of it:
LINUX
NETWORKING
TIME
POWER
SAFETY
TELEMETRY
CONFIGURATION
238. Final mental model
When a robot fails, imagine yourself walking through the machine.
Start at the high-level request:
"Go there."
Then follow it physically downward:
Did the behavior system accept it?
Did the planner create a path?
Did the controller create a motion command?
Did ROS deliver it?
Did the hardware driver convert it?
Did CAN/serial transmit it?
Did firmware receive it?
Did the watchdog allow it?
Did the motor driver enable?
Did current reach the motor?
Did the motor produce torque?
Did the drivetrain transmit that torque?
Did the wheel move?
Then follow feedback upward:
Did the encoder detect movement?
Did odometry update?
Did localization update?
Did the controller observe progress?
Did navigation report movement?
If you can traverse this complete causal chain confidently, you are no longer merely writing robotics code.
You are engineering robotic systems.
And that is where the real robotics software engineer lives.