3D Perception : Things to Master for Modern Robotics

UpdatedSeptember 2, 2026

I am an AI Research Engineer with a combined motivation of building AI models as well as developing AI integrated apps. I am currently exploring Robot Learning and groundbreaking DL, RL and Robotics papers and trying to understand how this is shaping the future.

From pixels and laser returns to a robot that understands physical space

A camera sees an image.

A LiDAR sees laser returns.

A depth camera sees distances.

But a robot ultimately needs something more fundamental:

What exists around me, where is it in 3D space, which space is free, and how certain am I about all of this?

That is 3D perception.

If ordinary computer vision asks:

“What is in this image?”

3D perception asks:

“What physically exists around me?”

And robotics adds one more question:

“Given that understanding, where can I safely move?”

That difference is enormous.

A self-driving vehicle does not merely need to know:

pedestrian detected

It needs approximately:

pedestrian
position: 12.3 m ahead, 1.7 m left
height: ~1.75 m
velocity: approaching road
uncertainty: moderate

An autonomous robot seeing a chair does not only need:

chair

It may need:

chair occupies this volume
floor is underneath it
this region is traversable
this region is not
chair is 2.4 m away
there is free space behind this point until ...

That is why 3D perception sits directly underneath:

SLAM
autonomous navigation
manipulation
grasping
drones
self-driving
warehouse robotics
AR/VR
digital twins
scene reconstruction
embodied AI

1. The master mental model

Almost everything in this chapter can be understood through four stages:

SENSOR
   ↓
3D MEASUREMENTS
   ↓
3D REPRESENTATION
   ↓
ROBOT DECISION

For example:

LiDAR
 ↓
laser ranges
 ↓
point cloud
 ↓
voxel map / occupancy map
 ↓
collision avoidance

Or:

RGB-D camera
 ↓
RGB + depth
 ↓
colored point cloud
 ↓
TSDF / mesh
 ↓
robot understands room geometry

Or modern learned perception:

multiple cameras
 ↓
images
 ↓
learned 3D features
 ↓
BEV / occupancy / object representation
 ↓
planning

The sensors change.

The representation changes.

The underlying question stays the same:

How should the physical world be represented computationally?

That is the central problem of 3D perception.

2. The most important distinction: measurement vs representation

Never confuse these.

A sensor produces measurements.

Your software converts those measurements into a representation of the world.

For example, LiDAR may give:

range = 8.31 m
angle = 42.1°

After conversion:

(x, y, z) = (5.64, 6.09, 0.72)

Now you have a point.

Collect thousands of them:

Point Cloud

But a point cloud is still not necessarily a map.

You might transform it into:

voxel grid
occupancy map
mesh
TSDF
ESDF
semantic map
learned feature map

depending on the job.

Senior-engineer rule

Do not ask “What is the best 3D representation?”

Ask:

Best representation for what downstream operation?

Because the answer changes completely.

For rendering?

NeRF or Gaussian representation may be attractive.

For collision checking?

Occupancy or ESDF may be better.

For surface reconstruction?

Mesh or TSDF.

For neural perception?

Point, voxel, BEV, implicit, or hybrid representations.

3. Point clouds

The most basic 3D representation is beautifully simple.

A point cloud is just a collection of 3D points:

Pi=(xi,yi,zi)P_i = (x_i,y_i,z_i)

For example:

[
  (1.2, 0.4, 2.1),
  (1.3, 0.4, 2.0),
  (1.4, 0.5, 1.9),
  ...
]

Each point says:

“The sensor observed something here.”

A point may contain extra information:

(x, y, z)
(x, y, z, intensity)
(x, y, z, R, G, B)
(x, y, z, semantic_class)
(x, y, z, timestamp)

LiDAR commonly adds intensity.

RGB-D cameras can produce:

XYZRGB

so each 3D point also has color.

4. Think of a point cloud as 3D graph paper without the paper

An image is organized.

Pixel:

image[row][column]

has neighbors:

left
right
up
down

A point cloud usually has no such convenient grid.

You might have:

P17 = (3.1, 8.2, 0.5)
P18 = (-1.2, 5.9, 2.7)

Nothing guarantees that P17 and P18 are spatial neighbors.

This makes point clouds powerful but awkward.

They are:

unordered
irregular
sparse
density-varying
potentially noisy

This matters enormously for algorithms and neural networks.

A normal CNN expects something like:

height × width × channels

A raw point cloud instead looks like:

N × features

with no inherent ordering.

This difficulty helped motivate architectures such as PointNet, which directly process sets of points while respecting the fact that point order should not matter.

5. Where point clouds come from

Point clouds can come from:

LiDAR

laser measurements
→ XYZ

RGB-D camera

depth image + camera intrinsics
→ XYZ

Stereo vision

left image + right image
→ disparity
→ depth
→ XYZ

Structure from Motion

many RGB photographs
→ feature matches
→ triangulation
→ sparse point cloud

Multi-view stereo

multiple calibrated images
→ dense reconstruction
→ dense point cloud

So:

Point cloud is a representation, not a specific sensor technology.

That is worth remembering.

6. Depth maps

A depth map looks like an ordinary image, except every pixel stores distance instead of color.

Instead of:

pixel = (R,G,B)

you might have:

pixel = depth

Example:

2.1  2.1  2.2
2.0  1.2  2.1
2.0  2.0  2.1

The 1.2 might be an object closer to the camera.

7. Depth map intuition

Imagine painting the world according to distance.

Near:

dark

Far:

bright

or vice versa.

Then the image becomes a map of geometry.

Suppose an RGB image contains:

chair
wall
table

A depth map might tell you:

chair: 1.7 m
table: 2.3 m
wall: 4.9 m

Now your robot understands not only appearance but spatial layout.

8. Depth map → 3D point

Suppose a pixel is at:

(u,v)(u,v)

and the measured depth is:

Using the camera intrinsics:

fx,fy,cx,cyf_x,\quad f_y,\quad c_x,\quad c_y

we obtain approximately:

X=(u−cx)ZfxX = \frac{(u-c_x)Z}{f_x}Y=(v−cy)ZfyY = \frac{(v-c_y)Z}{f_y}Z=ZZ = Z

So:

pixel + depth + camera calibration
→ 3D point

Do this for every valid depth pixel:

depth map
→ point cloud

This relationship is extremely important.

9. Depth map vs point cloud

They may describe almost the same geometry, but their structure differs.

Depth map

organized
H × W

Every depth value corresponds to a camera pixel.

Great for:

image-like processing
CNNs
RGB association
local neighborhood operations

Point cloud

N × 3

or:

N × features

Great for:

arbitrary 3D geometry
combining measurements from many viewpoints
spatial processing
global mapping

A useful mental conversion:

Depth map = 2.5D view from one camera
Point cloud = explicit samples in 3D space

Why “2.5D”?

Because a depth camera normally sees only the first visible surface along each ray.

If a chair is in front of a wall:

camera → chair → wall

the pixel usually records the chair's depth.

It does not tell you what exists behind the chair.

10. RGB-D

RGB-D means:

RGB + Depth

Example:

RGB camera:
"What does the object look like?"

Depth sensor:
"How far away is it?"

Together:

"What does it look like, and where is it?"

That is extremely useful for robotics.

11. RGB-D example

Suppose your robot sees:

RGB:
red cup on wooden table

Depth provides:

cup ≈ 0.8 m
table surface ≈ 0.82 m
wall ≈ 3.5 m

Now the robot can potentially estimate:

cup position in camera frame
→ transform to robot frame
→ plan arm motion
→ grasp cup

This is why RGB-D cameras became very important for indoor robotics and manipulation.

12. RGB-D alignment matters

A subtle engineering problem:

The RGB camera and depth sensor may not have exactly the same viewpoint.

You therefore need calibration between them:

Depth frame
   ↓ extrinsic transform
RGB frame

Otherwise:

red pixel from cup

could accidentally be associated with:

depth belonging to background

Always remember:

Multisensor perception is useless without correct geometry between sensors.

Calibration is not housekeeping.

Calibration is part of perception.

13. LiDAR data

LiDAR stands for:

Light Detection and Ranging

Very roughly:

emit laser
↓
laser hits object
↓
light returns
↓
measure travel
↓
estimate distance

A rotating LiDAR repeats this many thousands or millions of times.

The result is a geometric scan of the environment.

14. LiDAR does not fundamentally “see images”

A LiDAR thinks more like:

direction θ, φ
distance r
intensity I

which becomes:

x = r cos(...)
y = r sin(...)
z = ...

The exact equations depend on sensor geometry.

Result:

Point Cloud

Example:

x      y      z      intensity
--------------------------------
8.22   1.31   0.44   118
8.19   1.39   0.46   123
8.16   1.48   0.45   115

15. Why robotics loves LiDAR

LiDAR gives direct geometric measurements.

Advantages often include:

accurate range
long range
works without scene texture
straightforward geometry
useful day or night
strong for mapping and localization

But it has limitations:

sparse compared with camera images
expensive hardware depending on sensor class
reflective/transparent materials can be difficult
rain/fog/dust can affect measurements
moving sensor creates temporal distortion
mechanical systems have scanning patterns

No sensor is universally superior.

Modern systems often combine sensors.

16. One subtle LiDAR problem: the scan is not instantaneous

Suppose a spinning LiDAR requires some time to complete a rotation.

Meanwhile the vehicle moves:

start scan:
car at position A

middle:
car at position B

end:
car at position C

If you pretend all points were captured from position A:

walls bend
objects stretch
geometry distorts

The correction is often called:

deskewing

You use motion estimates from things such as:

IMU
odometry
high-rate pose estimation

to transform every point according to its measurement timestamp.

This is the kind of issue that separates a demo from a production robotics system.

17. Filtering

Real 3D data is messy.

Imagine LiDAR returns:

real wall points
real floor points
dust
multipath artifacts
random isolated points
vehicle body
extremely distant measurements
NaNs

Before reasoning about them, we often filter.

18. Pass-through filtering

The simplest filter:

Keep only points inside some range.

Example:

0 < x < 20 m
-10 < y < 10 m
-2 < z < 3 m

Useful when you know the robot only cares about a region.

Example:

A warehouse robot may ignore everything higher than:

2.5 m

for local ground navigation.

19. Range filtering

Discard points too close or too far:

r < minimum_range → reject
r > maximum_useful_range → reject

For example:

keep 0.5 m ≤ r ≤ 80 m

Why?

Measurements outside the reliable operating range may add noise without helping perception.

20. Statistical outlier removal

Suppose most points form a wall:

...............
...............
...............

but one random measurement appears far away:

For every point, estimate distances to neighboring points.

If one point's neighborhood distances are extremely unusual:

likely outlier

remove it.

Mental model:

Real surfaces create communities of points. Random noise often lives alone.

21. Radius outlier filtering

Ask:

“How many neighboring points exist within radius rr?”

If:

neighbors < threshold

remove the point.

Example:

within 10 cm:
only 1 neighbor

Probably noise.

22. Ground filtering

For autonomous vehicles and mobile robots, the ground may contain enormous numbers of points.

Sometimes you need it.

Sometimes you want to remove it before object detection.

Methods include:

height thresholds
plane fitting
RANSAC
slope reasoning
grid-based ground estimation
learned ground segmentation

But be careful.

This:

z < 0.2 → ground

works beautifully...

until the robot drives:

uphill
downhill
over a ramp
over uneven terrain

Production systems should reason about surfaces, not blindly assume the world is flat.

23. Voxelization

A voxel is the 3D equivalent of a pixel.

Pixel:

2D square

Voxel:

3D cube

Imagine dividing the world into tiny cubes:

┌───┬───┬───┐
│   │   │   │
├───┼───┼───┤
│   │ X │   │
├───┼───┼───┤
│   │   │   │
└───┴───┴───┘

but in three dimensions.

24. Why voxelize a point cloud?

Suppose you have:

2,000,000 points

Many may describe essentially the same small patch of wall.

Divide space into:

5 cm × 5 cm × 5 cm

voxels.

If 30 points fall inside one voxel, replace them with:

centroid

Now:

2,000,000 points
→ maybe 300,000 points

while retaining useful geometry.

This is called:

voxel grid downsampling

25. Voxelization performs two jobs

First:

Compression

Reduce redundant points.

Second:

Regularization

Convert irregular 3D samples into spatial cells.

That makes many algorithms easier.

26. The voxel-resolution tradeoff

Suppose voxel size:

1 cm

Result:

high geometric detail
large memory
high computation

Voxel size:

50 cm

Result:

low computation
poor detail
small objects disappear

This tradeoff appears everywhere in 3D perception:

Resolution versus memory versus computation.

27. Why dense voxel grids become expensive

Imagine a volume:

100 m × 100 m × 10 m

using:

0.1 m voxels

Number of cells:

1000 × 1000 × 100
= 100,000,000

Most may contain nothing.

Storing all of them is wasteful.

Hence modern 3D systems frequently use:

sparse voxels
octrees
hashed grids
sparse tensors
hierarchical structures

Store or compute only where useful.

28. KD-trees

Now suppose you have a point:

p = (3.1, 2.7, 1.0)

and one million points in your map.

You ask:

Which existing point is closest to p?

Brute force:

compare p against 1,000,000 points

Do that for every point during registration and you quickly get expensive computation.

A KD-tree is a spatial search structure designed to accelerate queries such as:

nearest neighbor
k nearest neighbors
points inside radius r

29. KD-tree intuition

Imagine repeatedly splitting space:

first by x
then by y
then by z
then x again
...

Instead of searching the entire world, you eliminate huge regions quickly.

Conceptually:

"Point is on the left side."
→ ignore right half

"Point is above this split."
→ ignore lower half

Eventually only a small region needs detailed searching.

30. Where KD-trees appear

They are everywhere in classical point-cloud processing:

ICP
normal estimation
outlier removal
surface analysis
clustering
nearest-neighbor matching

Whenever you hear:

“Find nearby points”

think:

spatial index
KD-tree
voxel hash
octree

depending on the application.

31. Surface normals

A point tells you:

where the surface is

A normal tells you:

which direction the surface faces

For a floor:

normal ≈ upward

For a vertical wall:

normal ≈ sideways

For a tilted ramp:

normal ≈ tilted upward

32. Normal intuition

Imagine placing a tiny arrow perpendicular to a surface:

          ↑ normal
          │
──────────┴──────────
        surface

That arrow is the surface normal.

Represented as:

n=(nx,ny,nz)n=(n_x,n_y,n_z)

usually normalized:

∥n∥=1\|n\|=1

33. How do you estimate a normal from points?

Suppose you want the normal at point PP.

Find nearby points:

P1 P2 P3 ... Pk

using something like a KD-tree.

Those points approximately lie on a local surface.

Find their centroid:

μ\mu

Build a covariance matrix from:

Pi−μP_i-\mu

Then compute its eigenvectors.

Here is the intuition that matters.

If the points lie roughly on a plane, they vary strongly:

along direction 1
along direction 2

but very little:

perpendicular to the plane

The eigenvector associated with the smallest eigenvalue therefore approximates the surface normal.

That is the memorable idea.

The normal points in the direction where neighboring surface points vary least.

34. Why normals matter

Normals help with:

Surface classification

horizontal normal
→ maybe floor/table

Registration

Point-to-plane ICP uses normals.

Mesh reconstruction

Normals help determine surface orientation.

Grasping

Robot needs object surface orientation.

Traversability

normal too steep
→ slope unsafe

Rendering

Lighting calculations depend heavily on normals.

35. Registration

Now we reach one of the most important concepts in robotics.

Suppose your robot scans a room at time:

t1

Then moves and scans again at:

t2

You obtain:

Point Cloud A
Point Cloud B

But B is measured from a different sensor position.

Registration asks:

What rigid transformation aligns B with A?

In other words, find:

rotation R
translation t

such that:

PA≈RPB+tP_A \approx R P_B+t

36. Registration in plain English

You have:

scan 1

and:

scan 2

Find how scan 2 must be:

rotated
+
shifted

so that both scans describe the same physical environment.

That is registration.

37. Why registration is fundamental

If the robot repeatedly collects:

scan1
scan2
scan3
scan4
...

and registers them together:

scan1
 + scan2
 + scan3
 + scan4
 ↓
larger map

This lies at the heart of:

LiDAR odometry
scan matching
mapping
SLAM
3D reconstruction
object pose estimation

38. Registration has two broad stages

A very useful engineering distinction:

Coarse registration

Get reasonably close.

Possible sources:

odometry
IMU
GPS
feature matching
descriptors
RANSAC
learned correspondence methods

Then:

Fine registration

Refine alignment.

The classic tool:

ICP

39. ICP — Iterative Closest Point

ICP is one of the classic algorithms that every robotics engineer should understand.

The name explains the algorithm:

Iterative
Closest
Point

Suppose:

Cloud A = target
Cloud B = source

Goal:

move B until it overlaps A

40. ICP algorithm

Repeat:

Step 1 — Find correspondences

For every point in B:

find nearest point in A

KD-tree often helps.

Step 2 — Estimate transform

Compute the:

rotation + translation

that best aligns those pairs.

Step 3 — Transform B

B ← R B + t

Step 4 — Repeat

Continue until:

change becomes tiny

or iteration limit is reached.

Conceptually:

wrong alignment
   ↓
find closest points
   ↓
move clouds closer
   ↓
find new closest points
   ↓
move closer
   ↓
...
   ↓
aligned

41. ICP's biggest weakness

ICP is generally a local optimization technique.

Suppose the correct alignment is:

10 meters away

but you initialize the clouds badly.

ICP may match:

wall A

to:

wrong wall B

and happily converge to a wrong solution.

Therefore:

ICP usually needs a reasonable initial guess.

That initial guess may come from:

wheel odometry
IMU
GPS
visual odometry
previous pose
feature-based registration

Never mentally classify ICP as:

“magic global alignment.”

It is not.

42. Point-to-point ICP

The simplest objective says:

Matched points should occupy the same location.

Approximately minimize:

∑i∥pi−(Rqi+t)∥2\sum_i \|p_i-(Rq_i+t)\|^2

where:

pᵢ = target point
qᵢ = source point

Good conceptual starting point.

43. Point-to-plane ICP

Suppose a source point lies slightly away from a wall.

Instead of minimizing full point-to-point distance:

point → point

minimize displacement along the target surface normal:

point → plane

Approximately:

∑i[ni⊤(pi−(Rqi+t))]2\sum_i \left[ n_i^\top \left(p_i-(Rq_i+t)\right) \right]^2

This often converges better for dense surfaces.

Why?

Because if two sampled points lie at different locations along the same wall:

x       target point

---------------- wall

        x source point

point-to-point may penalize sideways displacement unnecessarily.

Point-to-plane primarily asks:

“How far is this point from the actual surface?”

That is often what we really care about.

44. ICP engineering problems

Real ICP requires handling:

Partial overlap

Clouds may only partially see the same environment.

Outliers

Nearest neighbor may not be a true correspondence.

Dynamic objects

Cars and people move.

Repetitive geometry

Two walls may look geometrically identical.

Different point densities

One scan may be much denser.

Poor initial pose

Local minima.

Degenerate scenes

A long flat corridor may not constrain every degree of freedom equally.

This leads to an important robotics idea:

Not every environment gives enough geometric information to estimate every motion reliably.

45. Occupancy maps

A point cloud says:

“I observed surfaces here.”

But navigation needs another question:

Can I move through this location?

Enter occupancy maps.

Each cell represents whether space is:

free
occupied
unknown

For a 2D navigation map:

0 = free
1 = occupied
? = unknown

In 3D, the equivalent may be an occupancy voxel map.

46. Occupied and free space are different pieces of information

Suppose LiDAR sends a ray:

robot ----------------------------> wall

The laser travels through all the space before hitting the wall.

Therefore:

space before hit = probably free
endpoint = probably occupied
space behind wall = unknown

This is extremely important.

A point cloud records mainly:

the endpoint

An occupancy mapping process can record:

free ray
+
occupied endpoint
+
unknown behind it

That is much more useful for navigation.

47. Unknown is not free

One of the most dangerous conceptual mistakes in robotics:

unknown == free

No.

Suppose the robot cannot see behind a wall.

That region is:

unknown

It could contain:

empty space
another room
a person
a staircase
a machine

A robust navigation system treats these categories deliberately.

48. Probabilistic occupancy

Real sensors are noisy.

Instead of:

occupied = true

maintain:

P(occupied)P(\text{occupied})

For example:

0.95 → very likely occupied
0.50 → uncertain
0.05 → very likely free

Repeated observations update confidence.

49. Log-odds occupancy

Probabilities are often represented internally using log odds:

L=log⁡p1−pL = \log\frac{p}{1-p}

Why?

Because repeated Bayesian updates become convenient additions:

old belief
+ sensor evidence
= updated belief

Conceptually:

repeated obstacle hits
→ occupancy confidence increases

repeated rays passing through
→ free-space confidence increases

You do not need to memorize the derivation initially.

Remember the engineering meaning:

Occupancy mapping accumulates evidence instead of believing one measurement absolutely.

50. 2D vs 3D occupancy

A mobile robot moving on a floor may use:

2D occupancy grid

because navigation largely happens in:

x, y

A drone may need:

3D occupancy

because motion includes:

x, y, z

Again:

Representation follows task.

Do not build a giant 3D map merely because 3D sounds more advanced.

51. Meshes

Point clouds sample surfaces:

. . . . .
 . . . .
. . . . .

A mesh explicitly connects surface elements:

 /\____/\
/__\__/__\

The most common primitive is the triangle.

A mesh contains roughly:

vertices
+
edges
+
faces

Example:

vertex 1 = (x1,y1,z1)
vertex 2 = (x2,y2,z2)
vertex 3 = (x3,y3,z3)

face = [1,2,3]

The triangle connecting those vertices represents a surface.

52. Point cloud vs mesh

Point cloud:

samples of surface

Mesh:

connected surface

Imagine drawing a sphere.

Point cloud:

thousands of dots on sphere

Mesh:

thousands of small triangles forming sphere

53. Why meshes matter

Meshes are excellent for:

visualization
simulation
CAD-like geometry
surface reconstruction
collision geometry
texture mapping
digital twins
rendering

But meshes require topology:

which vertices belong together?

That can be hard to recover robustly from noisy sensor data.

54. Signed Distance Fields — SDF

Now we reach one of the most useful geometric representations in modern robotics.

An SDF answers:

For any location in space, how far am I from the nearest surface?

But distance has a sign.

Common convention:

outside object → positive
surface → zero
inside object → negative

So:

SDF(x,y,z)SDF(x,y,z)

returns a scalar.

Example:

+2.3 m → 2.3 m outside nearest surface
+0.1 m → 10 cm outside
 0.0   → on surface
-0.1 m → 10 cm inside

55. SDF intuition

Imagine a wall:

           distance
              ↓
+3 +2 +1 | 0 | -1 -2 -3
         wall

The zero level set:

SDF(x)=0SDF(x)=0

defines the actual surface.

This is powerful.

Instead of explicitly saying:

here are 10 million triangles

you can say:

“The surface consists of all locations where this function equals zero.”

56. Why distance fields are powerful for robots

Suppose a robot arm is planning motion.

At configuration A:

distance to obstacle = 0.8 m

At B:

distance = 0.15 m

At C:

distance = -0.02 m

C means collision.

This gives planners more information than:

collision?
true/false

It tells them:

how close

which creates useful gradients for optimization.

57. TSDF — Truncated Signed Distance Field

In mapping, you will often encounter:

TSDF

The T means:

Truncated

Instead of storing arbitrarily large distances from a surface:

8.7 m
12.3 m
54.1 m

we only care about a region close to surfaces.

Example:

distance > +0.3 m → clamp to +0.3
distance < -0.3 m → clamp to -0.3

Hence:

TSDF

This is very useful when fusing many depth observations into a smooth reconstructed surface.

Conceptually:

RGB-D frames
 ↓
camera poses
 ↓
TSDF fusion
 ↓
smooth surface model
 ↓
mesh

A common final step is extracting the:

zero-crossing

to recover the surface.

58. ESDF — Euclidean Signed Distance Field

Another acronym that matters tremendously in robotics:

ESDF

An ESDF stores approximately the Euclidean distance to the nearest obstacle.

That is especially valuable for:

motion planning
trajectory optimization
collision avoidance

Mental separation:

TSDF
→ excellent for integrating sensor measurements near surfaces

ESDF
→ excellent for asking "How far am I from obstacles?"

These are related but serve different downstream needs.

Remember that distinction.

59. Occupancy vs SDF

Occupancy asks:

occupied or not?

SDF asks:

how far from a surface?

Example:

Occupancy:
cell = FREE

SDF:
distance = 0.42 m from obstacle

For simple graph search, occupancy may be enough.

For smooth trajectory optimization, distance information can be extremely valuable.

60. Registration + mapping together

Let's connect everything.

Robot has LiDAR.

At time t1t_1:

scan A

At time t2t_2:

scan B

You estimate pose change:

ICP / LiDAR odometry

Transform B into map frame:

B_world = T_world_sensor × B_sensor

Then insert measurements into:

occupancy map
voxel map
TSDF
point cloud map

Repeat:

sense
→ estimate pose
→ transform measurements
→ update map
→ move
→ sense again

That loop is one of the central loops of robotics.

61. Coordinate frames: the invisible source of thousands of bugs

A 3D point is meaningless without a coordinate frame.

This:

(1,2,3)

means almost nothing.

You need:

(1,2,3) in camera frame

or:

(1,2,3) in LiDAR frame

or:

(1,2,3) in robot base frame

or:

(1,2,3) in world/map frame

These are different statements.

62. Typical robot frame chain

In ROS-like systems you may encounter:

map
 ↓
odom
 ↓
base_link
 ↓
lidar
 ↓
camera

A point measured in LiDAR frame:

pLp_L

must be transformed before inserting into map coordinates:

pmap=Tmap,baseTbase,lidarpLp_{map} = T_{map,base} T_{base,lidar} p_L

The exact notation varies.

The mental model does not:

Every sensor measures the world from its own coordinate frame.

63. Three things must agree

Successful 3D fusion requires:

SPACE
TIME
CALIBRATION

Meaning:

Space

Correct transforms.

Time

Measurements correspond to correct robot poses.

Calibration

Sensor models and inter-sensor transformations are accurate.

If one is wrong:

map becomes blurry
walls duplicate
objects ghost
ICP fails
depth colors misalign

This is a major practical lesson.

64. NeRF concepts

Now we move from classical geometric maps into neural scene representations.

NeRF means:

Neural Radiance Field

The original NeRF formulation represents a scene using a neural network mapping continuous spatial position and viewing direction to density and view-dependent color. Images are generated by querying points along camera rays and applying differentiable volume rendering.

Forget the equations for one minute.

The central idea is:

Instead of explicitly storing millions of points or triangles, learn a function describing the scene.

65. Classical representation

Point cloud says:

I explicitly store these points.

Mesh says:

I explicitly store these triangles.

Voxel map says:

I explicitly store these spatial cells.

NeRF says something closer to:

“Give me any position and viewing direction, and my learned function will tell you what this region contributes to the image.”

That is a profound change in representation.

66. What goes into a NeRF?

Original conceptual input:

3D location:
(x,y,z)

view direction:
(θ,φ)

Output:

density σ
color (r,g,b)

So conceptually:

F(x,y,z,θ,ϕ)→(r,g,b,σ)F(x,y,z,\theta,\phi) \rightarrow (r,g,b,\sigma)

Density roughly answers:

"How much matter is present here?"

Color answers:

"What color would this region appear from this direction?"

67. How does a NeRF render a pixel?

Camera emits a conceptual ray:

camera
  \
   \
    •
     •
      •
       •

Sample positions along the ray:

x1
x2
x3
...

Ask the neural field:

density?
color?

at every sample.

Then combine them with volume rendering.

Intuitively:

empty space
→ contributes almost nothing

surface-like dense region
→ contributes strongly

anything behind an opaque region
→ contributes little

The result becomes the pixel color.

68. How is NeRF learned?

Suppose you photograph a room from many viewpoints.

You know approximately:

image 1 + camera pose 1
image 2 + camera pose 2
image 3 + camera pose 3
...

The network renders what it predicts each camera should see.

Compare:

predicted pixel
vs
actual pixel

Adjust parameters using gradient descent.

Eventually the learned field explains the set of observations.

In beautiful conceptual shorthand:

many 2D images
+
camera geometry
+
differentiable rendering
→
learned 3D scene representation

69. Why NeRF was such an important idea

It demonstrated powerfully that scene geometry and appearance could be represented using a continuous learned field, rather than only explicit meshes, voxels, or points. The original work particularly targeted high-quality novel-view synthesis.

Later neural-field research explored many related representations, including faster encodings such as multiresolution hash features. NVIDIA's Instant-NGP, for example, combined trainable multiresolution hash-table features with a small neural network to greatly accelerate neural graphics primitives.

For a robotics engineer, however, one important caution is:

A representation optimized for photorealistic rendering is not automatically the representation you want for safety-critical geometric planning.

A robot may prefer an explicit:

occupancy
ESDF
collision mesh

for downstream planning even if a neural representation is also maintained.

70. Never remember NeRF as merely “AI making 3D”

Remember:

coordinates
      ↓
learned continuous function
      ↓
density + appearance
      ↓
differentiable ray rendering
      ↓
image

That is the conceptual heart.

71. Gaussian Splatting concepts

Another important modern scene representation is:

3D Gaussian Splatting

The original 3D Gaussian Splatting method represents a scene using a collection of anisotropic 3D Gaussians, optimizes their properties from images, and renders them efficiently using a visibility-aware splatting procedure. Its original work emphasized high-quality real-time novel-view rendering.

Again, first understand the intuition.

72. What is a Gaussian “splat”?

Instead of representing a scene as:

hard point

imagine placing a tiny fuzzy 3D blob:

      ....
    ........
   ...###...
    ........
      ....

The influence is strongest around its center and fades outward.

Each Gaussian can have properties such as:

position
size
orientation
opacity
appearance/color coefficients

And importantly, the blob does not need to be spherical.

It can be stretched:

───────

or flattened:

pancake-like

That anisotropic shape helps Gaussians approximate surfaces.

73. Gaussian representation intuition

Suppose a wall exists.

Rather than:

millions of mesh triangles

or:

one giant neural network queried repeatedly along every ray

you might approximate visible scene structure using many optimized Gaussians:

○ ○ ○ ○ ○ ○
 ○ ○ ○ ○ ○
○ ○ ○ ○ ○ ○

They collectively produce the appearance of the surface.

74. Why call it “splatting”?

When rendering, a 3D Gaussian is projected into the image.

Its projected contribution becomes something like an elliptical footprint:

   ....
 ........
...####...
 ........
   ....

That footprint is blended with other projected Gaussians.

Conceptually:

3D Gaussian
 ↓ projection
2D elliptical splat
 ↓ compositing
pixel colors

The original method's efficient rasterization is a major reason the representation can render quickly.

75. NeRF vs Gaussian Splatting: intuitive comparison

Do not reduce the distinction to:

NeRF slow
Gaussian fast

That is too shallow.

The deeper representational difference is approximately:

NeRF-style field

query continuous coordinates
→ neural function produces scene properties

3D Gaussian representation

explicit collection of optimized spatial primitives
→ project/rasterize them

A good mental image:

NeRF
= the scene lives largely inside a learned function

Gaussian splatting
= the scene lives as many explicit fuzzy 3D primitives

There are many hybrids and later variations, but this is the useful conceptual starting point.

76. Are Gaussian splats a normal point cloud?

Not really.

A normal point may store:

position
color

A Gaussian primitive may encode something more like:

mean position
3D covariance / shape
opacity
appearance information

So instead of:

"there is a sample exactly here"

you have:

“There is a spatially extended contribution centered here with this shape and appearance.”

77. Why robotics engineers should care about NeRFs and splats

Because the boundary between:

perception
mapping
reconstruction
graphics
world models

is increasingly blurry.

A robot may want a representation that allows:

novel-view rendering
localization
semantic querying
geometry extraction
simulation
planning
language grounding

potentially from a shared world model.

But this also creates an important engineering question:

Does the representation encode the information my controller actually needs?

Photorealism is not the same as:

metric accuracy
free-space certainty
collision safety
dynamic-object reasoning

Never forget that distinction.

78. Learned 3D representations

Classical geometry often builds representations manually:

points
voxels
meshes
SDF
occupancy

Modern learned 3D perception asks:

Can the system learn features or representations that are better suited for perception?

The answer has produced several families of approaches.

79. Family 1 — Point-based learning

Input:

raw points

Example:

N × (x,y,z,intensity,...)

Network learns directly from point sets.

Classic conceptual example:

PointNet

Its key insight was to process unordered points in a way that does not depend on arbitrary input ordering.

Later approaches build richer local neighborhood structures.

Useful for:

classification
point segmentation
object detection
scene understanding

80. Family 2 — Voxel-based learning

Convert points into voxels:

point cloud
 ↓
voxelization
 ↓
3D tensor / sparse tensor
 ↓
3D convolution

The problem with dense 3D convolution:

empty space everywhere

So modern systems often use:

sparse convolutions

which focus computation primarily on occupied/active regions.

Conceptually:

dense grid:
compute everywhere

sparse grid:
compute where data exists

81. Family 3 — Range-image representations

A spinning LiDAR has structured scan geometry.

Instead of treating measurements as fully unordered points, project them into something like:

vertical laser channel × horizontal angle

forming a range image.

Then:

LiDAR
→ 2D-like representation
→ CNN-style processing

Advantages:

efficient
preserves sensor structure
can use mature image-network ideas

Tradeoff:

The representation is tied more strongly to sensor viewpoint.

82. Family 4 — Bird's-Eye View

One of the most useful representations in autonomous driving is:

BEV — Bird's-Eye View

Imagine looking straight down at the world:

          road
  car      car

      ego vehicle

 pedestrian

Instead of reasoning primarily in camera perspective:

far objects tiny
near objects huge

BEV converts perception into a common ground-plane-centric spatial representation.

This is useful because planning also happens spatially.

83. Why BEV feels natural to a planner

Camera asks:

where in image?

Planner asks:

where in world?

BEV bridges that gap.

Multiple sensors may be fused into:

shared top-down feature space

allowing reasoning about:

lanes
vehicles
free space
occupancy
trajectories
map structure

84. Family 5 — Occupancy networks

Instead of explicitly storing every voxel, learn a function:

f(x,y,z)→P(occupied)f(x,y,z) \rightarrow P(\text{occupied})

Ask:

Is point (x,y,z) inside the object?

The network answers approximately:

0.02 → likely outside
0.97 → likely inside

The object surface occurs near the decision boundary.

Occupancy Networks formalized this style of continuous learned occupancy representation for 3D reconstruction.

This is another example of implicit geometry.

85. Family 6 — Learned SDFs

Instead of:

network → occupancy probability

use:

network → signed distance

fθ(x,y,z)→sf_\theta(x,y,z) \rightarrow s

where:

s > 0 outside
s = 0 surface
s < 0 inside

DeepSDF is an influential example of learning continuous signed-distance functions as shape representations.

Again, the geometry exists implicitly through:

f(x,y,z)=0

86. Family 7 — Radiance fields

NeRF belongs here.

Input:

position + viewing direction

Output:

density + appearance

Optimized heavily around explaining visual observations.

87. Family 8 — Gaussian scene representations

Represent the scene with many learned spatial Gaussian primitives.

These sit interestingly between:

explicit geometry

and:

learned appearance representation

because the Gaussians themselves have explicit spatial positions and shapes, while their parameters are optimized from observations.

88. Family 9 — Learned 3D feature fields

A modern robot may need more than geometry.

At position:

(x,y,z)

you may want a feature vector:

f(x,y,z)∈Rdf(x,y,z)\in \mathbb{R}^d

representing things such as:

geometry
appearance
semantics
object identity
language-related concepts
traversability
interaction affordances

Then the world is no longer merely:

occupied/free

but something closer to:

this region is probably floor

this region belongs to chair #4

this handle is graspable

this object corresponds to "red mug"

this space is traversable

That direction is highly relevant to embodied AI.

89. Geometry alone is not enough

Consider two surfaces with identical geometry:

surface A = concrete floor
surface B = deep water

Geometrically both might look:

flat

But for a ground robot:

A → traversable
B → dangerous

Therefore useful world models increasingly combine:

geometry
+
semantics
+
physics
+
affordances
+
uncertainty

90. A powerful hierarchy of representations

You can think of 3D understanding as increasingly semantic:

Level 1 — Measurements

depth
laser return

Level 2 — Geometry

point cloud
surface normal
mesh

Level 3 — Space structure

free
occupied
distance to obstacle

Level 4 — Objects

chair
car
person

Level 5 — Semantics

floor
door
table
road

Level 6 — Affordances

walkable
graspable
openable
sit-able
drivable

Level 7 — Dynamic/world reasoning

person moving toward doorway
door likely to open
vehicle trajectory
object permanence

Autonomous intelligence lives increasingly toward the bottom of this hierarchy.

But the lower levels depend on the upper ones being geometrically sound.

91. The representation cheat sheet

Depth Map

Think:

Distance image

Shape:

H × W

Best mental use:

geometry from one camera view

Point Cloud

Think:

3D dots

Shape:

N × 3+

Great for:

sensor geometry
registration
LiDAR processing

Voxels

Think:

3D pixels

Great for:

spatial discretization
occupancy
3D neural processing

Occupancy Map

Think:

Can I be here?

Values:

free
occupied
unknown

Great for:

navigation
mapping

Mesh

Think:

Connected skin of triangles

Great for:

surfaces
visualization
simulation

SDF

Think:

How far am I from the nearest surface?

Great for:

geometry
collision distance
optimization

TSDF

Think:

SDF near measured surfaces

Great for:

sensor fusion
surface reconstruction

ESDF

Think:

Distance to obstacles everywhere useful

Great for:

motion planning

NeRF

Think:

Learned continuous scene function for appearance + density

Great conceptual strength:

novel views
continuous neural scene representation

Gaussian Splatting

Think:

Scene made from optimized fuzzy 3D ellipsoids

Great conceptual strength:

explicit spatial primitives
+
fast high-quality rendering

92. A complete robot example

Imagine an autonomous warehouse robot.

Sensors:

LiDAR
RGB-D camera
IMU
wheel encoders

Stage 1 — measurements

LiDAR:

ranges + intensity

RGB-D:

RGB + depth map

IMU:

angular velocity + acceleration

Encoders:

wheel movement

Stage 2 — preprocessing

LiDAR:

remove invalid points
range filtering
deskew
voxel downsample

RGB-D:

remove invalid depth
align depth with RGB

Stage 3 — geometric features

Estimate:

surface normals
planes
edges
local descriptors

Stage 4 — registration

New LiDAR scan:

initial guess from odometry/IMU
↓
ICP / scan matching
↓
refined pose

Stage 5 — mapping

Transform points into world frame:

sensor frame
↓
robot frame
↓
map frame

Update:

occupancy map

Maybe also:

TSDF

for reconstruction.

Maybe:

ESDF

for trajectory planning.

Stage 6 — semantics

RGB network recognizes:

box
pallet
person
forklift

Depth associates them with 3D positions.

Now the map can contain:

geometry
+
semantic labels

Planner receives:

robot pose
goal pose
occupancy
distance field
dynamic obstacles

and generates:

safe trajectory

That is 3D perception becoming autonomy.

93. Another example: robot arm

A manipulation robot needs to grab a mug.

Camera obtains:

RGB-D

From RGB:

detect mug

From depth:

recover mug points

Filtering:

remove table plane

Point processing:

cluster mug points

Normals:

estimate surface orientation

Pose estimation:

estimate mug transform

Then:

mug frame
↓
grasp pose
↓
collision check using geometry/SDF
↓
arm trajectory
↓
grasp

Again, perception is not merely recognizing:

"mug"

It is converting sensory evidence into actionable physical coordinates.

94. Another example: drone

Drone enters unknown building.

Depth/LiDAR creates:

3D points

Registration estimates motion:

scan-to-scan
scan-to-map

Map updates:

3D occupancy

ESDF calculates:

distance to walls
distance to ceiling
distance to furniture

Planner searches for trajectory:

current pose
→ doorway
→ corridor
→ target room

A 2D map is insufficient because the drone can move:

up/down

Hence representation choice changes with robot dynamics.

95. What beginners often misunderstand

Mistake 1

Point cloud = complete 3D world.

No.

A point cloud usually contains sampled visible surfaces.

Occluded regions remain unknown.

Mistake 2

More points always means better perception.

No.

More points may mean:

more redundancy
more noise
more computation

A well-designed voxelized cloud can outperform an unnecessarily huge cloud computationally.

Mistake 3

ICP always finds correct alignment.

No.

ICP is typically local and depends on:

initialization
geometry
overlap
outlier handling

Mistake 4

Empty measurement means free space.

No.

There is a difference between:

observed free

and:

never observed

Mistake 5

Mesh, point cloud and occupancy grid are interchangeable.

No.

They answer different questions.

Mistake 6

Photorealistic reconstruction means geometrically perfect reconstruction.

Not necessarily.

Rendering quality and metric geometric accuracy are distinct engineering goals.

Mistake 7

Neural representations replace classical robotics geometry.

Usually the better mental model is:

classical geometry
+
learned perception
+
probabilistic estimation
+
task-specific representations

Modern robotics is heavily hybrid.

96. The hidden variable: uncertainty

Imagine a depth camera reports:

2.000 m

Do not mentally interpret that as:

the object is exactly 2.000000000 m away

Sensor measurements contain uncertainty.

Likewise:

LiDAR point
camera pose
ICP transform
normal
object position

all have uncertainty.

When several uncertain stages are chained:

sensor
→ pose
→ mapping
→ detection
→ planning

errors propagate.

Superior robotics engineering therefore asks not just:

“What is my estimate?”

but:

“How much should I trust my estimate?”

97. Dynamic scenes break static assumptions

Many classical mapping systems implicitly assume:

world is static
robot moves

Real world:

robot moves
people move
cars move
doors move
forklifts move
chairs move

Suppose a walking person contributes points to a static map.

After several seconds you might create:

ghost person
stretched geometry
false obstacles

Modern perception therefore frequently separates:

static structure
vs
dynamic objects

This is essential for autonomous systems.

98. Temporal perception matters

A single point cloud gives geometry now.

A sequence:

Pₜ₋₂
Pₜ₋₁
Pₜ

allows reasoning about:

velocity
motion
object persistence
trajectory

A person does not merely exist at:

(x,y,z)

but may be moving with:

(vx,vy,vz)

Planning needs the future:

Where will the person be when my robot reaches that location?

That is the bridge from:

3D perception

to:

4D perception

where time becomes a first-class dimension.

99. The most important practical pipeline to remember

For classical LiDAR robotics:

RAW LIDAR
   ↓
CALIBRATE / DESKEW
   ↓
FILTER
   ↓
DOWNSAMPLE
   ↓
FEATURES / NORMALS
   ↓
REGISTRATION
   ↓
POSE
   ↓
TRANSFORM INTO MAP FRAME
   ↓
MAP UPDATE
   ↓
OCCUPANCY / TSDF / ESDF
   ↓
PLANNING

Memorize the logic, not necessarily every algorithm.

100. RGB-D pipeline

RGB IMAGE
   +
DEPTH IMAGE
   ↓
ALIGNMENT / CALIBRATION
   ↓
DEPTH BACK-PROJECTION
   ↓
COLORED POINT CLOUD
   ↓
FILTERING
   ↓
SEGMENTATION / OBJECT DETECTION
   ↓
3D OBJECTS / SURFACES
   ↓
MAP / MANIPULATION

101. Modern learned pipeline

A modern system might instead look like:

CAMERAS + LIDAR + RADAR
        ↓
SENSOR ENCODERS
        ↓
LEARNED FEATURES
        ↓
COMMON 3D / BEV SPACE
        ↓
OBJECTS + OCCUPANCY + SEMANTICS
        ↓
TEMPORAL FUSION
        ↓
WORLD MODEL
        ↓
PLANNER

Notice something interesting.

Classical robotics explicitly computes:

point
normal
plane
correspondence

Modern systems may learn some of these relationships.

But the underlying geometry has not disappeared.

A model still has to solve the physical problem:

where?
how far?
which frame?
what moves?
what is free?

The computation is merely represented differently.

102. Explicit vs implicit 3D representation

This distinction is worth permanently remembering.

Explicit

Store geometry directly.

Examples:

points
voxels
meshes
Gaussians

Advantages:

easy to inspect
easy spatial lookup in many cases
straightforward geometry

Potential drawbacks:

memory
resolution
topology

Implicit

Store a function.

Examples:

SDF function
occupancy network
NeRF-like neural field
DeepSDF

Instead of:

store every surface location

ask:

f(x,y,z) = ?

Advantages can include:

continuous representation
compact learned structure
smooth interpolation

Potential drawbacks:

network queries
training/optimization
harder direct inspection
extracting explicit geometry may be needed

103. Dense vs sparse

Another fundamental axis:

Dense

Store every spatial location.

Example:

dense voxel grid

Good:

simple indexing

Bad:

huge memory in 3D

Sparse

Store only active regions.

Example:

sparse voxels
octree
hash map
point cloud

Good:

efficient in mostly empty environments

This is why sparsity matters far more in 3D than many beginners expect.

104. Surface vs volume representations

Point cloud:

mostly surface samples

Mesh:

surface

Occupancy:

volume

SDF:

field throughout space

NeRF:

volumetric radiance/density field

This distinction matters because robots often care about:

empty space

just as much as they care about surfaces.

Imagine a wall point cloud:

. . . . . .

You know a wall is there.

But the robot asks:

Can I move here?

Can I move 20 cm from it?

What about behind it?

Was that region observed?

How wide is the doorway?

That is why perception systems frequently transform raw surfaces into representations designed for planning.

106. Geometry quality is not uniform

A 3D system might know:

wall → very accurately
glass door → badly
dark reflective object → uncertain
distant pedestrian → sparse
floor nearby → excellent

Different materials and geometries affect sensors differently.

Therefore:

Never treat every point as equally trustworthy merely because it appears in the same point cloud.

Real systems often exploit:

sensor confidence
distance
incidence angle
intensity
temporal consistency
semantic class

to reason about reliability.

107. Incidence angle intuition

Suppose a laser hits a wall almost perpendicularly:

laser → | wall

Good geometry.

Now nearly parallel:

laser ───────→
              /
             / surface

Small measurement errors can produce larger spatial uncertainty along some directions.

Geometry has conditioning.

This matters in:

normal estimation
registration
plane fitting
depth reconstruction

108. Degeneracy

Suppose your robot sees only one perfectly flat wall.

Can scan matching determine every possible movement?

Imagine sliding sideways along an infinite featureless wall.

Many poses may explain nearly identical measurements.

The environment does not provide enough constraints.

This is called a form of:

degeneracy

Engineering consequence:

LiDAR geometry weak
→ rely more on IMU / odometry / additional sensors

Sensor fusion exists partly because no single sensing modality is always informative.

109. Choosing a representation by task

SLAM / scan matching

Often useful:

point clouds
features
voxels
surfel maps

Often useful:

occupancy map
cost map
ESDF

Manipulation

Often useful:

RGB-D
point cloud
object pose
mesh
SDF

Surface reconstruction

Often useful:

TSDF
mesh
neural implicit field

Photorealistic novel-view synthesis

Often useful:

NeRF-like fields
Gaussian splatting

Learned LiDAR perception

Often useful:

points
voxels
range view
BEV
hybrids

Embodied semantic world models

Potentially useful:

geometry
+
semantic features
+
language-aligned features
+
temporal state

There is no universally winning representation.

110. One world, many representations

A sophisticated robot may simultaneously maintain:

point cloud
+
occupancy grid
+
ESDF
+
semantic objects
+
camera features
+
dynamic tracks

All representing the same physical environment for different purposes.

That is not duplication by accident.

It is often good system design.

Think of it like a database with multiple indexes optimized for different queries.

111. The database analogy

The physical world is the underlying truth.

Different representations answer different queries.

Point cloud:

Where were surfaces measured?

Occupancy:

Can I move here?

ESDF:

How far am I from collision?

Mesh:

What does the surface look like?

Semantic map:

What objects exist here?

Neural field:

What would this scene look like from here?

Planner:

What actions are safe?

This analogy makes representation design much easier to reason about.

112. The deepest idea behind learned 3D representations

Traditional perception engineers hand-designed the representation:

occupancy cell
voxel
surface normal
feature descriptor

Learning-based systems increasingly ask the machine to discover useful internal features.

But learned representations still face unavoidable physical constraints:

3D geometry
occlusion
perspective
sensor noise
motion
time
coordinate frames

Deep learning changes how we approximate the solution.

It does not repeal geometry.

113. Occlusion: the permanent enemy of perception

Camera sees:

front surface

not:

everything behind it

LiDAR similarly often sees first-return geometry along rays.

Therefore the robot's world model is always partly inferred.

Suppose:

camera → box → hidden region

Behind the box could be:

nothing
wall
person
another box

This is why:

unknown

must remain a first-class concept.

114. Perception is not reconstruction of absolute truth

A useful philosophical correction:

A robot does not need a mathematically perfect replica of reality.

It needs a representation sufficient for its task.

Warehouse robot:

Is aisle free?
Where is pallet?
Can I turn safely?

does not necessarily require:

photorealistic reconstruction of every screw

Likewise a rendering system may care greatly about:

view-dependent reflections

while a navigation robot may not.

Engineering is choosing the information worth preserving.

115. The four questions to ask about every 3D representation

Whenever you encounter a new 3D representation, ask:

1. What does it store?

points?
occupancy?
distance?
density?
features?
Gaussians?
triangles?

2. What queries are cheap?

nearest neighbor?
collision?
rendering?
surface extraction?
semantic lookup?

3. What is expensive?

memory?
updates?
training?
rendering?
nearest-neighbor search?

4. What information is lost?

free space?
texture?
topology?
uncertainty?
dynamic state?

If you can answer those four questions, you understand the representation far better than someone who merely knows its name.

116. The same scene represented six ways

Imagine a chair.

Point cloud

• • • •
 • • •
•   •

Meaning:

sampled visible chair surfaces

Voxel grid

occupied cubes

Meaning:

chair occupies these regions

Mesh

connected triangles

Meaning:

chair surface geometry

SDF

distance field

Meaning:

distance from every queried location to chair surface

NeRF-like field

density + appearance function

Meaning:

how scene produces observed views

Gaussian representation

many oriented fuzzy spatial primitives

Meaning:

optimized explicit scene primitives that render the chair

Same physical object.

Completely different computational worlds.

117. The critical boundary between perception and planning

Perception might output:

obstacle at (x,y,z)

Planning needs:

obstacle volume
uncertainty
future motion
safety margin

Therefore a mature autonomy system does not simply pass raw detections directly into control.

There are transformations:

sensor observation
↓
estimated world state
↓
prediction
↓
planning representation
↓
trajectory

The better you understand 3D representations, the easier these interfaces become to design.

118. Safety margins

Suppose estimated obstacle boundary is:

x = 2.00 m

Your localization uncertainty is:

±5 cm

sensor uncertainty:

±3 cm

robot controller tracking error:

±4 cm

Planning directly against the mathematical boundary is foolish.

Instead systems often inflate obstacles or incorporate margins.

Conceptually:

estimated geometry
+
uncertainty
+
robot radius
+
safety margin
→
effective forbidden region

Perception cannot be separated from uncertainty when the robot physically moves.

119. Resolution is a design parameter, not a random constant

Suppose robot width:

0.6 m

Using occupancy voxels:

1.0 m

may destroy useful doorway geometry.

Using:

1 mm

may be computationally absurd.

Choose resolution based on:

sensor accuracy
robot size
object size
environment scale
compute budget
planning requirements

This is systems engineering.

120. What happens when resolution is too coarse?

Thin obstacle:

chair leg
wire
pole

may disappear.

Doorway geometry may change.

Two obstacles may merge.

A corridor may appear blocked.

121. What happens when resolution is too fine?

Memory explodes.

Processing slows.

Sensor noise becomes overrepresented.

Map becomes expensive to update.

Remember:

More precision in representation does not automatically mean more useful information.

122. Multi-resolution representations

A smart approach is often:

high resolution nearby
lower resolution far away

or:

fine geometry around surfaces
coarse representation elsewhere

Hierarchical structures such as octrees use this idea.

Neural multiresolution features use related intuition computationally.

The world does not need uniform computational attention everywhere.

123. 2D, 2.5D and 3D

Useful vocabulary:

2D

x, y

Example:

occupancy floor map

2.5D

One surface depth per image/grid location.

Example:

depth map
height map

Full 3D

Multiple structures may exist at same x/y but different z.

Example:

tabletop
space below table
floor

represented separately.

This matters when deciding map structures.

124. Point density is not uniform

LiDAR points often spread apart with distance.

Near sensor:

•••••••••••

Far away:

•    •    •

because angular spacing translates into larger metric spacing.

Therefore algorithms using a fixed neighbor radius behave differently at different ranges.

This affects:

normal estimation
clustering
object detection
outlier filtering

Never assume point density is uniform unless your representation explicitly makes it so.

125. RGB-D noise also depends on geometry

Depth measurements can become unreliable around:

object boundaries
reflective surfaces
transparent objects
very dark materials
large distances
grazing angles

Suppose depth:

cup edge = invalid

If you blindly convert to a cloud, the object geometry may contain holes.

3D perception engineering is largely the art of treating measurements according to how they were produced.

126. Sensor models matter

A measurement is not simply:

number

It is:

number produced by a physical sensing process

Understanding that process explains:

noise
bias
failure modes
resolution
field of view
latency

This principle applies to:

LiDAR
stereo
ToF depth cameras
structured light
radar
sonar

A superior engineer learns sensor physics sufficiently to understand what the data means.

127. Data association: the hidden problem

Registration asks:

which point in scan A corresponds to which point in scan B?

Tracking asks:

is this person the same person as last frame?

Stereo asks:

which left pixel corresponds to which right pixel?

3D reconstruction asks:

which observations belong to the same physical surface?

This family of problems is called:

data association

Many perception failures are fundamentally correspondence failures.

128. Geometry optimization needs good correspondences

Suppose two points are matched:

chair leg ↔ chair leg

Excellent.

Wrong match:

chair leg ↔ wall

Now even perfect mathematics optimizes the wrong relationship.

This is why robust estimators such as:

RANSAC
robust losses
correspondence rejection

are so important.

Optimization cannot rescue incorrect assumptions indefinitely.

129. ICP and SLAM relationship

ICP answers approximately:

How did I move between overlapping geometric observations?

SLAM asks:

Where am I, and what does the environment look like, while both are initially unknown?

ICP can therefore be one component inside a LiDAR SLAM system.

Example:

new scan
↓
ICP / scan matching
↓
relative pose
↓
pose graph / state estimator
↓
map update

Do not equate:

ICP = SLAM

ICP is an alignment technique.

SLAM is the larger estimation problem.

130. Registration and loop closure are different scales

Local registration:

scan 101 ↔ scan 102

Loop closure:

current location
↔
place visited 20 minutes ago

Local geometric methods can accumulate drift.

Loop closures give global constraints that can correct it.

Again:

3D perception
+
state estimation
+
optimization

work together.

131. Learned representations do not remove coordinate frames

Suppose a transformer produces brilliant 3D features.

The robot still must know:

camera pose
sensor calibration
temporal relationship
map orientation
robot body location

Even very learned systems frequently contain explicit geometry somewhere in the pipeline.

Do not let neural-network terminology hide physical structure from you.

132. NeRF, SDF and occupancy are all functions—but answer different questions

This is a great connection.

Occupancy field

f(x,y,z)→P(occupied)f(x,y,z) \rightarrow P(\text{occupied})

Question:

Is material here?

SDF

f(x,y,z)→distancef(x,y,z) \rightarrow distance

Question:

How far is the surface?

NeRF-style field

f(x,y,z,d)→density,colorf(x,y,z,d) \rightarrow density,color

Question:

How does this scene contribute to images?

Same concept:

coordinate → function → scene property

Different outputs.

Different tasks.

This is one of the deepest unifying ideas in modern 3D representations.

133. Explicit geometry can be extracted from implicit fields

Suppose you have an SDF:

f(x,y,z)f(x,y,z)

Surface:

f(x,y,z)=0f(x,y,z)=0

Algorithms such as Marching Cubes can extract a mesh near that zero-level surface.

So the pipeline may be:

implicit field
↓
sample grid
↓
find zero crossings
↓
mesh

Therefore explicit and implicit representations are not isolated worlds.

Systems often convert between them.

134. Learned occupancy example

Imagine a network trained on incomplete chair scans.

Input:

partial point cloud

Visible:

front half of chair

Network may infer:

likely complete chair geometry

Output occupancy field can represent the completed object.

This is powerful—but note the philosophical shift.

Classical geometry says:

I reconstruct what was observed.

Learned reconstruction may say:

I infer what probably exists based on learned priors.

For robotics, the distinction between:

observed

and:

inferred

can matter tremendously.

135. Do not confuse inference with measurement

Suppose a network predicts:

there is probably a back surface here

That may be useful.

But it is not equivalent to:

LiDAR directly measured a back surface here.

A sophisticated world model should ideally retain provenance or confidence:

observed
predicted
uncertain

Especially for safety-critical decisions.

136. The future-facing robotics world model

A useful conceptual target is a representation containing:

GEOMETRY
What shape does the world have?

SEMANTICS
What are these things?

INSTANCE
Which individual object is this?

DYNAMICS
How is it moving?

UNCERTAINTY
How sure am I?

AFFORDANCE
What can I do with it?

TEMPORAL MEMORY
What existed before?

PREDICTION
What will likely happen next?

That is much richer than:

point cloud

But point clouds, occupancy, SDFs and geometry remain foundational because all those higher-level concepts must eventually correspond to physical space.

137. The simplest possible memory chain

If you forget everything, remember this:

DEPTH
= distance per pixel

POINT CLOUD
= 3D dots

VOXEL
= 3D pixel

KD-TREE
= fast nearby-point search

NORMAL
= which way surface faces

REGISTRATION
= align two 3D observations

ICP
= repeatedly match closest points and refine alignment

OCCUPANCY
= free / occupied / unknown

MESH
= connected triangular surface

SDF
= signed distance to surface

TSDF
= truncated distance field useful for reconstruction

ESDF
= obstacle-distance field useful for planning

NeRF
= learned continuous radiance/density field

GAUSSIAN SPLATTING
= scene represented by optimized fuzzy 3D Gaussian primitives

LEARNED 3D REPRESENTATION
= let neural networks learn spatial features/functions useful for downstream tasks

That is the entire chapter compressed.

138. And the master pipeline

Burn this into memory:

        PHYSICAL WORLD
              ↓
           SENSORS
       ┌──────┴──────┐
     CAMERA         LiDAR
       ↓               ↓
 RGB + DEPTH       POINTS
       └──────┬────────┘
              ↓
        PREPROCESSING
              ↓
     3D REPRESENTATION
   ┌──────────┼──────────┐
 POINTS     VOXELS      FIELDS
   │           │           │
 normals    occupancy    SDF
   │           │         NeRF
   └──────────┴───────────┘
              ↓
        REGISTRATION
              ↓
         WORLD MODEL
              ↓
  GEOMETRY + SEMANTICS
  + MOTION + UNCERTAINTY
              ↓
          PLANNING
              ↓
           ACTION

That is modern robotic perception in one picture.

139. What a superior robotics engineer understands

A beginner sees:

a point cloud

A strong engineer sees:

measurements
coordinate frames
timestamps
noise
sampling density
occlusion
calibration
sensor physics
uncertainty
representation choices

A beginner sees:

ICP returned a transform

A strong engineer asks:

Was initialization good?
How much overlap existed?
Was geometry degenerate?
Were dynamic objects removed?
What were residuals?
How certain is this transform?

A beginner sees:

beautiful NeRF reconstruction

A strong engineer asks:

How accurate is metric geometry?
Where is free space?
How are unseen regions treated?
Can I query collision distance?
What happens when objects move?

A beginner sees:

AI 3D model

A strong engineer asks:

What representation?
What coordinate system?
What supervision?
What information is explicit?
What is inferred?
What downstream queries are efficient?

That difference is engineering maturity.

140. The final principle

3D perception is ultimately about building an internal model of physical reality from incomplete, noisy observations.

The robot never sees the entire truth.

It receives pieces:

a laser return
a depth pixel
an RGB image
an IMU reading
another scan
another frame

and gradually constructs:

something is here

nothing was observed here

this surface faces this way

this scan overlaps the previous scan

this object is moving

this region is safe

this region is unknown

this wall is 42 cm away

this mug can be grasped

That is the real meaning of perception.

And the final lesson to keep forever is:

Sensors do not give the robot a world. They give it evidence.

3D perception is the machinery that turns that evidence into a useful belief about the world.

And autonomy begins when that belief becomes good enough to act on.

Command Palette

From pixels and laser returns to a robot that understands physical space

1. The master mental model

2. The most important distinction: measurement vs representation

Senior-engineer rule

3. Point clouds

4. Think of a point cloud as 3D graph paper without the paper

5. Where point clouds come from

LiDAR

RGB-D camera

Stereo vision

Structure from Motion

Multi-view stereo

6. Depth maps

7. Depth map intuition

8. Depth map → 3D point

9. Depth map vs point cloud

Depth map

Point cloud

10. RGB-D

11. RGB-D example

12. RGB-D alignment matters

13. LiDAR data

14. LiDAR does not fundamentally “see images”

15. Why robotics loves LiDAR

16. One subtle LiDAR problem: the scan is not instantaneous

17. Filtering

18. Pass-through filtering

19. Range filtering

20. Statistical outlier removal

21. Radius outlier filtering

22. Ground filtering

23. Voxelization

24. Why voxelize a point cloud?

25. Voxelization performs two jobs

Compression

Regularization

26. The voxel-resolution tradeoff

27. Why dense voxel grids become expensive

28. KD-trees

29. KD-tree intuition

30. Where KD-trees appear

31. Surface normals

32. Normal intuition

33. How do you estimate a normal from points?

34. Why normals matter

Surface classification

Registration

Mesh reconstruction

Grasping

Traversability

Rendering

35. Registration

36. Registration in plain English

37. Why registration is fundamental

38. Registration has two broad stages

Coarse registration

Fine registration

39. ICP — Iterative Closest Point

40. ICP algorithm

Step 1 — Find correspondences

Step 2 — Estimate transform

Step 3 — Transform B

Step 4 — Repeat

41. ICP's biggest weakness

42. Point-to-point ICP

43. Point-to-plane ICP

44. ICP engineering problems

Partial overlap

Outliers

Dynamic objects

Repetitive geometry

Different point densities

Poor initial pose

Degenerate scenes

45. Occupancy maps

46. Occupied and free space are different pieces of information

47. Unknown is not free

48. Probabilistic occupancy

49. Log-odds occupancy