Master Computer Vision: From Pixels → Geometry → Meaning → 3D Understanding
Computer vision looks enormous when learned as separate algorithms:
camera calibration
SIFT
stereo
optical flow
YOLO
segmentation
PnP
RANSAC
homographies
SLAM
depth estimation
tracking
But underneath, almost everything is trying to answer a small number of questions:
What produced these pixels?
Where is the camera?
Where are objects in the world?
What are those objects?
How are they moving?
That is Computer Vision.
0. The Mental Model You Should Never Forget
A camera sees this:
3D WORLD
↓
light
↓
camera lens
↓
2D IMAGE
↓
pixels
Computer vision tries to run this process partially backward:
pixels
↓
edges / features / learned representations
↓
objects / correspondences
↓
geometry
↓
depth / pose / motion
↓
understanding of the 3D world
The fundamental difficulty is immediately visible.
A camera converts:
3D → 2D
Computer vision often wants:
2D → 3D
But information was lost during projection.
That single fact explains a huge amount of computer vision.
For example, suppose you see this photograph:
□
Is that:
a tiny square close to the camera?
a huge square far away?
the face of a cube?
a rectangular sign viewed from an angle?
One image alone often cannot tell you.
So computer vision combines:
geometry,
multiple images,
camera motion,
assumptions about the world,
learned patterns,
temporal information.
PART I — HOW A CAMERA CREATES AN IMAGE
1. Image Formation
Before trying to understand an image, understand how the image came into existence.
An object does not inherently "send pixels."
Light interacts with the world.
Very roughly:
light source
↓
object
↓ reflected light
camera
↓
sensor
↓
pixels
A pixel intensity depends on many things:
illumination
surface reflectance
surface orientation
distance
camera exposure
lens characteristics
sensor response
This matters because:
A change in pixel value does not necessarily mean a change in the object.
For example, the same white wall can look:
bright white under sunlight,
gray indoors,
orange during sunset,
blue under certain artificial lights.
The object stayed the same.
The image changed.
This is why computer vision is fundamentally harder than simply reading numbers from an image.
2. The Pinhole Camera Model
The most important camera model in computer vision is surprisingly simple.
Imagine a box with a tiny hole.
object
↑
|
|
|\
| \
---------O--\---------------- optical axis
| \
| \
| ↓
image plane
Light rays pass through one point:
camera center
and hit the image plane.
This is called the pinhole camera model.
Real cameras contain complicated lenses, but the pinhole model gives us the mathematical foundation for nearly all camera geometry.
2.1 Perspective Projection
Suppose a 3D point in camera coordinates is:
P = (X, Y, Z)
The image point is:
p = (x, y)
Then:
x=fXZx = f \frac{X}{Z}y=fYZy = f \frac{Y}{Z}
where:
f = focal length
This equation is incredibly important.
Notice the division by:
Z
That is where perspective comes from.
2.2 Why Faraway Objects Look Smaller
Take an object with:
X = 2
If:
Z = 2
then:
x=fx = f
But if:
Z = 20
then:
x=0.1fx = 0.1f
Same physical size.
Ten times farther away.
Ten times smaller in the image.
That is perspective projection.
3. Camera Coordinate Systems
Computer vision uses several coordinate systems.
This can initially feel annoying, but it becomes simple if you remember:
Every coordinate system is just answering: "Where is this point measured from?"
Typical systems:
World coordinates
↓
Camera coordinates
↓
Normalized image coordinates
↓
Pixel coordinates
4. Extrinsic Parameters
Suppose a robot sees a chair.
The chair may have world coordinates:
(4.2 m, 1.7 m, 0 m)
But the camera needs to know where that chair lies relative to itself.
That transformation is controlled by the camera's extrinsic parameters.
Extrinsics describe:
camera position
+
camera orientation
Mathematically:
Pc=RPw+tP_c = RP_w + t
where:
PwP_w = world point
PcP_c = point expressed in camera coordinates
RR = rotation matrix
tt = translation
So:
WORLD
↓
[R | t]
↓
CAMERA COORDINATES
Never-forget interpretation
Intrinsics
Describe the camera itself.
"How does this camera convert camera-space rays into pixels?"
Extrinsics
Describe where the camera is.
"Where is the camera relative to the world?"
This distinction is foundational.
5. Intrinsic Parameters
Once the point is expressed in camera coordinates, we need to convert the projected position into actual pixels.
A typical camera intrinsic matrix is:
K=[fx0cx0fycy001]K = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}
The major parameters are:
fx,fyf_x, f_y
Effective focal length measured in pixels.
They control image scaling.
cx,cyc_x, c_y
The principal point.
Usually somewhere near the center of the image.
For a 1920×1080 image it might approximately be:
cx ≈ 960
cy ≈ 540
but not necessarily exactly.
Intrinsics vs extrinsics
Memorize this:
INTRINSICS
"What camera do I have?"
fx
fy
cx
cy
distortion
EXTRINSICS
"Where is the camera?"
rotation
translation
6. The Complete Camera Projection
Now combine everything.
We begin with a 3D world point:
PwP_w
First transform world → camera:
Pc=[R∣t]PwP_c = [R|t]P_w
Then camera → pixel:
p∼K[R∣t]Pwp \sim K[R|t]P_w
The famous camera equation is therefore:
p∼K[R∣t]P\boxed{p \sim K[R|t]P}
Conceptually:
3D world point
↓
extrinsics [R|t]
↓
camera coordinates
↓
perspective projection
↓
intrinsics K
↓
image pixel
This equation appears everywhere:
calibration
SLAM
augmented reality
PnP
stereo vision
structure from motion
visual odometry
robotics
7. Camera Calibration
Suppose OpenCV gives you a pixel:
u = 842
v = 317
What direction in the real world does that pixel represent?
You cannot answer accurately unless you know the camera's intrinsic parameters and distortion.
Calibration estimates those parameters.
7.1 Chessboard Calibration
The classical method uses a pattern whose geometry is known:
■ □ ■ □ ■ □
□ ■ □ ■ □ ■
■ □ ■ □ ■ □
□ ■ □ ■ □ ■
You already know the real-world distances between corners.
The camera observes where those corners appear in the image.
Calibration then solves:
"What camera parameters best explain these observations?"
You normally photograph the calibration board:
from different distances,
different angles,
different image locations.
The output might contain:
fx
fy
cx
cy
k1
k2
k3
p1
p2
The first group describes camera intrinsics.
The second describes distortion.
8. Lens Distortion
The pinhole model assumes perfect geometry.
Real lenses are imperfect.
Straight lines may bend near image boundaries.
You have probably seen wide-angle camera distortion:
ideal:
| | | |
| | | |
| | | |
distorted:
) ) | ( (
Two important kinds exist.
8.1 Radial Distortion
Pixels are displaced radially outward or inward.
Typical coefficients:
k1
k2
k3
Barrel distortion
Lines bulge outward.
Common in wide-angle lenses.
Pincushion distortion
Lines bend inward.
8.2 Tangential Distortion
Occurs when the lens and image sensor are not perfectly aligned.
Typical coefficients:
p1
p2
Why distortion correction matters
Imagine trying to perform stereo geometry while the camera bends straight rays incorrectly.
Your geometry becomes inaccurate.
So many pipelines begin with:
raw image
↓
undistortion
↓
geometry
PART II — THE MATHEMATICAL LANGUAGE OF VISION
9. Homogeneous Coordinates
This concept looks abstract until you understand why engineers use it.
Normally a 2D point is:
(x,y)(x,y)
In homogeneous coordinates we write:
(x,y,1)(x,y,1)
A 3D point becomes:
(X,Y,Z,1)(X,Y,Z,1)
Why add a useless-looking 1?
Because it lets us represent:
translation
rotation
projection
perspective transformation
using matrix multiplication.
Example: translation
Without homogeneous coordinates:
x′=x+txx' = x + t_xy′=y+tyy' = y + t_y
Addition is required.
With homogeneous coordinates:
[x′y′1]=[10tx01ty001][xy1]\begin{bmatrix} x'\\ y'\\ 1 \end{bmatrix} = \begin{bmatrix} 1&0&t_x\\ 0&1&t_y\\ 0&0&1 \end{bmatrix} \begin{bmatrix} x\\ y\\ 1 \end{bmatrix}
Now translation becomes matrix multiplication.
That allows entire transformation chains to become:
matrix × matrix × matrix × point
Extremely useful.
9.1 Scale Equivalence
In homogeneous coordinates:
(x,y,1)(x,y,1)
and:
(2x,2y,2)(2x,2y,2)
represent the same point.
To recover ordinary coordinates:
x=X/Wx = X/Wy=Y/Wy = Y/W
This idea makes perspective projection mathematically elegant.
10. Projective Geometry
Euclidean geometry describes normal measurements such as:
distance,
angles,
parallel lines.
But a camera does not preserve all of those.
For example:
railway tracks in reality:
| |
| |
| |
| |
image:
\ /
\ /
\ /
\ /
.
Parallel railway tracks appear to meet.
That meeting point is a:
vanishing point
Projective geometry describes geometry under perspective projection.
It is the natural geometry of cameras.
11. Homographies
A homography is a transformation between two images of the same plane.
Mathematically:
p′∼Hpp' \sim Hp
where HH is a 3×3 matrix.
Think of photographing a poster
You photograph a rectangular poster at an angle:
real poster:
+-------------+
| |
| |
+-------------+
camera image:
/---------/
/ /
/---------/
A homography can transform the slanted quadrilateral back into a rectangle.
Homographies are useful for
panorama stitching
document scanning
perspective correction
bird's-eye-view transformation
planar AR markers
sports-field transformation
image registration
When does a homography work?
Most importantly when:
Case 1
The observed points lie approximately on one plane.
or:
Case 2
The camera rotates without translating significantly.
Important limitation
A homography cannot generally describe arbitrary 3D scenes under camera translation.
Why?
Because objects at different depths move differently.
That phenomenon is called:
parallax
PART III — TWO CAMERAS REVEAL DEPTH
12. Epipolar Geometry
This is one of the most important ideas in stereo vision.
Suppose two cameras see the same 3D point.
P
/ \
/ \
/ \
C1 C2
The cameras and point define a plane.
That plane intersects each image in an:
epipolar line
Why this matters
Suppose you identify point p1p_1 in camera 1.
Where should you search for its corresponding pixel in camera 2?
Without geometry:
search the entire image
With epipolar geometry:
search along one line
Huge simplification.
13. Essential and Fundamental Matrices
Two matrices describe epipolar relationships.
Essential matrix EE
Used when camera intrinsics are known and points are expressed in normalized camera coordinates.
It contains information about:
relative rotation
+
relative translation direction
between cameras.
Fundamental matrix FF
Works directly with pixel coordinates.
The core relationship is:
x2TFx1=0x_2^T F x_1 = 0
Interpretation:
If x1x_1 is a pixel in image 1, Fx1F x_1 gives the corresponding epipolar line in image 2.
You don't need to stare at the equation.
Remember what it does:
point in image A
↓
epipolar geometry
↓
line in image B where matching point must lie
14. Stereo Vision
Humans have two eyes for a reason.
Close one eye.
Estimating depth becomes harder.
Open both.
Your brain uses the difference between what each eye sees.
Stereo cameras do the same thing.
Left camera Right camera
C_L ---------------- C_R
baseline
A nearby object shifts significantly between the images.
A faraway object shifts only slightly.
That shift is:
disparity
15. Disparity and Depth
For rectified stereo cameras:
Z=fBdZ = \frac{fB}{d}
where:
ZZ = depth
ff = focal length
BB = baseline distance between cameras
dd = disparity
This equation gives a beautiful intuition:
Z∝1dZ \propto \frac{1}{d}
Meaning:
large disparity → close object
small disparity → far object
Example
Suppose:
f = 800 pixels
B = 0.1 m
d = 40 pixels
Then:
Z=800×0.140Z = \frac{800 \times 0.1}{40}Z=2mZ = 2m
So the object is approximately two meters away.
16. Stereo Rectification
Epipolar lines can originally appear at arbitrary angles.
Stereo rectification warps the two images so corresponding pixels lie on the same horizontal row.
Before:
Image L Image R
* \
*
After rectification:
Image L Image R
---*-------------*---
Now matching becomes:
search left/right
rather than:
search arbitrary 2D regions
This makes stereo computation much easier.
17. Triangulation
Suppose two cameras observe the same point.
Each pixel corresponds to a ray leaving a camera.
P
/ \
/ \
/ \
/ \
C1 C2
If we know:
camera poses,
camera intrinsics,
corresponding image points,
we can estimate where the rays intersect in 3D.
That process is:
triangulation
Conceptually:
pixel in camera 1 → 3D ray
pixel in camera 2 → 3D ray
intersection of rays
↓
estimated 3D point
Due to noise, rays rarely intersect perfectly.
Real systems find the 3D point that best fits them.
PART IV — FINDING THINGS THAT CAN BE MATCHED
18. Feature Detection
Suppose a robot takes these two images:
Frame 1
Frame 2
You want to determine how the camera moved.
One approach:
Find recognizable points in frame 1 and find those same points in frame 2.
But which pixels should we track?
A giant white wall is terrible.
...............
...............
...............
...............
Every pixel looks similar.
An edge is somewhat better:
......|########
......|########
......|########
But if the point moves vertically along the edge, it is difficult to tell where it went.
A corner is much better:
........
....####
....####
....####
The pixel neighborhood changes strongly in multiple directions.
Therefore good features are often:
corners
blobs
distinctive textured regions
19. Famous Feature Detectors
Classical feature detectors include:
Harris Corner
Shi-Tomasi
FAST
DoG
SIFT
SURF
ORB
Different detectors optimize different things:
repeatability
speed
rotation invariance
scale invariance
robustness to lighting
20. Feature Descriptor
Detection answers:
"Where is an interesting point?"
Descriptor answers:
"What does the neighborhood around this point look like?"
Suppose the detector finds:
(x = 423, y = 207)
A descriptor converts the local appearance around it into a vector.
For example:
[0.18, 0.02, 0.73, ...]
Now another image contains another feature vector.
We compare them.
Similar vectors probably represent the same physical location.
Detector vs descriptor
Never confuse them.
DETECTOR
Where should I look?
DESCRIPTOR
What does it look like?
Some algorithms provide both.
For example:
SIFT
ORB
21. SIFT
SIFT stands for:
Scale-Invariant Feature Transform
It became famous because its features are relatively robust to:
scale change
rotation
moderate viewpoint change
lighting change
It produces a floating-point descriptor.
Historically, SIFT became one of the foundational techniques for:
image matching
panorama stitching
localization
structure from motion
22. ORB
ORB is often chosen when speed matters.
It combines techniques related to:
FAST detector
+
BRIEF-style descriptor
Its descriptor is binary.
Instead of expensive floating point distances, descriptors can be compared using:
Hamming distance
That makes ORB attractive for real-time robotics.
ORB was famously used in systems such as:
ORB-SLAM
23. Feature Matching
Suppose image A contains descriptor:
A1
and image B contains:
B1
B2
B3
B4
...
We find which descriptor is closest.
Common distance metrics:
Euclidean distance
Often used for floating-point descriptors such as SIFT.
Hamming distance
Often used for binary descriptors such as ORB.
24. Why Nearest Match Is Not Enough
Suppose descriptor A matches:
B1 distance = 20
B2 distance = 21
That is ambiguous.
But:
B1 distance = 20
B2 distance = 92
is much more convincing.
This motivates techniques such as the:
nearest-neighbor ratio test
commonly associated with SIFT matching.
25. Bad Matches Are Inevitable
Even excellent descriptors produce incorrect matches.
Imagine windows on a building:
[] [] [] [] []
[] [] [] [] []
[] [] [] [] []
They all look similar.
A feature from window 3 might match window 7.
Therefore vision systems need robust estimation.
Enter one of the most useful algorithms in computer vision:
RANSAC
26. RANSAC
RANSAC stands for:
RANdom SAmple Consensus
Its job is:
Estimate a model even when your data contains many incorrect observations.
Suppose you have 100 feature matches.
75 correct
25 wrong
You want to estimate a homography.
If you fit everything blindly, the bad matches may ruin the result.
RANSAC does roughly this:
1. Randomly select a small subset.
2. Fit a candidate model.
3. Ask how many observations agree with it.
4. Repeat.
5. Keep the model with the strongest support.
Intuition
Imagine these points:
*
*
*
------------------------
* * * * * * ** ** *
------------------------
*
*
Most lie near one line.
Some are random garbage.
Ordinary fitting can be pulled by garbage.
RANSAC tries many hypotheses until it discovers:
"Ah. Most points support THIS structure."
Inliers and outliers
Observations supporting the model:
inliers
Observations inconsistent with it:
outliers
RANSAC is everywhere:
homography estimation
fundamental matrix estimation
essential matrix estimation
PnP
plane fitting
visual odometry
SLAM
PART V — MOTION
27. Optical Flow
Feature matching can compare distinct images.
But sometimes we want to estimate pixel motion between nearby video frames.
That is:
optical flow
Suppose:
Frame t:
●
Frame t+1:
●
Optical flow estimates something approximately like:
motion = (+4 px, +1 px)
28. Sparse vs Dense Optical Flow
Sparse optical flow
Tracks selected points.
Example:
corners → track them over time
Lucas-Kanade is a classic example.
Useful for:
feature tracking
visual odometry
stabilization
Dense optical flow
Estimates motion for nearly every pixel.
Produces something like:
pixel → velocity vector
Useful for:
motion understanding
video analysis
dynamic-scene reasoning
motion segmentation
Modern deep-learning methods can estimate very powerful dense optical flow.
29. The Optical Flow Assumption
A classic assumption is:
The same physical point has approximately the same brightness between nearby frames.
So:
I(x, y, t)
≈
I(x + dx, y + dy, t + dt)
This is called:
brightness constancy
It is not perfectly true, but useful.
30. The Aperture Problem
Imagine seeing only this edge through a tiny window:
/////
/////
/////
If the edge moves along itself, determining the motion is difficult.
You can reliably estimate motion mostly perpendicular to the edge.
This ambiguity is the:
aperture problem
Corners are better because they provide structure in multiple directions.
Again we discover why corners are powerful.
PART VI — CAMERA POSE
31. Pose Estimation
Pose means:
position
+
orientation
Usually represented through:
translation t
rotation R
For a robot camera, pose answers:
Where is the camera, and which direction is it facing?
This is central to:
robotics
AR
visual odometry
SLAM
autonomous vehicles
32. Pose from 2D–3D Correspondences
Suppose you know these world points:
3D landmark A
3D landmark B
3D landmark C
...
And your camera observes them at:
pixel a
pixel b
pixel c
...
Then you can estimate the camera pose that best explains the mapping:
3D world point → observed 2D pixel
This problem is called:
PnP
33. PnP — Perspective-n-Point
Given:
known 3D points
+
their observed 2D image locations
+
camera intrinsics
estimate:
camera rotation
camera translation
This is Perspective-n-Point.
Conceptually:
3D landmarks
P1 P2 P3 P4
↓ ↓ ↓ ↓
image observations
p1 p2 p3 p4
↓
PnP
↓
camera pose
Example: AR marker
Suppose a square marker has known physical geometry.
You detect its four corners.
You know:
corner 1 = (0,0,0)
corner 2 = (1,0,0)
corner 3 = (1,1,0)
corner 4 = (0,1,0)
From their image locations, PnP can estimate where the camera lies relative to the marker.
Then an AR system can correctly draw a virtual cube on top of it.
34. PnP + RANSAC
Real feature correspondences contain bad matches.
Therefore practical systems commonly perform something conceptually like:
2D–3D correspondences
↓
PnP + RANSAC
↓
reject bad matches
↓
robust camera pose
This combination is enormously important in localization systems.
PART VII — FROM PIXELS TO SEMANTIC UNDERSTANDING
Until now, much of our vision discussion was geometric.
But modern computer vision also needs to answer:
What is in the image?
This introduces three related but different problems.
35. Image Classification
Given the entire image:
[ IMAGE ]
predict:
cat
or:
dog
Classification answers:
What is this image mostly about?
It does not normally say where the object is.
36. Object Detection
Detection answers:
What objects exist, and where are they?
Example:
+-------------------------------+
| |
| +-------+ |
| | person| +------+ |
| +-------+ | car | |
| +------+ |
| |
+-------------------------------+
Output might be:
person:
bbox = [x1,y1,x2,y2]
confidence = 0.97
car:
bbox = [...]
confidence = 0.94
Popular detector families have included systems such as:
Faster R-CNN
SSD
YOLO
DETR-style detectors
The exact architecture changes over time.
The underlying task does not.
37. Bounding Boxes
A bounding box can be represented as:
x_min
y_min
x_max
y_max
or:
center_x
center_y
width
height
Always inspect the expected representation when working with APIs.
Mistaking one convention for another causes many practical bugs.
38. IoU — Intersection over Union
How do we measure whether a predicted box matches the true box?
Use:
Intersection over Union
Conceptually:
IoU=overlap areacombined areaIoU = \frac{\text{overlap area}} {\text{combined area}}
If boxes overlap perfectly:
IoU = 1
No overlap:
IoU = 0
IoU appears everywhere in detection and segmentation evaluation.
39. Non-Maximum Suppression
A detector may produce:
person 0.97
person 0.92
person 0.88
all around the same human.
We usually want one detection, not three.
Non-Maximum Suppression, or NMS, roughly performs:
1. Keep strongest box.
2. Remove weaker boxes that overlap it heavily.
3. Continue.
Modern detector designs may handle duplicate predictions differently, but the idea is foundational.
40. Segmentation
Detection gives boxes.
Segmentation gives pixel-level regions.
Instead of:
[ rectangle around person ]
you obtain:
exact pixels belonging to person
41. Semantic Segmentation
Every pixel receives a class:
road
road
car
car
person
sky
building
...
But two people can receive the same label:
person
without distinguishing which person is which.
Useful in:
road understanding
agriculture
medical imaging
scene understanding
robotics
42. Instance Segmentation
Instance segmentation distinguishes individual objects.
Instead of:
person pixels
you get:
person #1
person #2
person #3
This combines:
object detection
+
pixel masks
43. Panoptic Segmentation
Panoptic segmentation attempts to describe essentially everything.
It combines:
"things"
+
"stuff"
Things:
person
car
dog
Stuff:
road
sky
grass
wall
So a scene can receive a comprehensive pixel-level interpretation.
PART VIII — TRACKING
44. Object Tracking
Detection answers:
Where is the person in this frame?
Tracking answers:
Is this the same person across frames?
For example:
Frame 1:
Person ID 7
Frame 2:
Person ID 7
Frame 3:
Person ID 7
Tracking assigns temporal identity.
45. Detection vs Tracking
Do not confuse them.
DETECTION
"What objects are present now?"
TRACKING
"Which current object corresponds to which previous object?"
A common system is:
video
↓
detector
↓
detections
↓
motion prediction
+
appearance matching
↓
track association
↓
persistent IDs
46. Tracking-by-Detection
Many modern tracking systems repeatedly run an object detector.
Frame 1:
detections:
A B C
Frame 2:
detections:
D E F
The tracker decides:
A → D
B → F
C → E
This is called:
data association
47. Kalman Filter in Tracking
Suppose a car was moving right.
The next frame temporarily fails to detect it.
A motion model can predict:
Based on previous position and velocity, the car should probably be here.
Kalman filters are often used for this type of state estimation.
Conceptually:
previous state
+
motion model
↓
predicted state
prediction
+
measurement
↓
updated state
This connects computer vision with estimation theory.
48. Appearance Embeddings
Motion alone may fail when objects cross paths.
Suppose:
person A
person B
walk past each other.
Appearance models can encode each person's visual characteristics into vectors.
Then tracking can use both:
motion similarity
+
appearance similarity
for data association.
PART IX — DEPTH
49. Depth Estimation
Depth answers:
How far is each visible point from the camera?
A depth image might look conceptually like:
pixel → distance
For example:
chair pixel = 2.4 m
wall pixel = 5.8 m
person = 1.7 m
Depth is enormously important for physical AI.
50. Ways to Obtain Depth
There are several fundamentally different approaches.
Stereo cameras
Use geometry and disparity.
RGB-D cameras
Hardware directly estimates depth using technologies such as structured light or active sensing.
LiDAR
Measures distance using laser returns.
Monocular depth estimation
Predicts depth from one RGB image using learned visual priors.
Multi-view geometry
Uses multiple camera observations and camera movement.
51. Metric vs Relative Depth
This distinction matters.
A monocular model might correctly understand:
person closer than wall
but not know the exact distances.
It may output:
person depth = 0.3
wall depth = 0.8
These can be useful relative values without representing meters.
Metric depth attempts to provide:
person = 1.8 m
wall = 5.2 m
Never assume a model outputs metric distance merely because it outputs a "depth map."
52. Why Monocular Depth Is Hard
From one photograph:
small nearby object
and:
large distant object
can create similar image sizes.
Geometrically the problem is ambiguous.
Humans resolve it using learned knowledge:
cars have typical sizes
doors have typical sizes
floors have perspective structure
objects occlude other objects
shadows provide cues
Deep monocular depth networks learn similar statistical cues from data.
PART X — 3D RECONSTRUCTION
53. 3D Reconstruction
Now combine the ideas.
Suppose you walk around a statue taking photographs.
Each photograph sees different portions of the statue.
If we know:
which image points correspond
+
where each camera was
we can triangulate many points.
Eventually:
images
↓
features
↓
matching
↓
camera poses
↓
triangulation
↓
3D points
That is the basic spirit of 3D reconstruction.
54. Structure from Motion — SfM
Suppose neither camera poses nor 3D structure are known.
You have only photographs.
Structure from Motion attempts to recover:
camera motion
+
3D scene structure
from those images.
A simplified pipeline:
Images
↓
Detect features
↓
Describe features
↓
Match across images
↓
Estimate geometric relationships
↓
Recover relative camera poses
↓
Triangulate points
↓
Add more cameras
↓
Optimize everything
The output initially may be a:
sparse point cloud
55. Sparse Reconstruction
Feature-based methods reconstruct distinctive locations:
. .
. .
. . .
. .
Only some points exist.
This is sparse reconstruction.
It is enough for tasks such as:
camera localization
geometry recovery
visual mapping
56. Dense Reconstruction
Dense reconstruction tries to estimate much more of the visible surface.
sparse:
. . .
.
.
dense:
################
################
################
################
Multi-view stereo is one family of techniques for turning calibrated multi-view images into denser 3D structure.
57. Point Clouds
A point cloud is simply a collection of 3D points:
P1 = (x1,y1,z1)
P2 = (x2,y2,z2)
P3 = (x3,y3,z3)
...
Points may also contain:
RGB color
surface normal
confidence
semantic label
LiDAR, RGB-D cameras, stereo systems, and reconstruction pipelines often produce point clouds.
58. Meshes
Point clouds are just disconnected samples.
A mesh connects points into surfaces.
Typically using triangles:
•
/ \
•---•
A large collection of triangles approximates a 3D object's surface.
Meshes are useful for:
simulation
rendering
measurement
digital twins
AR/VR
robotics
59. Texture
Geometry tells you shape.
Texture tells you appearance.
A reconstructed object may contain:
mesh
+
images projected onto mesh
producing a realistic 3D model.
PART XI — THE CONNECTION BETWEEN EVERYTHING
At this point the topics stop looking independent.
Consider an autonomous robot.
It receives camera frames.
CAMERA
↓
raw pixels
First, camera geometry:
calibration
intrinsics
distortion correction
Then geometric information:
feature detection
descriptors
matching
optical flow
Then robust estimation:
RANSAC
essential matrix
PnP
Then pose:
Where am I?
Then depth:
stereo
triangulation
depth network
RGB-D
Then semantic understanding:
detection
segmentation
Then temporal reasoning:
tracking
Then world representation:
3D map
objects
free space
obstacles
Then planning:
Where should the robot go?
Computer vision therefore sits directly underneath robotics autonomy.
PART XII — A COMPLETE EXAMPLE
Imagine a mobile robot sees a red chair.
Let's follow the entire stack.
Stage 1 — Camera capture
The physical chair reflects light.
The camera sensor produces:
1920 × 1080 RGB image
Stage 2 — Calibration correction
The camera knows:
K
distortion coefficients
The frame may be undistorted.
Stage 3 — Detection
A neural detector returns:
class = chair
confidence = 0.96
bbox = [...]
Now we know:
There is probably a chair here.
But not necessarily its physical distance.
Stage 4 — Segmentation
A segmentation model may identify exactly which pixels belong to the chair.
chair mask
Better than the rough bounding box.
Stage 5 — Depth
Suppose an RGB-D camera reports:
chair center depth = 2.4 m
Now the robot knows approximate 3D location relative to the camera.
Stage 6 — Camera pose
Visual odometry / SLAM estimates:
robot camera pose in map
Now the chair's camera-relative position can be transformed into world coordinates.
Stage 7 — Tracking
Next frame:
chair appears elsewhere in image
The tracking system determines:
same chair
Stage 8 — Mapping
The robot can maintain something like:
Object #23
class = chair
world position = ...
Now pixels have become persistent world knowledge.
That is where computer vision becomes machine perception.
PART XIII — CLASSICAL CV VS DEEP CV
Modern engineers need both mental models.
60. Classical Computer Vision
Classical vision explicitly designs operations such as:
corners
edges
SIFT
ORB
optical flow
homographies
epipolar geometry
RANSAC
PnP
triangulation
Advantages:
geometry is interpretable
often computationally efficient
can require little training data
strong mathematical guarantees exist in some settings
Weaknesses:
brittle under difficult appearance conditions
manually engineered representations may struggle with semantic complexity
61. Deep Computer Vision
Deep learning learns representations directly from data.
Typical flow:
pixels
↓
neural network
↓
learned feature representation
↓
prediction
Tasks include:
classification
detection
segmentation
depth
optical flow
pose
tracking
feature matching
Modern neural networks have replaced or complemented classical algorithms in many areas.
But geometry has not become obsolete.
Far from it.
62. Geometry + Learning Is Extremely Powerful
A weak mental model is:
old CV = geometry
new CV = neural networks
A better model is:
physical world
↓
GEOMETRY
+
LEARNING
+
OPTIMIZATION
+
ESTIMATION
↓
robust perception
For autonomous systems, you want to understand both.
For example:
neural network
↓
feature correspondences
↓
RANSAC
↓
essential matrix
↓
camera pose
or:
neural detector
↓
object pixel coordinates
+
depth camera
+
camera calibration
↓
object 3D position
This hybrid thinking is what makes a strong robotics / AI engineer.
PART XIV — SOME IMPORTANT CONNECTIONS ENGINEERS OFTEN MISS
63. Pixels Are Not Directions Until You Use Camera Intrinsics
Suppose YOLO detects something at:
pixel (900, 400)
That does not directly tell you a physical direction.
Using camera intrinsics:
pixel
↓
K⁻¹
↓
normalized camera ray
you can determine the direction leaving the camera center.
If depth is known:
ray × depth
gives a 3D point in camera coordinates.
This is one of the most important bridges between:
AI detection
and:
robotics geometry
64. Bounding Boxes Do Not Give True Object Position
Suppose YOLO gives:
person bbox center = (740, 430)
You know where the person lies in the image.
You do not automatically know:
X = ?
Y = ?
Z = ?
To recover real-world position you need something more:
depth,
stereo,
known object geometry,
multi-view observations,
ground-plane assumptions,
another ranging sensor.
This distinction prevents many beginner robotics mistakes.
65. Depth Alone Is Usually Camera-Relative
Suppose:
depth = 3 m
That says:
The point is 3 m relative to the camera's geometry.
To express it in the robot/world frame:
camera point
↓
camera-to-robot transform
↓
robot coordinates
↓
robot-to-map transform
↓
map coordinates
In ROS this is exactly why transformation systems such as:
TF / TF2
are fundamental.
66. Calibration Error Becomes 3D Error
If:
focal length estimate is wrong
or:
stereo cameras are poorly calibrated
then estimated rays are wrong.
If rays are wrong:
triangulation is wrong
If triangulation is wrong:
depth is wrong
If depth is wrong:
mapping and navigation can be wrong
Calibration is therefore not a boring setup step.
It affects the entire geometric stack.
67. Correspondence Is One of the Central Problems of Vision
Many apparently different problems reduce to:
Which observation here corresponds to which observation there?
Examples:
feature matching:
point in image A ↔ point in image B
stereo:
left pixel ↔ right pixel
optical flow:
pixel at t ↔ pixel at t+1
tracking:
object in frame t ↔ object in frame t+1
3D reconstruction:
image feature ↔ same physical world point
Correspondence is everywhere.
Once you notice this, computer vision becomes much more unified.
PART XV — HOW THE MAJOR CONCEPTS RELATE
Here is the master dependency map.
COMPUTER VISION
│
┌────────────────┴────────────────┐
│ │
GEOMETRY SEMANTICS
│ │
camera formation classification
│ detection
pinhole camera segmentation
│ tracking
intrinsics/extrinsics
│
calibration
│
projective geometry
│
┌──────┴───────┐
│ │
homographies multi-view
│
epipolar geometry
│
stereo
│
triangulation
│
3D points
A second path:
images
↓
feature detection
↓
descriptors
↓
matching
↓
RANSAC
↓
geometry
↓
camera pose
↓
3D reconstruction
And temporal vision:
video
↓
features / detections
↓
optical flow / association
↓
tracking
↓
motion understanding
PART XVI — RAPID "WHEN DO I USE WHAT?" GUIDE
Pinhole camera model
Use when:
reasoning about projection from 3D → image
Think:
How does the camera see the world?
Intrinsics
Use when:
pixel ↔ camera ray conversion
Think:
What are the internal geometric properties of this camera?
Extrinsics
Use when:
world frame ↔ camera frame
Think:
Where is the camera?
Calibration
Use when:
intrinsics/distortion/extrinsics need estimation
Think:
Determine the geometry of my real camera.
Homogeneous coordinates
Use when:
transformations and projection need one unified matrix language
Think:
Add one dimension so transformations become elegant matrix operations.
Homography
Use when:
mapping one planar view to another
Think:
Flatten this poster/document/floor/plane.
Epipolar geometry
Use when:
two cameras observe the same 3D scene
Think:
A point here restricts where its match can appear there.
Stereo vision
Use when:
depth from two cameras
Think:
Difference between the left and right view reveals distance.
Triangulation
Use when:
multiple rays observe the same point
Think:
Where in 3D do these observations intersect?
Feature detector
Use when:
finding stable interesting image locations
Think:
Where should I look?
Descriptor
Use when:
matching interesting locations
Think:
What does this neighborhood look like?
Feature matching
Use when:
finding correspondences between images
Think:
Is this the same physical point?
Optical flow
Use when:
estimating image motion across nearby frames
Think:
Where did these pixels move?
RANSAC
Use when:
fitting geometry despite bad observations
Think:
Find the model supported by the trustworthy majority.
Pose estimation
Use when:
determining camera/object position and orientation
Think:
Where am I?
PnP
Use when:
known 3D landmarks correspond to observed 2D pixels
Think:
Which camera pose would create these image observations?
Detection
Use when:
objects + approximate locations
Think:
What objects are here, and approximately where?
Segmentation
Use when:
pixel-level understanding
Think:
Which exact pixels belong to what?
Tracking
Use when:
maintaining identities over time
Think:
Is this the same object as before?
Depth estimation
Use when:
distance from camera is required
Think:
How far away is every visible point?
3D reconstruction
Use when:
building geometry from observations
Think:
Turn many 2D views into a 3D world.
PART XVII — THE EQUATIONS ACTUALLY WORTH REMEMBERING
You do not need to memorize pages of equations.
Keep these mental anchors.
Perspective
x=fX/Zx = fX/Zy=fY/Zy = fY/Z
Meaning:
Divide by depth.
Farther things become smaller.
Camera projection
p∼K[R∣t]Pp \sim K[R|t]P
Meaning:
world
↓
camera pose
↓
camera projection
↓
pixels
Epipolar constraint
x2TFx1=0x_2^T F x_1 = 0
Meaning:
A point in one image corresponds to a line in the other.
Stereo depth
Z=fBdZ = \frac{fB}{d}
Meaning:
More disparity means closer.
Homography
p′∼Hpp' \sim Hp
Meaning:
Perspective mapping between planar views.
Those equations recover most of the geometric intuition you need.
PART XVIII — FAILURE MODES A WORKING ENGINEER SHOULD EXPECT
Real computer vision is rarely:
input
↓
perfect answer
You should instinctively ask what can fail.
Feature matching failure
Causes:
repeated textures
motion blur
low texture
large illumination change
large viewpoint change
Stereo failure
Causes:
textureless regions
reflective surfaces
repetitive patterns
occlusion
poor calibration
very distant objects
Optical flow failure
Causes:
fast motion
occlusions
lighting changes
large inter-frame displacement
non-rigid motion
Object detection failure
Causes:
occlusion
domain shift
tiny objects
unusual viewpoints
poor lighting
classes absent from training
Tracking failure
Causes:
crossing objects
long occlusion
detector misses
similar appearances
camera motion
Depth failure
Causes depend on the system:
Stereo:
correspondence ambiguity
Monocular:
scale ambiguity / unusual scenes
RGB-D:
range limits
reflective / transparent surfaces
sensor interference
Pose estimation failure
Causes:
too few correspondences
bad feature matches
degenerate geometry
poor calibration
low-texture environments
Real engineering means designing around these failure modes.
PART XIX — DEGENERATE CASES
A degenerate configuration is a situation in which the mathematical problem does not provide enough independent information for a reliable solution.
This concept is important far beyond computer vision.
For example, trying to determine full 3D structure when all observations provide almost the same geometric information can become poorly constrained.
Good engineers ask:
Is the algorithm failing because my implementation is wrong, or because the geometry itself contains insufficient information?
That distinction saves enormous debugging time.
PART XX — ESTIMATION, UNCERTAINTY, AND OPTIMIZATION
Computer vision is full of noisy measurements.
Suppose a theoretically perfect correspondence should be:
(423.2731, 281.9283)
But the detector reports:
(423.8, 282.4)
Another frame reports another noisy estimate.
Real measurements are never exact.
Therefore computer vision frequently becomes:
Find the parameters that best explain many imperfect observations.
This is an optimization problem.
68. Reprojection Error
Suppose you estimated a 3D point.
Project that point back into the camera.
estimated 3D point
↓
camera model
↓
predicted pixel
Compare:
predicted pixel
vs
observed pixel
Their difference is:
reprojection error
A good reconstruction should produce small reprojection errors.
69. Bundle Adjustment
In 3D reconstruction and SLAM, you may have:
many cameras
+
many 3D points
+
many image observations
Initial estimates are noisy.
Bundle adjustment jointly optimizes things such as:
camera poses
+
3D landmark positions
to reduce reprojection errors.
Conceptually:
"Move cameras and 3D points slightly until
the projected points line up with the actual images."
Bundle adjustment is one of the central optimization procedures in geometric computer vision.
PART XXI — COMPUTER VISION FOR AUTONOMOUS SYSTEMS
For your autonomous-systems mental model, separate vision into four levels.
Level 1 — Imaging
camera
calibration
intrinsics
distortion
Question:
What do my pixels geometrically mean?
Level 2 — Geometry
features
matching
optical flow
epipolar geometry
stereo
PnP
triangulation
Questions:
How did the camera move?
Where are things in 3D?
Level 3 — Semantics
detection
segmentation
classification
tracking
Questions:
What are the things I see?
Level 4 — World Understanding
3D reconstruction
semantic maps
object maps
occupancy
dynamic-object understanding
Question:
What kind of world am I operating inside?
Then robotics adds:
PERCEPTION
↓
LOCALIZATION
↓
MAPPING
↓
PLANNING
↓
CONTROL
↓
ACTION
Computer vision primarily contributes to the first three, but increasingly helps all of them.
PART XXII — A ROS ROBOT EXAMPLE
Imagine your TurtleBot has an RGB-D camera.
The robot sees a person.
YOLO outputs:
person bbox
Take the bounding box center:
(u,v)
Retrieve depth:
Z
Use camera intrinsics:
X=(u−cx)Z/fxX = (u-c_x)Z/f_xY=(v−cy)Z/fyY = (v-c_y)Z/f_y
Now you have approximately:
person position in camera frame
=
(X,Y,Z)
Then TF transforms:
camera_link
↓
base_link
↓
odom
↓
map
Now the robot knows where the person lies in map coordinates.
Tracking estimates whether the person is moving.
Navigation can then choose a goal while obstacle avoidance keeps the robot safe.
Your full pipeline becomes:
RGB image
↓
object detection
↓
person pixel location
depth image
↓
person depth
camera calibration
↓
pixel + depth → 3D camera point
TF
↓
camera point → map point
tracking
↓
person velocity
navigation
↓
follow safely
That single example combines a huge portion of practical computer vision.
PART XXIII — WHAT "MASTER / WORKING" SHOULD MEAN
You do not need to derive every theorem from memory.
For working mastery, you should be able to reason through this chain without confusion:
World point
↓
extrinsics
↓
camera coordinates
↓
projection
↓
intrinsics
↓
pixel
And reverse reasoning:
pixel
↓
intrinsics
↓
camera ray
↓
depth / another observation
↓
3D camera point
↓
extrinsics / TF
↓
world point
You should understand:
why two cameras recover depth
why calibration matters
why correspondence matters
why RANSAC exists
why PnP estimates pose
why homography is planar
why detection ≠ depth
why segmentation ≠ detection
why tracking ≠ detection
why 3D reconstruction needs multiple observations
If those connections are intuitive, you possess the core engineer's model of computer vision.
PART XXIV — THE "NEVER FORGET" STORY
If everything else disappears from your memory, reconstruct Computer Vision through this story.
Imagine a robot opening its eyes.
It receives nothing but:
pixels
First it asks:
How did my camera create these pixels?
That gives you:
image formation
pinhole model
intrinsics
extrinsics
calibration
distortion
Then:
How does perspective behave mathematically?
That gives you:
homogeneous coordinates
projective geometry
homographies
Then the robot opens a second eye:
Can two views tell me where things are in 3D?
That gives you:
epipolar geometry
stereo
disparity
triangulation
Then:
How do I know which pixel in one image corresponds to which pixel in another?
That gives you:
feature detection
descriptors
matching
optical flow
But some matches are wrong:
How do I ignore garbage measurements?
That gives you:
RANSAC
Then:
Can these observations tell me where my camera is?
That gives you:
pose estimation
PnP
visual odometry
SLAM
Then:
What am I actually looking at?
That gives you:
classification
detection
segmentation
Then video begins:
Is that the same object I saw one frame ago?
That gives you:
tracking
Then:
How far away is everything?
That gives you:
depth estimation
Finally:
Can I combine all these observations into a persistent geometric world?
That gives you:
3D reconstruction
mapping
That is Computer Vision.
MASTER CHEAT SHEET
IMAGE FORMATION
3D world → light → camera → 2D image
PINHOLE CAMERA
x = fX/Z
y = fY/Z
Perspective comes from division by depth.
INTRINSICS
Internal camera geometry.
fx, fy, cx, cy.
EXTRINSICS
Camera position and orientation.
R, t.
CALIBRATION
Estimate real camera geometry.
DISTORTION
Real lenses bend the ideal pinhole projection.
HOMOGENEOUS COORDINATES
Represent geometric transformations elegantly with matrices.
PROJECTIVE GEOMETRY
Geometry preserved under perspective cameras.
HOMOGRAPHY
Plane ↔ plane perspective mapping.
EPIPOLAR GEOMETRY
Point in image 1 → search line in image 2.
STEREO
Two views → disparity → depth.
DEPTH
Z = fB/d
TRIANGULATION
Multiple image rays → 3D point.
FEATURE DETECTION
Find interesting stable locations.
DESCRIPTOR
Encode what a local image neighborhood looks like.
MATCHING
Find corresponding locations across images.
OPTICAL FLOW
Estimate pixel motion over time.
RANSAC
Fit a geometric model despite outliers.
POSE
Position + orientation.
PnP
2D–3D correspondences → camera pose.
DETECTION
Object class + bounding box.
SEGMENTATION
Pixel-level object/scene labels.
TRACKING
Maintain object identity through time.
DEPTH ESTIMATION
Estimate distance from camera.
3D RECONSTRUCTION
Images + correspondences + poses + triangulation
→ 3D scene.
FINAL ENGINEER'S MODEL
Computer vision is not fundamentally about manipulating pictures.
It is about inferring the world that must have produced those pictures.
The deepest structure is:
WORLD
│
▼
CAMERA
│
▼
PIXELS
│
┌─────────────┼─────────────┐
▼ ▼ ▼
APPEARANCE GEOMETRY MOTION
│ │ │
▼ ▼ ▼
detection depth optical flow
segmentation pose tracking
recognition 3D
│ │ │
└─────────────┼─────────────┘
▼
WORLD UNDERSTANDING
│
▼
ROBOT DECISION MAKING
And perhaps the most useful single sentence to remember is:
Computer vision turns pixels into estimates about objects, geometry, depth, pose, motion, and ultimately the structure of the physical world.
For an autonomous-systems engineer, the winning mental model is therefore not:
"I know YOLO."
or:
"I know OpenCV."
It is:
I understand how a physical 3D world becomes pixels,
how multiple pixel observations constrain geometry,
how learned models extract semantic meaning,
how uncertainty and bad measurements are handled,
and how those estimates become a persistent world model
that an autonomous machine can act upon.
That is Computer Vision at MASTER/WORKING level.