Computer Architecture An Intuition-First Guide
Computer architecture is the bridge between:
the code you write and the electrical machine that actually executes it.
When you write:
c = a + b;
you see one line of code.
The computer sees something closer to:
load a from memory
load b from memory
place values in CPU registers
send them through the ALU
perform addition
store result in a register
write result back to memory
And underneath even that are billions of microscopic transistors switching between electrical states.
Understanding computer architecture means learning how these layers fit together.
1. The Big Picture
A simplified computer looks like this:
┌─────────────────────┐
│ CPU │
│ │
│ ┌───────────────┐ │
│ │ Registers │ │
│ └───────┬───────┘ │
│ │ │
│ ┌───────▼───────┐ │
│ │ ALU │ │
│ └───────┬───────┘ │
│ │ │
│ Control / Decode │
└──────────┬──────────┘
│
┌─────▼─────┐
│ Cache │
│ L1/L2/L3 │
└─────┬─────┘
│
┌─────▼─────┐
│ RAM │
└─────┬─────┘
│
┌─────▼─────┐
│ SSD / HDD │
└───────────┘
The central idea is:
The CPU computes. Memory stores.
But performance depends enormously on how quickly data can move between them.
That single observation explains caches, registers, memory hierarchies, GPUs, SIMD, prefetching, branch prediction, and much of modern computer architecture.
2. Binary Representation
Computers ultimately represent information using bits.
A bit has two possible states:
0
1
Why binary?
Because electronic circuits can reliably distinguish two states:
low voltage → 0
high voltage → 1
A sequence of bits can represent numbers.
For example:
0000 = 0
0001 = 1
0010 = 2
0011 = 3
0100 = 4
0101 = 5
Binary is positional just like decimal.
Decimal:
347
= 3×100 + 4×10 + 7×1
= 3×10² + 4×10¹ + 7×10⁰
Binary:
1011
= 1×2³ + 0×2² + 1×2¹ + 1×2⁰
= 8 + 0 + 2 + 1
= 11
So:
1011₂ = 11₁₀
Bits and bytes
Eight bits usually form one byte.
1 byte = 8 bits
00000000
11111111
Common sizes:
8 bits = 1 byte
16 bits = 2 bytes
32 bits = 4 bytes
64 bits = 8 bytes
A 32-bit unsigned integer can represent:
0 → 2³² - 1
while a 64-bit integer can represent vastly larger values.
3. Signed Integers
Unsigned integers cannot represent negative numbers.
So how do computers represent:
-5
Modern computers almost universally use two's complement.
For an 8-bit integer:
5 = 00000101
To produce -5:
invert every bit
add 1
00000101
→ 11111010
+ 1
-----------
11111011
Therefore:
-5 = 11111011
The brilliant part of two's complement is that the same addition hardware works for positive and negative numbers.
Example:
00000101 +5
+ 11111011 -5
-----------
1 00000000
The overflowing bit is discarded.
Result:
00000000
which is zero.
For an n-bit signed integer, the range is:
-2^(n-1) → 2^(n-1)-1
For 8 bits:
-128 → 127
For signed 32-bit integers:
-2,147,483,648
to
2,147,483,647
4. Floating-Point Numbers
Integers are easy.
But what about:
3.14159
0.000001
6.022 × 10²³
Computers usually represent these using floating point.
The idea resembles scientific notation.
Decimal scientific notation:
6.25 × 10³
Floating point uses powers of 2 instead.
Conceptually:
value ≈ sign × significand × 2^exponent
A common format is IEEE 754.
A 32-bit floating-point number contains:
┌──────┬──────────┬───────────────────────┐
│ sign │ exponent │ fraction / mantissa │
│ 1 bit│ 8 bits │ 23 bits │
└──────┴──────────┴───────────────────────┘
The important lesson is:
Floating-point numbers are usually approximations.
For example:
0.1 + 0.2
may internally produce something equivalent to:
0.30000000000000004
Why?
Because 0.1 has no finite exact representation in binary floating point, similar to how:
1 / 3 = 0.333333333...
cannot be represented exactly with finite decimal digits.
This matters enormously in:
numerical computing
robotics
graphics
machine learning
simulations
financial software
scientific computing
Never casually assume:
a == b
is safe for independently calculated floating-point values.
Often you instead test whether they are sufficiently close.
5. Boolean Logic
Computers also need to make decisions.
Boolean logic works with:
TRUE
FALSE
which map naturally onto:
1
0
The most fundamental Boolean operations are:
NOT
NOT 0 = 1
NOT 1 = 0
AND
Both inputs must be true.
A B | A AND B
----|--------
0 0 | 0
0 1 | 0
1 0 | 0
1 1 | 1
OR
At least one must be true.
A B | A OR B
----|-------
0 0 | 0
0 1 | 1
1 0 | 1
1 1 | 1
XOR
Exactly one must be true.
A B | A XOR B
----|--------
0 0 | 0
0 1 | 1
1 0 | 1
1 1 | 0
These operations seem simple, but they are the foundation of digital computers.
6. Digital Gates
Boolean operations become physical circuits called logic gates.
A ───┐
├── AND ── output
B ───┘
or:
A ───┐
├── OR ─── output
B ───┘
Transistors are combined to construct these gates.
Gates are then combined into more complicated circuits.
For example, we can construct a binary adder.
A ───┐
├── XOR ── Sum
B ───┘
A ───┐
├── AND ── Carry
B ───┘
This is called a half adder.
By combining many adders, computers can add entire 32-bit or 64-bit numbers.
The hierarchy becomes:
transistors
↓
logic gates
↓
adders / multiplexers / flip-flops
↓
registers / ALUs
↓
CPU
↓
computer
Your Python program is therefore ultimately executed by carefully coordinated transistor switching.
7. Registers
Registers are extremely small storage locations inside the CPU.
They are among the fastest storage available in the entire computer.
Suppose the CPU wants to calculate:
7 + 5
It might conceptually do:
R1 = 7
R2 = 5
R3 = R1 + R2
Here:
R1
R2
R3
are registers.
A CPU might have registers with names such as:
rax
rbx
rcx
rdx
rsp
rbp
on x86-64.
Or:
x0
x1
x2
...
x30
on ARM64.
Registers hold things like:
arithmetic operands
function arguments
addresses
loop counters
return values
temporary calculations
The CPU loves registers because accessing RAM is much slower.
So optimizing compilers try hard to keep important values in registers.
8. The ALU
The Arithmetic Logic Unit is the part of the CPU responsible for basic computation.
It performs operations such as:
addition
subtraction
AND
OR
XOR
bit shifting
comparisons
Conceptually:
operand A
│
▼
┌──────────────┐
│ │
│ ALU │
│ │
└──────────────┘
▲
│
operand B
│
▼
result
The CPU's control logic tells the ALU which operation to perform.
For example:
ADD R1, R2
might mean:
R1 ← R1 + R2
9. CPU Architecture
A CPU is much more than an ALU.
A simplified modern CPU core contains:
Instructions
│
▼
┌───────────┐
│ Fetch │
└─────┬─────┘
▼
┌───────────┐
│ Decode │
└─────┬─────┘
▼
┌──────────────────┐
│ Register File │
└────────┬─────────┘
▼
┌──────────────────┐
│ Execution Units │
│ ALU / FPU / SIMD │
└────────┬─────────┘
▼
┌───────────┐
│ Memory │
└─────┬─────┘
▼
┌───────────┐
│ Retire │
└───────────┘
Modern CPUs may contain several different execution units:
integer ALUs
floating-point units
load/store units
vector units
branch units
And modern CPUs can execute multiple instructions simultaneously.
10. Instruction Sets
A CPU does not understand C++ or Python.
It understands machine instructions.
The collection of instructions a processor understands is called its:
Instruction Set Architecture — ISA
Examples:
x86-64
ARM64
RISC-V
An ISA defines things such as:
available instructions
registers
memory addressing
data types
calling behavior
privilege levels
Example conceptual instructions:
LOAD
STORE
ADD
SUB
MUL
COMPARE
JUMP
CALL
RETURN
Different CPUs may implement the same ISA internally in completely different ways.
That distinction is important.
ISA
= what instructions the processor promises to support
Microarchitecture
= how that processor actually implements them
Two x86 CPUs from different generations may support the same programs while having radically different internal designs.
11. Assembly Basics
Assembly language is a human-readable representation of machine instructions.
Consider:
int c = a + b;
Assembly might resemble:
mov eax, a
add eax, b
mov c, eax
Conceptually:
move a → register eax
add b to eax
store eax → c
Another example:
if (x == 5)
y = 10;
could become something resembling:
cmp eax, 5
jne skip
mov ebx, 10
skip:
So high-level control flow eventually becomes:
compare
branch
jump
Assembly is useful because it exposes what your code eventually becomes.
You do not need to become an assembly expert to understand architecture.
But being comfortable reading simple assembly is extremely valuable for:
debugging
embedded systems
operating systems
robotics
performance optimization
compiler understanding
cybersecurity
12. The Stack
You probably know a stack as the data structure:
push
pop
LIFO
CPUs and programs use a special region of memory called the call stack.
When functions call each other:
main()
↓
foo()
↓
bar()
the stack may conceptually look like:
┌────────────────────┐
│ bar local variables│
│ bar return address │
├────────────────────┤
│ foo local variables│
│ foo return address │
├────────────────────┤
│ main variables │
└────────────────────┘
Each function invocation gets a stack frame.
A stack frame can contain:
local variables
saved registers
function arguments
return address
A special register usually tracks the top of the stack.
On x86-64:
rsp = stack pointer
This is why excessive recursion can cause:
stack overflow
Each recursive call adds another stack frame until the available stack memory is exhausted.
13. Registers and Function Calls
Suppose you call:
add(10, 20);
How does the CPU know where the arguments are?
There needs to be an agreement.
Perhaps:
first argument → register R1
second argument → register R2
return value → register R0
This agreement is part of the calling convention.
14. Calling Conventions
A calling convention defines how functions communicate.
It specifies things such as:
where arguments go
where return values go
which registers functions must preserve
how the stack is managed
For example, on one common x86-64 convention, early integer arguments might be placed in registers such as:
rdi
rsi
rdx
rcx
r8
r9
while the return value commonly appears in:
rax
So:
int add(int a, int b)
might conceptually become:
rdi = a
rsi = b
call add
rax = result
Calling conventions allow separately compiled code to work together.
Without them:
library A:
"I put argument 1 in rdi."
library B:
"I expected it on the stack."
Everything would break.
15. Why Memory Speed Matters
CPUs became extraordinarily fast.
Memory did not become equally fast.
This created a fundamental problem:
CPU speed >>> RAM access speed
Imagine a chef who can prepare a dish in one second but ingredients take one minute to arrive.
The chef spends most of the time waiting.
Modern CPUs would face the same problem if they directly accessed RAM for everything.
The solution is a memory hierarchy.
16. Memory Hierarchy
Computer storage is roughly organized like this:
Fast
▲
│
Registers
│
L1 Cache
│
L2 Cache
│
L3 Cache
│
RAM
│
SSD/HDD
│
▼
Slow
But capacity tends to increase downward:
Registers → tiny
Cache → small
RAM → large
SSD → enormous
And cost per byte generally decreases downward.
A useful mental model:
Speed Capacity
Registers insane tiny
L1 cache extremely tiny
L2 cache very fast small
L3 cache fast medium
RAM slower GBs
SSD very slow TBs
Exact timings vary enormously between architectures, but the relative hierarchy is what matters.
17. Cache Hierarchy
Caches are small, fast memories close to the CPU.
Modern processors commonly have:
L1
L2
L3
L1 cache
Very small and extremely fast.
Often each CPU core has its own.
L2 cache
Larger but slightly slower.
L3 cache
Much larger and commonly shared between multiple cores.
Conceptually:
Core 1 Core 2 Core 3 Core 4
│ │ │ │
L1 L1 L1 L1
│ │ │ │
L2 L2 L2 L2
\ | | /
─────── L3 CACHE ───────
│
RAM
When the CPU needs data:
check L1
↓ miss
check L2
↓ miss
check L3
↓ miss
fetch from RAM
This is why cache-friendly algorithms can be dramatically faster.
18. Locality
Caches work because programs tend to exhibit locality.
There are two major forms.
Temporal locality
If you used something recently, you may use it again soon.
Example:
for (...) {
sum += x;
}
sum is accessed repeatedly.
Spatial locality
If you accessed one memory location, nearby locations may soon be accessed.
Example:
for (int i = 0; i < n; i++)
sum += array[i];
Arrays are stored contiguously.
So when the CPU loads:
array[0]
it may fetch neighboring values too.
This is why sequential array traversal is usually much faster than random memory traversal.
19. Cache Lines
Caches usually do not fetch individual bytes.
They fetch blocks called cache lines.
A typical cache line might be around:
64 bytes
Suppose you request:
array[100]
The CPU may fetch a whole block containing:
array[100]
array[101]
array[102]
...
This makes sequential processing highly efficient.
It also explains why data layout matters.
For high-performance robotics, simulation, game engines, databases, ML runtimes, and numerical computing:
How data is arranged in memory can matter almost as much as the algorithm itself.
20. RAM
RAM is the computer's main working memory.
Programs, variables, buffers, models, images, sensor data, and operating-system structures live there while being actively used.
RAM is:
much larger than cache
much faster than SSD
much slower than CPU registers/cache
RAM is volatile.
If power disappears:
RAM contents disappear.
Persistent information therefore lives on SSDs, HDDs, flash storage, etc.
21. Virtual Memory
Suppose your computer has:
16 GB RAM
Does every process directly manipulate physical RAM addresses?
Usually not.
Processes operate using virtual addresses.
Example:
Program believes:
0x1000
0x1001
0x1002
...
The operating system and hardware translate these into physical RAM locations.
Virtual Address
│
▼
┌───────────────┐
│ Page Tables │
└───────┬───────┘
▼
Physical Address
│
▼
RAM
This gives powerful benefits.
Each process can behave as if it owns its own private memory space.
Program A might see:
0x1000
Program B might also see:
0x1000
while those virtual addresses point to completely different physical locations.
This provides:
isolation
security
easier memory management
memory sharing when desired
paging
larger virtual address spaces
22. Pages and Page Tables
Virtual memory is normally managed in chunks called pages.
A common page size is:
4 KiB
The operating system maintains mappings such as:
Virtual Page 17 → Physical Frame 291
Virtual Page 18 → Physical Frame 812
Virtual Page 19 → Physical Frame 101
These mappings live in page tables.
Because repeatedly walking large page tables would itself be slow, CPUs have a cache for translations called the:
TLB — Translation Lookaside Buffer
So memory lookup may conceptually involve:
virtual address
↓
TLB
↓
physical address
↓
cache
↓
RAM
Architecture is full of caches.
Caches exist because avoiding slow work is often better than making the slow work itself faster.
23. Interrupts
Imagine the CPU had to constantly ask:
Keyboard, did someone press a key?
Keyboard, now?
Now?
Now?
Now?
That would waste enormous amounts of CPU time.
Instead, devices can send the CPU an interrupt.
Conceptually:
CPU executing program
│
│
▼
normal work
DEVICE: "I NEED ATTENTION!"
│
▼
CPU temporarily pauses
current execution
│
▼
Interrupt Handler
│
▼
handle device
│
▼
resume previous program
Interrupts are fundamental to operating systems and embedded systems.
Examples:
keyboard input
network packet arrival
timers
disk completion
sensor events
hardware faults
In microcontrollers and robotics, you will encounter interrupt service routines directly.
24. Pipelining
Imagine doing laundry.
Without pipelining:
wash load 1
dry load 1
fold load 1
wash load 2
dry load 2
fold load 2
Instead:
Washer: load 2
Dryer: load 1
Different stages work simultaneously.
CPUs do the same thing with instructions.
A basic CPU pipeline might contain:
Fetch
Decode
Execute
Memory
Writeback
Without pipelining:
Instruction 1:
F → D → E → M → W
then
Instruction 2:
F → D → E → M → W
With pipelining:
Cycle → 1 2 3 4 5 6 7
Instruction 1 F D E M W
Instruction 2 F D E M W
Instruction 3 F D E M W
Instruction 4 F D E M W
After the pipeline fills, many instructions are in flight simultaneously.
This greatly improves throughput.
25. Pipeline Hazards
Pipelining creates complications.
Consider:
Instruction 1: R1 = R2 + R3
Instruction 2: R4 = R1 + R5
Instruction 2 needs the result of instruction 1.
But instruction 1 may not have finished yet.
This creates a:
data hazard
CPUs use techniques such as:
forwarding
stalling
out-of-order execution
register renaming
to deal with these dependencies.
26. Branch Prediction
Consider:
if (temperature > threshold) {
stop_motor();
}
The CPU eventually encounters a branch:
condition true?
├── yes → execute one path
└── no → execute another path
But modern CPUs have deep pipelines.
Waiting until the condition is fully known would waste cycles.
So the CPU predicts:
"I think the condition will be false."
and starts executing that path.
If correct:
great
If wrong:
discard speculative work
clear part of pipeline
start correct path
This is called a branch misprediction.
Modern branch predictors are extremely sophisticated because accurate prediction produces enormous performance improvements.
This explains why unpredictable branches inside tight loops can sometimes hurt performance.
27. Speculative Execution
Branch prediction leads to a broader concept:
The CPU performs work before knowing whether that work will definitely be needed.
This is speculative execution.
The CPU might execute future instructions while earlier conditions are still unresolved.
If its speculation was correct:
keep results
If incorrect:
discard results
This contributes heavily to the enormous speed of modern processors.
It also created subtle security problems such as the family of speculative-execution vulnerabilities exposed by Spectre-like attacks.
28. Out-of-Order Execution
Imagine these instructions:
1. load A from RAM
2. add A + 10
3. calculate B * C
Instruction 2 must wait for A.
But instruction 3 does not depend on A.
A simplistic processor waits:
1 → waiting...
2 → blocked
3 → blocked
A modern processor may instead do:
1 → waiting for memory
3 → execute now
2 → execute once A arrives
This is called:
Out-of-Order Execution
The CPU dynamically finds independent work.
The programmer still observes the result as though instructions executed in the required logical order.
This distinction between:
program order
and
actual execution order
becomes important when understanding concurrency and memory consistency.
29. SIMD
SIMD means:
Single Instruction, Multiple Data
Suppose you want to add:
A = [1, 2, 3, 4]
B = [5, 6, 7, 8]
A normal scalar CPU might conceptually perform:
1 + 5
2 + 6
3 + 7
4 + 8
four separate operations.
SIMD hardware can operate on several values simultaneously:
[1 2 3 4]
+
[5 6 7 8]
↓
[6 8 10 12]
One vector instruction operates across several values.
Examples of SIMD instruction technologies include:
SSE
AVX
AVX2
AVX-512
ARM NEON
SIMD is particularly useful in:
image processing
computer vision
matrix operations
audio processing
physics
machine learning
robotics perception
30. Multicore CPUs
Eventually increasing the clock speed of individual CPUs became increasingly difficult due to:
power consumption
heat
physical limits
So manufacturers increasingly added more CPU cores.
A 12-core CPU is effectively capable of running multiple instruction streams simultaneously.
Conceptually:
┌─────────────┐
│ Shared L3 │
└──────┬──────┘
│
┌────────────┼────────────┐
│ │ │
Core 1 Core 2 Core 3
L1/L2 L1/L2 L1/L2
This means software can use threads to execute tasks concurrently.
Example robotics system:
Core 1 → camera processing
Core 2 → localization
Core 3 → planning
Core 4 → networking
Core 5 → control
But concurrency creates another enormous problem:
What happens when multiple cores access the same memory?
31. Cache Coherence
Suppose:
Core 1 cache says:
x = 5
and:
Core 2 cache says:
x = 5
Then Core 1 modifies:
x = 10
Core 2 must not continue believing forever that:
x = 5
Processors therefore implement cache coherence protocols.
Their job is to keep multiple cached copies of shared memory logically consistent.
Protocols such as MESI track states resembling:
Modified
Exclusive
Shared
Invalid
You do not need to memorize MESI immediately.
The important intuition is:
Multiple cores have private caches, so hardware needs protocols to coordinate shared data.
32. Memory Consistency
Cache coherence and memory consistency are related but different.
Consider two threads.
Thread 1:
data = 42;
ready = true;
Thread 2:
if (ready)
print(data);
You might assume:
If Thread 2 sees ready == true,
it must see data == 42.
Not necessarily.
Compilers and CPUs may reorder operations when doing so preserves normal single-threaded behavior.
Modern processors therefore define memory consistency models describing which ordering guarantees software receives.
Languages then expose tools such as:
mutexes
atomic variables
memory barriers
locks
to enforce synchronization.
The important lesson:
In concurrent programs, the apparent order of source-code operations does not automatically guarantee the order other CPU cores observe them.
This is why concurrent programming can become surprisingly difficult.
33. Memory Hierarchy Revisited
By now, you can understand memory as something much richer than simply "RAM."
A memory request might travel through:
CPU instruction
↓
register
↓
virtual address
↓
TLB
↓
L1 cache
↓
L2 cache
↓
L3 cache
↓
memory controller
↓
DRAM
Potentially across several CPU cores with cache coherence protocols coordinating shared data.
That is why:
x += 1;
can be trivial in one situation and surprisingly expensive in another.
34. CPU vs GPU vs TPU vs FPGA
Now we reach one of the most important architectural distinctions for modern AI and robotics.
These machines all compute.
But they are optimized for different kinds of computation.
CPU
A CPU is a general-purpose processor.
Its philosophy is approximately:
A relatively small number of extremely powerful, flexible cores.
CPU strengths:
complex control flow
operating systems
branching
sequential algorithms
diverse workloads
low-latency logic
Example:
CPU
Core ── powerful
Core ── powerful
Core ── powerful
Core ── powerful
Think:
A few extremely skilled workers capable of solving almost anything.
35. GPU
A GPU follows a different philosophy.
Instead of a few extremely sophisticated workers:
Provide enormous numbers of simpler workers.
Conceptually:
CPU
┌────┐ ┌────┐ ┌────┐ ┌────┐
│Big │ │Big │ │Big │ │Big │
└────┘ └────┘ └────┘ └────┘
GPU
┌─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┐
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
├─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┤
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
├─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┤
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
└─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┘
This is excellent when the same operation must be performed across huge amounts of data.
For example:
multiply millions of matrix elements
which is exactly what neural networks require.
36. GPU Architecture
Modern GPU terminology varies by manufacturer, but the broad structure is approximately:
GPU
│
┌──────────┼───────────┐
│ │ │
SM SM SM
│ │ │
many cores many cores many cores
On NVIDIA GPUs, major computation occurs in units called:
Streaming Multiprocessors — SMs
Each SM contains many execution resources.
Threads are grouped into blocks.
Blocks execute on SMs.
Inside an SM, groups of threads known as warps execute together.
A warp on modern NVIDIA hardware traditionally contains:
32 threads
Conceptually:
Thread 0 ─┐
Thread 1 │
Thread 2 │
... ├── Warp
Thread 31 ┘
Those threads typically execute the same instruction over different data.
This is related to SIMD, though NVIDIA commonly describes its model as:
SIMT — Single Instruction, Multiple Threads.
37. GPU Branch Divergence
Suppose one GPU warp contains 32 threads.
You write:
if (x > 0)
do_A();
else
do_B();
Imagine:
Thread 0 → A
Thread 1 → A
Thread 2 → B
Thread 3 → B
...
Threads inside the warp want different control-flow paths.
This creates warp divergence.
The GPU may effectively execute both paths while masking threads that do not belong to each path.
So GPUs strongly prefer workloads where many threads perform similar operations.
This is why GPUs excel at:
matrix multiplication
image filtering
neural networks
physics simulation
but may perform poorly on highly irregular branching workloads.
38. GPU Memory Hierarchy
GPUs also have a memory hierarchy.
Conceptually:
Registers
↓
Shared memory / L1
↓
L2 cache
↓
Global GPU memory
GPU global memory may have enormous bandwidth.
Modern high-end accelerators can move astonishing amounts of data each second.
But latency still matters.
High-performance GPU programming therefore focuses heavily on:
coalesced memory access
shared memory
tiling
avoiding unnecessary transfers
keeping threads occupied
39. Why GPUs Are Excellent for AI
Consider matrix multiplication:
C = A × B
A large matrix contains millions or billions of independent multiply-and-add operations.
That is perfect for GPUs.
Instead of:
one powerful CPU core
performing millions of operations sequentially
we can use:
thousands of GPU execution units
performing large numbers simultaneously
Deep learning is largely built from operations such as:
matrix multiplication
convolution
attention
normalization
element-wise functions
Many of these parallelize extremely well.
Thus:
AI revolution
+
massive parallelism
+
GPU hardware
became deeply connected.
40. TPU
A TPU is a specialized accelerator originally designed around machine-learning workloads.
The philosophy is:
If matrix multiplication dominates neural-network computation, build hardware specifically optimized for matrix multiplication.
Instead of being as general-purpose as a CPU, a TPU dedicates much more hardware toward tensor operations.
A key architectural idea is the systolic array.
Conceptually:
data → □ → □ → □
↓ ↓ ↓
□ → □ → □
↓ ↓ ↓
□ → □ → □
Data flows through a grid of multiply-accumulate units.
Instead of repeatedly moving values between distant memory and processors, data is reused as it flows through the array.
This makes large matrix operations extremely efficient.
41. FPGA
FPGA means:
Field-Programmable Gate Array
A CPU executes instructions.
An FPGA lets you configure the underlying digital logic itself.
CPU:
hardware fixed
instructions change
FPGA:
hardware structure itself can be configured
Conceptually:
Programmable logic blocks
+
Programmable interconnections
You can construct custom pipelines for a task.
For example:
camera
↓
FPGA image preprocessing
↓
object detector
↓
control system
FPGA advantages:
very low latency
deterministic timing
energy efficiency
custom interfaces
massive custom parallelism
Disadvantages:
harder to program
more specialized knowledge
longer development cycles
They are widely useful in:
robotics
telecommunications
industrial control
aerospace
automotive systems
high-frequency trading
specialized AI inference
42. CPU vs GPU vs TPU vs FPGA — Intuition
Think of a restaurant.
CPU
A few brilliant chefs.
"Give me any dish.
I can figure it out."
Excellent flexibility.
GPU
Thousands of workers performing similar tasks.
"Everyone chop one carrot."
Exceptional throughput.
TPU
A specialized factory designed to produce one family of dishes extremely efficiently.
"We optimized the entire building around making this."
Excellent neural-network tensor throughput.
FPGA
A restaurant whose kitchen layout itself can be redesigned.
"For this recipe,
we'll construct a custom production line."
Maximum specialization and deterministic behavior.
43. Parallel Computation
Parallel computation means performing multiple pieces of work simultaneously.
Suppose you need to process:
1,000 camera images
Sequentially:
image 1
image 2
image 3
...
image 1000
Parallel:
Worker 1 → image 1
Worker 2 → image 2
Worker 3 → image 3
Worker 4 → image 4
...
But there are several kinds of parallelism.
44. Instruction-Level Parallelism
The CPU executes multiple independent instructions simultaneously.
For example:
a = b + c
d = e * f
Since these calculations do not depend on each other, different execution units may handle them simultaneously.
This happens inside modern CPUs automatically.
Mechanisms include:
pipelining
superscalar execution
out-of-order execution
45. Data-Level Parallelism
Apply the same operation to many data elements.
Example:
brightness(pixel[i]) += 10
for millions of pixels.
Use:
SIMD
GPU
vector processors
This is one of the most common forms of parallelism in AI.
46. Thread-Level Parallelism
Different threads execute different work.
Example:
Thread 1 → read camera
Thread 2 → detect objects
Thread 3 → localization
Thread 4 → path planning
Multiple CPU cores can execute these concurrently.
This is especially important in robotics software.
47. Task-Level Parallelism
Entire tasks run independently.
Imagine an autonomous robot:
ROBOT
│
┌────────────┼─────────────┐
│ │ │
▼ ▼ ▼
Camera LiDAR Encoders
│ │ │
▼ ▼ ▼
Detection SLAM Odometry
│ │ │
└──────┬─────┴─────────────┘
▼
Planning
│
▼
Control
Many parts can execute concurrently.
This is one reason ROS 2 is naturally built around independent nodes, executors, callbacks, threads, and message passing.
48. Parallelism Is Not Free
Suppose a task takes:
10 seconds
Will ten CPU cores automatically make it:
1 second?
No.
Some work cannot be parallelized.
If:
20% of the program is inherently sequential
then no amount of parallel hardware can eliminate that section.
This idea is captured by Amdahl's Law.
The intuition matters more than memorizing the equation:
The sequential portion eventually becomes the bottleneck.
Parallel programs also pay overhead for:
thread creation
communication
synchronization
locks
cache coherence
data transfer
49. Synchronization
Suppose two threads modify:
counter++;
simultaneously.
This appears to be one operation.
Internally it might be:
load counter
add 1
store counter
Suppose:
counter = 5
Thread A:
load 5
Thread B:
load 5
Thread A calculates:
6
Thread B calculates:
6
Both store:
6
But logically two increments should produce:
7
This is a race condition.
We use synchronization primitives such as:
mutex
atomic
semaphore
condition variable
to coordinate concurrent access.
50. CPU Clock
CPUs operate according to a clock.
For example:
3 GHz
means roughly:
3 billion clock cycles per second
But:
3 GHz does NOT mean exactly 3 billion instructions per second.
Modern CPUs may:
execute multiple instructions per cycle
stall waiting for memory
mispredict branches
execute instructions out of order
perform vector operations involving many values
So performance cannot be understood from clock speed alone.
A lower-frequency modern CPU may easily outperform a higher-frequency older CPU.
51. IPC — Instructions Per Cycle
A useful performance concept is:
IPC — Instructions Per Cycle
Imagine:
CPU A:
4 GHz
1 instruction/cycle
≈ 4 billion instructions/s
and:
CPU B:
3 GHz
2 instructions/cycle
≈ 6 billion instructions/s
This is oversimplified, but illustrates why:
clock frequency ≠ overall performance
Architecture matters enormously.
52. Latency vs Throughput
Two concepts appear everywhere in computer architecture.
Latency
How long one operation takes.
request
↓
... waiting ...
↓
result
Throughput
How many operations can be completed per unit time.
A GPU may have enormous throughput:
process millions of numbers simultaneously
while an individual operation may not necessarily have lower latency than a CPU.
This is crucial when deciding between architectures.
Real-time robotics often cares deeply about:
latency
Large-scale neural-network training cares heavily about:
throughput
Often you care about both.
53. Why Data Movement Matters
A major modern architecture lesson is:
Computation is often cheaper than moving data.
Consider AI.
You might think:
matrix multiplication is expensive
But frequently the harder problem is moving:
model weights
activations
KV cache
sensor tensors
through memory quickly enough to keep compute units busy.
This produces a distinction:
compute-bound workload
versus:
memory-bound workload
Compute-bound
The arithmetic units are the bottleneck.
Memory-bound
The processor spends much of its time waiting for data.
This distinction becomes extremely important for GPU kernels and AI inference.
54. A Useful Architecture Mental Model
When trying to understand why some program is slow, ask:
1. Where is the data?
Registers?
Cache?
RAM?
GPU memory?
SSD?
2. How much data must move?
3. How many calculations must happen?
4. Can those calculations happen simultaneously?
5. Are branches predictable?
6. Are threads competing for shared resources?
7. Is the workload latency-sensitive or throughput-sensitive?
These questions will take you surprisingly far.
55. From Python to Transistors
Consider:
c = a + b
You can now mentally travel downward.
Python
↓
Interpreter / runtime
↓
native machine instructions
↓
ISA instructions
↓
CPU instruction decoder
↓
register operands
↓
ALU operation
↓
digital logic
↓
logic gates
↓
transistors
↓
electrical signals
And upward:
electrical signals
↓
transistors
↓
logic gates
↓
ALU/registers
↓
CPU instructions
↓
operating system/runtime
↓
program
↓
robot / AI system behavior
That is computer architecture.
56. Why This Matters for Robotics
A robotics engineer frequently has multiple computational systems inside one robot.
For example:
ROBOT
│
┌────────────────┼────────────────┐
│ │ │
▼ ▼ ▼
Microcontroller CPU GPU
│ │ │
motor control ROS 2 perception
interrupts Nav2 YOLO
PWM SLAM neural nets
encoders planning tensors
An MCU may handle deterministic motor control.
The CPU may handle:
ROS 2
navigation
planning
communication
system coordination
The GPU may handle:
object detection
depth estimation
segmentation
vision-language models
An FPGA might eventually handle:
custom sensor pipelines
high-speed camera processing
hard real-time interfaces
Understanding architecture helps you decide:
Which computation belongs where?
57. Why This Matters for AI Engineering
AI systems are fundamentally architecture-sensitive.
Training a neural network involves:
CPU
↓
prepare batches
↓
transfer data
↓
GPU
↓
matrix multiplication
↓
GPU memory
↓
more computation
Large models introduce further bottlenecks:
GPU memory capacity
memory bandwidth
tensor-core throughput
GPU-to-GPU communication
CPU-GPU transfer
KV cache
batch size
precision
Terms such as:
FP32
FP16
BF16
INT8
CUDA
tensor cores
VRAM
memory bandwidth
batching
quantization
all make much more sense once you understand computer architecture.
58. What You Should Actually Master
You do not need to become a CPU designer.
For a robotics / AI / systems engineer, you should have strong intuition about:
Representation
Understand:
binary
signed integers
floating point
Boolean logic
CPU fundamentals
Understand:
registers
ALU
instructions
assembly basics
stack
calling conventions
Memory
Understand very well:
registers
cache
RAM
virtual memory
memory hierarchy
locality
Modern CPU performance
Develop intuition about:
pipelining
branch prediction
out-of-order execution
SIMD
multicore
cache coherence
memory consistency
Accelerators
Be comfortable reasoning about:
CPU
GPU
TPU
FPGA
especially:
GPU parallelism
GPU memory hierarchy
SIMD/SIMT
matrix operations
memory bandwidth
59. The One Diagram Worth Remembering
If you forget almost everything in this chapter, remember this:
SOFTWARE
Python / C++ / ROS 2
│
▼
Compiler
│
▼
Machine Instructions
│
▼
┌───────────────┐
│ CPU │
│ │
│ Registers │
│ ALU / SIMD │
│ Control │
└───────┬───────┘
│
L1 / L2 Cache
│
L3 Cache
│
RAM
│
SSD
+
┌───────────────┐
│ GPU │
│ │
│ Thousands of │
│ parallel │
│ workers │
└───────────────┘
│
▼
Digital Logic Gates
│
▼
Transistors
│
▼
PHYSICS
Computer architecture lives between:
software abstractions
and
physical computation.
60. Final Mental Model
The entire subject can be compressed into a few principles.
Principle 1 — Everything becomes bits
numbers
text
images
instructions
neural networks
ultimately become binary information.
Principle 2 — CPUs execute instructions
Programs become instructions operating primarily on registers and memory.
Principle 3 — Fast memory is small
Therefore computers use:
registers
↓
L1
↓
L2
↓
L3
↓
RAM
↓
storage
Principle 4 — Data movement is expensive
Keeping computation near data is one of the central problems of modern hardware.
Principle 5 — Modern CPUs cheat intelligently
They use:
pipelining
branch prediction
speculation
out-of-order execution
caches
SIMD
to execute programs much faster than naïve sequential execution.
Principle 6 — More cores require coordination
Parallel execution introduces:
synchronization
race conditions
cache coherence
memory ordering
Principle 7 — Different hardware exists for different workloads
CPU → flexible sequential/general computation
GPU → huge data-parallel computation
TPU → neural-network tensor computation
FPGA → custom hardware pipelines
Principle 8 — Modern AI is partly a hardware problem
Model performance is not determined only by:
number of FLOPs
but also by:
memory bandwidth
cache behavior
parallelism
precision
data movement
accelerator utilization
The Architecture Intuition to Keep Forever
Whenever you write code, imagine that underneath it there is a machine continuously asking:
What instruction should I execute?
Where is the data?
Is it already in a register?
Is it in cache?
Do I need RAM?
Can I perform several operations simultaneously?
Can I predict what instruction comes next?
Can another CPU core help?
Can this work be vectorized?
Would thousands of GPU threads be better?
Am I spending more time moving data than computing?
Once those questions start appearing naturally in your head, computer architecture stops being a collection of hardware terms.
You begin seeing it as what it really is:
The engineering of moving data and performing computation as efficiently as possible under physical constraints.