Modern AI is not only about neural-network architecture.

It is also about moving enormous amounts of data through enormous amounts of computation efficiently.

That is why GPU computing matters.

If you train a transformer, run a diffusion model, serve an LLM, fine-tune with thousands of GPUs, or optimize inference latency, you are constantly dealing with questions such as:

Where is the data stored?
Which processor is doing the work?
How many operations can run simultaneously?
Are the GPU cores waiting for memory?
Are GPUs waiting for each other?
Are we using expensive 32-bit arithmetic when 16-bit would work?
Is the model too large for one GPU?
Is communication slower than computation?
Are tensor cores actually being used?
Why is GPU utilization only 40%?

Understanding these questions is the foundation of high-performance AI systems.

1. CPU vs GPU: the most important intuition

A CPU is designed to be excellent at doing a relatively small number of complicated things very quickly.

A GPU is designed to do an enormous number of relatively simple operations simultaneously.

Think of a CPU as a few extremely skilled engineers.

A GPU is more like an enormous factory containing thousands of workers.

Suppose we need to add two arrays:

A = [1, 2, 3, 4, ..., 1,000,000]
B = [5, 6, 7, 8, ..., 1,000,000]

We want:

C[i] = A[i] + B[i]

A traditional CPU might process several values simultaneously using multiple CPU cores and vector instructions.

A GPU may schedule tens or hundreds of thousands of lightweight threads.

Conceptually:

Thread 0      -> C[0] = A[0] + B[0]
Thread 1      -> C[1] = A[1] + B[1]
Thread 2      -> C[2] = A[2] + B[2]
...
Thread 999999 -> C[999999] = A[999999] + B[999999]

This type of computation is called massively parallel computing.

Neural networks happen to contain an enormous amount of this kind of work.

For example:

matrix multiplication
convolutions
attention
element-wise activation functions
normalization
embedding lookup
gradient computation

This is one major reason GPUs became the dominant hardware for modern deep learning.

2. Why neural networks love GPUs

Consider a simple neural-network layer:

Y=XWY = XW

where:

XX = input matrix
WW = weight matrix
YY = output matrix

If:

X = 1024 × 4096
W = 4096 × 4096

then computing XWXW requires billions of multiply-and-add operations.

One output element is approximately:

Yij=∑kXikWkjY_{ij} = \sum_k X_{ik}W_{kj}

Each output cell can largely be computed independently.

That means thousands of GPU execution units can work simultaneously.

This is exactly the kind of workload GPUs were designed for.

Modern AI therefore spends enormous amounts of time doing:

GEMM

which means:

General Matrix-Matrix Multiplication

When people optimize transformer performance, they are often indirectly asking:

How efficiently are we keeping the GPU's matrix multiplication hardware busy?

3. Simplified GPU architecture

A modern NVIDIA GPU contains many execution units organized into larger structures.

The exact terminology differs across architectures, but conceptually we can imagine:

GPU
│
├── Streaming Multiprocessor
│   ├── CUDA cores
│   ├── Tensor cores
│   ├── registers
│   ├── shared memory
│   └── warp schedulers
│
├── Streaming Multiprocessor
│   └── ...
│
├── L2 cache
│
└── Global GPU memory / HBM / GDDR

NVIDIA calls the major compute unit an:

SM

or:

Streaming Multiprocessor

Each SM contains resources required to execute many GPU threads.

Depending on the GPU generation, an SM contains things such as:

CUDA cores
Tensor Cores
warp schedulers
registers
shared memory
load/store units
special-function units

A high-end GPU may contain many dozens or even hundreds of SMs.

This lets thousands of operations happen concurrently.

4. CUDA

CUDA stands for:

Compute Unified Device Architecture

CUDA is NVIDIA's software and programming platform for general-purpose GPU computing.

Before CUDA, GPUs were primarily graphics processors.

CUDA allowed developers to use GPUs for general computations such as:

physics simulation
scientific computing
cryptography
machine learning
deep learning
numerical optimization

When PyTorch executes:

tensor = tensor.cuda()

you are ultimately entering the CUDA ecosystem.

PyTorch sits on top of lower-level technologies such as:

PyTorch
   ↓
ATen / operators
   ↓
cuBLAS / cuDNN / Triton / custom kernels
   ↓
CUDA
   ↓
NVIDIA driver
   ↓
GPU

Depending on the operation, different libraries may perform the work.

For example:

matrix multiplication → cuBLAS
deep-learning primitives → cuDNN
collective communication → NCCL
custom fused operations → CUDA or Triton kernels

You usually do not manually write CUDA while developing an ordinary model.

But understanding CUDA's execution model explains why some AI workloads are dramatically faster than others.

5. Host and device

CUDA traditionally distinguishes between:

Host

and:

Device

The host is usually the CPU.

The device is the GPU.

For example:

CPU RAM

     data transfer

GPU VRAM

If your tensor exists on the CPU:

x = torch.randn(1000, 1000)

and your model exists on the GPU:

model.cuda()

you cannot directly perform:

model(x)

because the data and model live on different devices.

You usually need:

x = x.cuda()

or:

x = x.to("cuda")

This transfer is not free.

Moving data between CPU and GPU can become a serious bottleneck.

That is why efficient training pipelines try to avoid unnecessary host-device transfers.

6. Kernel

A kernel is a function executed on the GPU.

For example, imagine:

__global__ void add(
    float* A,
    float* B,
    float* C
) {
    int i = threadIdx.x;

    C[i] = A[i] + B[i];
}

Conceptually, this kernel performs:

C[i] = A[i] + B[i]

for many values of i.

Instead of running the function once, CUDA launches many copies of the function across GPU threads.

This process is called:

kernel launch

A deep-learning model may execute many GPU kernels per forward pass.

For example:

matrix multiplication kernel
bias kernel
activation kernel
dropout kernel
normalization kernel
attention kernel

A major optimization technique is therefore:

reduce unnecessary kernel launches.

This leads to an important idea:

kernel fusion

Instead of:

kernel 1 → add bias
kernel 2 → apply activation
kernel 3 → dropout

we may create:

one fused kernel

that performs everything together.

Benefits can include:

fewer launches
fewer global-memory reads
fewer global-memory writes
less overhead
better GPU utilization

This idea is extremely important in modern LLM optimization.

7. Threads, blocks, and grids

CUDA organizes computation using a hierarchy:

Grid
 └── Blocks
      └── Threads

Suppose you launch:

256 threads per block
4096 blocks

Then the logical kernel launch contains:

256×4096=1,048,576256 \times 4096 = 1,048,576

threads.

The hierarchy looks like:

Grid

Block 0
 ├── Thread 0
 ├── Thread 1
 ├── ...
 └── Thread 255

Block 1
 ├── Thread 0
 ├── Thread 1
 └── ...

...

Threads within the same block can cooperate efficiently.

They can:

access shared memory
synchronize
exchange intermediate information

Threads belonging to different blocks generally cannot cheaply synchronize during ordinary kernel execution.

That difference strongly influences CUDA algorithm design.

8. Why so many threads?

A common misunderstanding is:

"My GPU only has a few thousand CUDA cores. Why launch millions of threads?"

Because CUDA threads are not equivalent to physical processor cores.

Threads represent work.

The hardware schedules them onto physical execution units.

Suppose a GPU can run some number of threads simultaneously.

When one group waits on memory, another group can execute.

This helps hide latency.

This concept is called:

latency hiding

The GPU keeps many threads available so that compute units remain productive even while some work is waiting.

9. Warps

GPU threads are not executed completely independently.

On NVIDIA GPUs, threads are scheduled in groups called:

warps

A warp traditionally contains:

32 threads

Conceptually:

Warp

Thread 0
Thread 1
Thread 2
...
Thread 31

The GPU generally executes instructions for these threads together.

For example:

all 32 threads perform multiply
all 32 threads perform add
all 32 threads perform load

This leads us to SIMT.

10. SIMD vs SIMT

CPU vectorization is often described using:

SIMD

which means:

Single Instruction, Multiple Data

For example:

add 8 floating-point numbers simultaneously

The same instruction operates across multiple values.

GPUs use a related model called:

SIMT

or:

Single Instruction, Multiple Threads

Each thread has its own logical state.

But groups of threads execute instructions together.

Conceptually:

Thread 0: C[0] = A[0] + B[0]
Thread 1: C[1] = A[1] + B[1]
Thread 2: C[2] = A[2] + B[2]
...

The threads appear independent to the programmer.

Internally, the GPU efficiently executes them in groups.

11. Warp divergence

SIMT becomes inefficient when threads inside one warp take different branches.

Consider:

if (threadIdx.x % 2 == 0) {
    do_work_A();
} else {
    do_work_B();
}

Half the warp wants:

and half wants:

The GPU may effectively need to execute both paths while masking inactive threads.

Conceptually:

execute A
threads 0,2,4,... active

execute B
threads 1,3,5,... active

This is called:

warp divergence

Divergence reduces parallel efficiency.

In deep learning, optimized kernels try to arrange work so threads cooperate predictably.

12. Memory hierarchy

GPU performance is not determined only by computational power.

Very frequently, performance is determined by:

memory

A simplified GPU memory hierarchy is:

Registers
   ↓
Shared Memory / L1 Cache
   ↓
L2 Cache
   ↓
Global GPU Memory

As we move downward:

capacity increases
latency increases

Very roughly:

Registers
smallest, fastest

Shared memory
small, extremely fast

L1/L2 cache
larger

Global memory
huge, much slower

This is one of the most important ideas in GPU programming:

Computation is cheap compared with repeatedly moving data.

13. Registers

Registers are extremely fast storage located very close to GPU execution units.

Threads use registers for local values.

Example:

float a;
float b;
float result;

These values may live in registers.

Registers are fast but limited.

If a kernel uses too many registers per thread, fewer threads can run simultaneously.

This can reduce:

occupancy

So surprisingly:

using too many fast resources can reduce overall performance.

14. Shared memory

Shared memory is a small, fast memory region shared by threads within one block.

Imagine matrix multiplication.

Naively, every thread may repeatedly fetch values from global memory.

Instead, threads can cooperatively load useful matrix tiles into shared memory.

Example:

Global Memory
     ↓
load matrix tile
     ↓
Shared Memory
     ↓
many threads reuse it

This reduces expensive global-memory traffic.

A simplified tiled matrix multiplication does something like:

1. Load small tile of A into shared memory
2. Load small tile of B into shared memory
3. Synchronize threads
4. Perform many calculations using cached tiles
5. Load next tiles
6. Repeat

This idea is central to high-performance matrix multiplication.

15. Global memory

Global memory usually means the GPU's large external memory:

VRAM

Examples include:

GDDR6
HBM2
HBM3
HBM3e

Global memory is much larger than registers or shared memory.

But it is also much slower.

Therefore efficient GPU algorithms try to:

load once
reuse many times

rather than:

load
compute
write
load again
compute
write

This concept is sometimes described using:

data locality

16. Memory bandwidth

Memory bandwidth measures how much data can be transferred per second.

Example:

1 TB/s
3 TB/s
5 TB/s

A GPU may have extraordinary compute throughput but still sit idle if data cannot reach the compute units quickly enough.

This creates two broad categories of workloads:

compute-bound

and:

memory-bound

A compute-bound operation is limited primarily by arithmetic throughput.

A memory-bound operation is limited primarily by data movement.

For example, large matrix multiplication often has high arithmetic intensity and can be compute-heavy.

Something like:

Y = X + 1

may be much more memory-bound.

Why?

Because for every element we might:

load X
perform one addition
store Y

Very little computation happens relative to memory traffic.

17. Arithmetic intensity

Arithmetic intensity measures roughly:

computationdata movement\frac{\text{computation}} {\text{data movement}}

For example:

1000 operations for each byte fetched

has high arithmetic intensity.

Meanwhile:

1 operation for every several bytes fetched

has low arithmetic intensity.

High arithmetic-intensity workloads are more likely to fully utilize computational hardware.

Matrix multiplication is powerful partly because data can be reused many times.

18. The roofline model

A useful performance model in high-performance computing is the:

Roofline Model

The basic intuition is that performance is constrained by whichever resource runs out first:

compute throughput

or:

memory bandwidth

Conceptually:

Performance=min⁡(Peak Compute,Bandwidth×Arithmetic Intensity)Performance = \min( Peak\ Compute, Bandwidth \times Arithmetic\ Intensity )

This tells us something important.

If your kernel is memory-bound, getting a GPU with twice as many compute units may barely help.

If your kernel is compute-bound, faster memory may not solve the problem.

Performance engineering begins by determining what resource is actually limiting the workload.

19. Coalesced memory access

GPUs perform best when neighboring threads access nearby memory locations.

Good:

Thread 0 → A[0]
Thread 1 → A[1]
Thread 2 → A[2]
Thread 3 → A[3]

Poorer:

Thread 0 → A[0]
Thread 1 → A[10000]
Thread 2 → A[73]
Thread 3 → A[850000]

The first access pattern allows memory transactions to be combined efficiently.

This is known as:

coalesced memory access

Memory layout therefore matters enormously.

This is partly why tensor shapes, contiguous tensors, strides, transpose operations, and layout changes matter for AI performance.

20. Tensor Cores

Modern NVIDIA GPUs contain specialized hardware called:

Tensor Cores

Tensor Cores are specifically designed to accelerate matrix operations.

Instead of executing ordinary scalar operations individually, tensor cores perform small matrix multiply-accumulate operations extremely efficiently.

Conceptually:

D=A×B+CD = A \times B + C

on small matrix tiles.

These operations are fundamental to neural networks.

Tensor cores support numerical formats such as:

FP16
BF16
TF32
FP8
INT8

depending on GPU generation.

This is one reason lower-precision deep learning became so important.

21. CUDA cores vs Tensor Cores

CUDA cores are general arithmetic execution units.

Tensor Cores are specialized matrix-processing units.

An oversimplified analogy:

CUDA Core:
general-purpose calculator

Tensor Core:
specialized matrix multiplication engine

If you are multiplying enormous matrices, tensor cores can provide dramatically greater throughput.

Modern deep-learning libraries therefore structure matrix operations so Tensor Cores can be used whenever possible.

22. Tensor shapes affect speed

An operation mathematically equivalent to another operation may run much slower because its dimensions do not align well with hardware.

For example, tensor cores often work efficiently with dimensions divisible by particular values such as:

depending on data type, kernel, and hardware.

This means:

hidden size = 4096

may be more hardware-friendly than an arbitrary size like:

hidden size = 4091

This is one reason neural-network architectures frequently use dimensions that look suspiciously like powers of two or highly divisible numbers.

Hardware affects architecture.

23. Batching

Imagine processing one image:

image → GPU → result

The GPU might not have enough parallel work to stay busy.

Instead we can process:

64 images

simultaneously.

That is batching.

Batch

Image 1
Image 2
Image 3
...
Image 64

A batch lets large matrix operations replace many small operations.

This usually increases:

GPU utilization
throughput

24. Throughput vs latency

Batching introduces an important distinction.

Latency

How long does one request take?

Example:

50 ms/request

Throughput

How many requests can be processed per second?

Example:

1000 requests/second

Increasing batch size often improves throughput.

But it can hurt individual request latency.

For training, maximizing throughput is often desirable.

For interactive LLM serving, latency may be extremely important.

This creates engineering trade-offs.

25. Batch size and training

Suppose:

batch size = 1

The GPU performs relatively small operations.

With:

batch size = 128

we often obtain much larger matrix multiplications.

This can improve GPU utilization.

However, batch size affects:

memory consumption
optimizer behavior
gradient noise
generalization
training stability
learning-rate scaling

Therefore the largest possible batch is not automatically the best batch.

26. Microbatching

Suppose the desired batch size is:

but GPU memory only supports:

32 samples

at once.

We can process eight microbatches:

32 × 8 = 256

and accumulate gradients.

This is:

gradient accumulation

Example:

optimizer.zero_grad()

for _ in range(8):
    loss = model(batch)
    loss = loss / 8
    loss.backward()

optimizer.step()

The effective batch size becomes larger than the physical batch loaded at one time.

27. Precision

A number in a neural network must be represented using bits.

Common formats include:

FP32
FP16
BF16
FP8
INT8
INT4

FP means:

floating point

INT means:

integer

FP32 uses 32 bits per value.

FP16 uses 16 bits.

Therefore:

1 billion FP32 parameters
≈ 4 GB

1 billion FP16 parameters
≈ 2 GB

ignoring additional training state.

Lower precision reduces memory and can increase computational throughput.

28. FP32

FP32 is standard 32-bit floating point.

Conceptually, it stores:

sign
exponent
mantissa

with enough precision and dynamic range for general numerical computing.

Historically, most deep-learning training used FP32.

But many neural-network operations do not need all 32 bits of precision.

This led to mixed-precision training.

29. FP16

FP16 uses 16 bits.

It offers:

lower memory usage
higher Tensor Core throughput
faster memory movement

But it has reduced numerical range and precision.

Certain values can:

overflow

or:

underflow

This can destabilize training.

That is why FP16 training historically used techniques such as:

loss scaling

30. BF16

BF16 means:

Brain Floating Point 16

BF16 also uses 16 bits, but its bit distribution differs from FP16.

Its major advantage is that it has approximately FP32's exponent range.

That makes it less susceptible to overflow.

For large neural networks, BF16 is often easier to train with than FP16.

Simplified comparison:

FP16
more mantissa precision
smaller exponent range

BF16
less mantissa precision
much larger exponent range

Modern large-model training commonly uses BF16 when supported.

31. Mixed precision

Mixed precision means different parts of training use different numerical precisions.

For example:

matrix multiplication → BF16
accumulation          → FP32
optimizer states      → FP32

The goal is:

use low precision where it is safe and fast, while retaining high precision where numerical stability matters.

PyTorch provides automatic mixed precision.

Example:

with torch.autocast(
    device_type="cuda",
    dtype=torch.bfloat16
):
    output = model(x)

Mixed precision offers:

less memory
faster tensor-core computation
higher throughput

while usually retaining model quality.

32. TF32

NVIDIA introduced:

TensorFloat-32

or TF32.

TF32 is designed to accelerate certain FP32 matrix operations on Tensor Cores.

From a programmer's perspective, you may still be using FP32 tensors.

But internally, compatible matrix calculations can use a reduced-precision representation optimized for tensor cores.

TF32 became particularly important on Ampere-generation GPUs.

33. FP8

Newer accelerators increasingly support:

FP8

FP8 uses only 8 bits per value.

This further reduces:

memory usage
communication bandwidth
storage bandwidth

and can dramatically increase Tensor Core throughput.

But FP8 introduces more numerical challenges.

Large-scale training systems therefore use carefully designed scaling strategies.

FP8 is increasingly important for frontier-model training and inference.

34. Training memory is much larger than model size

Suppose a model contains:

1 billion parameters

At FP16:

weights ≈ 2 GB

You might assume:

"Then a 4 GB GPU should train it."

Usually not.

Training also requires storing:

weights
gradients
optimizer states
activations
temporary buffers

With Adam, optimizer states alone can be very large.

For each parameter, Adam often tracks:

first moment
second moment

These may be stored in FP32.

So total training memory can be many times larger than raw model weights.

This is a major reason distributed-training techniques exist.

35. Activations

During the forward pass, neural networks generate intermediate values called:

activations

These are often saved because backpropagation needs them.

For deep models:

input
↓
layer 1 activation
↓
layer 2 activation
↓
...
↓
layer 100 activation

Activation memory can become enormous.

And unlike parameter memory, activation memory generally grows with:

batch size
sequence length
hidden dimension
number of layers

This becomes especially important with long-context transformers.

36. Activation checkpointing

Instead of storing every activation, we can discard some of them.

During backward propagation, we recompute those activations.

This is called:

activation checkpointing

or:

gradient checkpointing

Trade-off:

less memory
more computation

For large models, this is often an excellent trade.

GPU compute may be cheaper than additional memory capacity.

37. High-performance computing

High-performance computing, or:

HPC

refers to using powerful computing systems to solve computationally intensive problems.

Traditional HPC includes:

weather simulation
computational fluid dynamics
molecular simulation
astrophysics
nuclear simulation
genomics

Modern AI has effectively become a major form of HPC.

Training frontier models requires:

large GPU clusters
high-speed networking
distributed storage
parallel algorithms
fault tolerance
performance profiling
distributed scheduling

So today's serious AI infrastructure overlaps heavily with supercomputing.

38. One GPU is eventually not enough

Suppose a model requires:

300 GB

just for useful training state.

But one GPU has:

80 GB

of memory.

The model cannot fit on a single device.

Even if it did fit, training may take years.

So we distribute the workload across:

multiple GPUs

and eventually:

multiple machines

This is distributed training.

39. Distributed training

Suppose we have:

8 GPUs

There are several ways we can divide the work.

The major strategies are:

data parallelism
model parallelism
tensor parallelism
pipeline parallelism

Large systems often combine several simultaneously.

This produces:

hybrid parallelism

40. Data parallelism

Data parallelism is the simplest distributed-training strategy.

Each GPU receives:

a complete copy of the model

but processes different data.

Example:

GPU 0
Model copy
Batch A

GPU 1
Model copy
Batch B

GPU 2
Model copy
Batch C

GPU 3
Model copy
Batch D

Each GPU computes gradients independently.

Then the gradients are synchronized.

Conceptually:

GPU 0 gradients ─┐
GPU 1 gradients ─┤
GPU 2 gradients ─┼→ combine → same model update
GPU 3 gradients ─┘

This is extremely common.

PyTorch provides:

DistributedDataParallel

commonly called:

DDP

41. Why gradients must synchronize

Suppose GPU 0 sees:

batch A

and computes:

gradient A

GPU 1 sees:

batch B

and computes:

gradient B

If both independently update their models, the models diverge.

Instead we combine:

g=g0+g1+⋯+gnng = \frac{g_0 + g_1 + \cdots + g_n}{n}

Then every GPU applies the same gradient.

Now the replicas stay synchronized.

This operation commonly uses:

AllReduce

42. Communication collectives

Distributed computing repeatedly performs operations involving groups of processes.

These are called:

collective communication operations

Important collectives include:

Broadcast
Reduce
AllReduce
AllGather
ReduceScatter
Scatter
Gather

These concepts appear constantly in distributed AI systems.

43. Broadcast

One process has data.

Everyone needs it.

GPU 0
  ↓
  ├──── GPU 1
  ├──── GPU 2
  └──── GPU 3

This is:

Broadcast

Example use:

sending initial model parameters

44. Reduce

Every process has data.

We combine the values at one destination.

Example:

GPU 0: 2
GPU 1: 3
GPU 2: 5
GPU 3: 7

Reduction using sum:

2 + 3 + 5 + 7 = 17

The result may end up on GPU 0.

45. AllReduce

AllReduce combines everyone's values and distributes the result back to everyone.

Input:

GPU 0: 2
GPU 1: 3
GPU 2: 5
GPU 3: 7

After AllReduce sum:

GPU 0: 17
GPU 1: 17
GPU 2: 17
GPU 3: 17

This is heavily used for gradient synchronization.

46. AllGather

Suppose each GPU owns a piece:

GPU 0 → A
GPU 1 → B
GPU 2 → C
GPU 3 → D

After AllGather:

GPU 0 → ABCD
GPU 1 → ABCD
GPU 2 → ABCD
GPU 3 → ABCD

This becomes extremely important in tensor and sharded parallelism.

47. ReduceScatter

ReduceScatter combines values and distributes different portions of the reduced output across devices.

Conceptually:

Reduce
+
Scatter

It is useful because distributed training frequently does not need every GPU to store the complete final result.

Libraries can reduce communication and memory consumption by combining collectives intelligently.

48. NCCL

For NVIDIA GPU systems, a key library is:

NCCL

NVIDIA Collective Communications Library.

NCCL provides highly optimized GPU communication primitives such as:

AllReduce
AllGather
Broadcast
ReduceScatter

A distributed PyTorch program may invoke NCCL beneath the surface.

Conceptually:

PyTorch Distributed
        ↓
       NCCL
        ↓
NVLink / PCIe / InfiniBand
        ↓
other GPUs

49. Communication is expensive

Imagine each GPU can perform:

hundreds of trillions of operations per second

But every training step requires enormous parameter or gradient transfers.

Eventually GPUs may spend time waiting.

Instead of:

compute
compute
compute

we get:

compute
wait for communication
compute
wait
compute

At large scale, communication optimization becomes just as important as computation optimization.

50. PCIe, NVLink, and networking

Communication speed depends heavily on hardware topology.

Different links can include:

PCI Express
NVLink
NVSwitch
InfiniBand
Ethernet

Inside one server, GPUs may communicate over PCIe or NVLink.

Across machines, networking such as InfiniBand may be used.

A simplified hierarchy:

GPU
 │
NVLink
 │
GPU
 │
NVSwitch
 │
other GPUs in node
 │
InfiniBand NIC
 │
other servers

Moving a tensor to a neighboring GPU connected by NVLink may be very different from moving it to a GPU several network hops away.

Therefore:

distributed training performance depends on physical hardware topology.

51. Data-parallel limitation

Data parallelism requires every GPU to store a complete model.

Suppose:

model training state = 200 GB

and each GPU has:

80 GB

Then ordinary data parallelism cannot solve the memory problem.

Each GPU still needs 200 GB.

We now need to partition the model itself.

This introduces model parallelism.

52. Model parallelism

Model parallelism means different GPUs own different parts of the model.

Example:

GPU 0
Layers 1–20

GPU 1
Layers 21–40

GPU 2
Layers 41–60

GPU 3
Layers 61–80

Input flows through the devices:

Input
  ↓
GPU 0
  ↓
GPU 1
  ↓
GPU 2
  ↓
GPU 3
  ↓
Output

This lets models larger than one GPU's memory be trained.

But naive model parallelism can waste enormous amounts of hardware.

While GPU 0 works, GPU 3 may be waiting.

That leads to pipeline parallelism.

53. Pipeline parallelism

Suppose:

GPU 0 → layers 1–20
GPU 1 → layers 21–40
GPU 2 → layers 41–60
GPU 3 → layers 61–80

Instead of processing an entire batch as one unit, we divide it into microbatches.

For example:

Microbatch A
Microbatch B
Microbatch C
Microbatch D

Then:

Time →

GPU 0: A B C D
GPU 1:   A B C D
GPU 2:     A B C D
GPU 3:       A B C D

Now different GPUs can work simultaneously on different microbatches.

This resembles an assembly line.

Hence:

pipeline parallelism

54. Pipeline bubbles

Pipeline parallelism has periods when GPUs are idle.

At startup:

GPU 3

cannot do anything until earlier pipeline stages have processed the first microbatch.

Likewise there may be idle periods near the end.

These gaps are known as:

pipeline bubbles

The goal of good pipeline scheduling is to minimize these bubbles.

More microbatches can improve pipeline utilization, although communication and scheduling complexity increase.

55. Tensor parallelism

Tensor parallelism splits an individual mathematical operation across multiple GPUs.

Suppose a transformer layer performs:

Y=XWY = XW

Instead of keeping all of WW on one GPU, split it:

W = [W1 | W2]

Then:

GPU 0 computes XW1
GPU 1 computes XW2

Later the results are combined.

So rather than splitting:

layers

we split:

tensors inside the layer

This is tensor parallelism.

56. Row and column parallelism

Matrix multiplication can be partitioned in multiple ways.

Suppose:

Y=XWY = XW

We can split WW by columns:

W = [W1 W2]

Then:

GPU 0 → XW1
GPU 1 → XW2

The results are concatenated.

Or split by rows:

W =
[ W1
  W2 ]

which requires appropriately splitting or communicating portions of the input/output.

Frameworks such as Megatron-LM use carefully designed tensor-parallel transformer layers.

57. Why tensor parallelism requires fast interconnects

Tensor parallelism introduces communication inside individual layers.

A transformer may have dozens or hundreds of layers.

If every layer requires inter-GPU communication, network speed becomes critical.

Therefore tensor-parallel GPUs are often placed within a single high-bandwidth node using:

NVLink
NVSwitch

rather than being arbitrarily spread across slow network connections.

This demonstrates an important principle:

Parallelism strategy must match hardware topology.

58. Sequence parallelism

Transformers introduce another dimension that can sometimes be partitioned:

sequence length

Suppose:

sequence length = 32,768

Different GPUs may handle different portions of sequence-related computation.

Various architectures and systems use forms of:

sequence parallelism
context parallelism

These techniques become increasingly relevant as context windows grow.

59. Expert parallelism

Mixture-of-Experts models introduce another form of parallelism.

Suppose a layer has:

64 experts

Different GPUs may store different experts.

A router sends tokens to selected experts.

Example:

Token A → Expert 4 → GPU 1
Token B → Expert 27 → GPU 5
Token C → Expert 4 → GPU 1

This is often called:

expert parallelism

It can make models enormous while activating only a subset of parameters for each token.

But it creates significant communication challenges.

60. Hybrid parallelism

Frontier-model training rarely uses just one parallelism technique.

A large cluster may use:

data parallelism
×
tensor parallelism
×
pipeline parallelism
×
context parallelism
×
expert parallelism

Suppose:

4096 GPUs

A hypothetical configuration could be:

tensor parallel = 8
pipeline parallel = 16
data parallel = 32

Then:

8×16×32=40968 \times 16 \times 32 = 4096

GPUs.

Each dimension solves a different problem.

Tensor parallelism
splits operations.

Pipeline parallelism
splits layers.

Data parallelism
splits training samples.

Understanding this hierarchy is essential for large-scale AI systems.

61. Fully Sharded Data Parallel

Traditional DDP replicates:

parameters
gradients
optimizer states

on every GPU.

This wastes memory when models become large.

Fully Sharded Data Parallel, or:

FSDP

shards these states across GPUs.

Conceptually:

GPU 0 → 1/4 weights
GPU 1 → 1/4 weights
GPU 2 → 1/4 weights
GPU 3 → 1/4 weights

When a layer needs its parameters, GPUs temporarily gather the required pieces.

Then the parameters can be released or reshared afterward.

This trades:

communication

for:

dramatically lower memory usage

62. ZeRO

DeepSpeed introduced a family of techniques called:

ZeRO

Zero Redundancy Optimizer.

The idea is to eliminate unnecessary replicated training state.

Simplified:

ZeRO Stage 1
shard optimizer states

ZeRO Stage 2
shard optimizer states + gradients

ZeRO Stage 3
shard optimizer states + gradients + parameters

ZeRO Stage 3 is conceptually related to fully sharded training.

These techniques made extremely large-model training much more memory-efficient.

63. Scaling efficiency

Suppose:

1 GPU trains at 100 samples/sec

Perfect scaling would mean:

2 GPUs → 200
4 GPUs → 400
8 GPUs → 800

Real systems do not scale perfectly.

Suppose:

8 GPUs → 650 samples/sec

Scaling efficiency is approximately:

6508×100=81.25%\frac{650}{8 \times 100} = 81.25\%

Why is scaling imperfect?

Because GPUs spend time on:

communication
synchronization
load imbalance
kernel overhead
data loading
pipeline bubbles

At large scale, optimizing these losses becomes a major engineering discipline.

64. Strong scaling vs weak scaling

HPC often distinguishes two scaling ideas.

Strong scaling

Keep the total problem size fixed.

Add more GPUs.

Example:

same dataset
same batch
1 GPU → 10 hours
8 GPUs → ideally 1.25 hours

Weak scaling

Increase the workload as hardware increases.

Example:

1 GPU → batch 64
8 GPUs → batch 512

while aiming for similar step time.

Large-scale AI training often relies heavily on weak scaling.

65. Communication-computation overlap

Suppose a GPU performs:

layer 100 backward

while gradients from:

layer 110

are being transferred.

Then communication and computation overlap.

Instead of:

compute
wait
communicate
wait
compute

we want:

compute
████████████████████

communication
      ███████████████

This overlap can dramatically improve distributed-training efficiency.

Frameworks often bucket gradients specifically to enable communication to begin before backward propagation finishes.

66. Synchronization

GPUs occasionally need to wait for each other.

This is synchronization.

Imagine four GPUs:

GPU 0 finished
GPU 1 finished
GPU 2 finished
GPU 3 still computing

If the next operation requires all four:

GPU 0 waits
GPU 1 waits
GPU 2 waits
GPU 3 computes

One slow worker can delay everyone.

This is called a:

straggler

At large cluster scale, tiny imbalances can become expensive.

67. Load balancing

Ideally every GPU receives roughly equal work.

Bad:

GPU 0 → 100 ms
GPU 1 → 101 ms
GPU 2 → 99 ms
GPU 3 → 170 ms

Everyone may effectively run at:

170 ms

because synchronization waits for GPU 3.

Load balancing becomes especially difficult with:

Mixture-of-Experts
variable sequence lengths
dynamic routing
heterogeneous hardware

68. GPU utilization

People frequently ask:

"Why does nvidia-smi say my GPU is at 40%?"

GPU utilization is only one rough metric.

Low utilization may occur because:

CPU cannot feed data fast enough
batch is too small
model is too small
kernels are tiny
communication dominates
storage is slow
Python overhead is high
GPU waits for synchronization
kernel is memory-bound

Therefore:

GPU utilization ≠ complete performance diagnosis

You need profiling.

69. Profiling

Profiling means measuring where time and resources are actually being spent.

Without profiling, optimization is guessing.

Questions we may ask:

Which kernels consume the most GPU time?

Is matrix multiplication efficient?

Are tensor cores active?

Is the CPU starving the GPU?

Are dataloaders slow?

Is communication dominating?

Are kernels memory-bound?

Are there synchronization stalls?

How much time is spent in attention?

How much memory is allocated?

Why is one rank slower than the others?

Profiling transforms vague performance problems into measurable ones.

70. PyTorch profiler

PyTorch includes a profiler.

A simplified example:

import torch
from torch.profiler import (
    profile,
    ProfilerActivity
)

with profile(
    activities=[
        ProfilerActivity.CPU,
        ProfilerActivity.CUDA
    ]
) as prof:

    output = model(x)

print(
    prof.key_averages().table(
        sort_by="cuda_time_total"
    )
)

This can show operations such as:

aten::matmul
aten::linear
aten::softmax
cuda kernels
memory copies

You can identify which operations consume the most time.

71. NVIDIA Nsight Systems

NVIDIA provides:

Nsight Systems

which is extremely useful for timeline-level analysis.

You may see something conceptually like:

CPU
| preprocessing | launch | launch | waiting |

GPU
        | GEMM | softmax | GEMM | NCCL |

Network
                             | AllReduce |

This reveals:

CPU/GPU overlap
kernel-launch gaps
communication phases
synchronization
idle periods

For distributed systems, timeline visualization is extremely valuable.

72. Nsight Compute

Another NVIDIA tool is:

Nsight Compute

Nsight Compute focuses much more deeply on individual GPU kernels.

It can inspect:

memory bandwidth
occupancy
warp efficiency
instruction throughput
cache hit rate
Tensor Core utilization
register usage
shared-memory usage

Think of the difference roughly as:

Nsight Systems
Where did the time go?

Nsight Compute
Why is this specific kernel slow?

73. Occupancy

Occupancy roughly describes how many warps are active relative to what an SM could support.

Higher occupancy can help hide memory latency.

But:

100% occupancy

does not automatically mean maximum performance.

A kernel can have lower occupancy yet run faster because it uses:

more registers
better data reuse
fewer memory accesses

Occupancy is a diagnostic metric, not the ultimate goal.

The goal is:

application throughput

74. Kernel launch overhead

Launching GPU work has overhead.

Suppose a kernel does almost no computation:

5 microseconds of useful work

while launch and scheduling costs are significant relative to it.

Launching thousands of tiny kernels can become inefficient.

This is one reason:

operator fusion
torch.compile
CUDA graphs
Triton kernels
FlashAttention

can produce large speedups.

They reduce fragmentation of GPU work.

75. CUDA Graphs

Normally the CPU repeatedly launches GPU operations:

launch A
launch B
launch C
launch D

For workloads with a stable execution pattern, CUDA Graphs can capture the sequence.

Then the GPU work can be replayed with lower CPU launch overhead.

Conceptually:

Normal:

CPU → A
CPU → B
CPU → C
CPU → D

CUDA Graph:

CPU → replay graph [A B C D]

This can improve performance for workloads where launch overhead matters.

76. `torch.compile`

Modern PyTorch includes compilation mechanisms through:

torch.compile(model)

The system can analyze operations and generate more optimized execution.

Potential benefits include:

kernel fusion
fewer Python transitions
better memory behavior
specialized kernels
reduced launch overhead

Underneath, components such as TorchDynamo, AOTAutograd, and Inductor may be involved.

The general idea is:

instead of executing every PyTorch operation independently, optimize a larger computation graph.

77. Triton

Triton is a language and compiler designed for writing high-performance GPU kernels more easily than traditional CUDA in many situations.

Instead of manually controlling every low-level CUDA detail, developers write array-oriented kernels.

Modern AI systems increasingly use Triton for specialized operations.

Examples include:

normalization kernels
softmax
attention components
quantization kernels
fused operators

Triton has become particularly important in PyTorch compiler infrastructure and AI kernel research.

78. FlashAttention

Attention is conceptually:

Attention(Q,K,V)=softmax(QKTd)VAttention(Q,K,V) = softmax \left( \frac{QK^T} {\sqrt{d}} \right)V

A naive implementation may create the full attention matrix in GPU memory.

For long sequences, this matrix is enormous.

Traditional intuition might say:

"We need fewer arithmetic operations."

FlashAttention demonstrated something deeper:

memory movement can matter more than arithmetic count.

FlashAttention reorganizes the computation into tiles that fit efficiently into faster GPU memory.

Instead of repeatedly moving enormous matrices through global memory, it performs computation in blocks.

This reduces memory traffic dramatically.

The result can be:

faster
less memory
mathematically equivalent attention

FlashAttention is one of the best modern examples of hardware-aware AI algorithm design.

79. Memory-efficient algorithms

Modern AI optimization increasingly asks not only:

How many FLOPs?

but:

How many bytes moved?

An algorithm may perform slightly more arithmetic but still run faster if it dramatically reduces expensive memory traffic.

This is an important mindset shift.

For modern accelerators:

data movement is frequently more expensive than computation.

80. FLOPs

FLOP means:

Floating Point Operation

For example:

a + b

is approximately one floating-point operation.

A GPU advertised as:

500 TFLOPS

can theoretically execute approximately:

500×1012500 \times 10^{12}

floating-point operations per second for a particular numerical format and workload.

However, peak FLOPs are theoretical.

Real application performance depends on:

kernel efficiency
memory bandwidth
tensor dimensions
precision
communication
utilization

81. FLOPS vs FLOPs

These two are easy to confuse.

FLOP
floating-point operation

FLOPs
number of floating-point operations

FLOPS
floating-point operations per second

For example:

Model training requires
1 × 10^23 FLOPs.

Machine throughput is
1 × 10^15 FLOPS.

82. Model FLOP utilization

Large-model researchers sometimes measure:

Model FLOP Utilization

or:

MFU

The rough idea is:

What percentage of the accelerator's theoretical useful compute throughput is the model actually achieving?

Suppose a cluster theoretically provides:

100 PFLOPS

but your workload effectively achieves:

45 PFLOPS

Then utilization is roughly:

45%

MFU is useful for comparing training-system efficiency.

83. Why peak GPU numbers can mislead you

A GPU specification may advertise extremely high throughput, such as:

1000+ TFLOPS

But that number might correspond to:

FP8 Tensor Core throughput

under ideal conditions.

Your workload may instead be:

FP32
memory-bound
poorly shaped
communication-heavy

So you might achieve only a fraction of peak.

Always ask:

Peak for which precision?
Peak for which operation?
Peak dense or sparse?
Peak Tensor Core or CUDA core?
Real workload or theoretical maximum?

84. Data loading can bottleneck the GPU

Suppose the GPU can train one batch in:

20 ms

but the CPU requires:

40 ms

to load and preprocess the next batch.

Then the GPU waits.

Timeline:

GPU
compute █████
idle        ██████████
compute                █████

The GPU may be extremely powerful but underutilized because the input pipeline is slow.

Possible improvements include:

multiple DataLoader workers
pinned memory
prefetching
cached datasets
faster storage
GPU-side preprocessing
asynchronous transfer

85. Pinned memory

Normally CPU memory can be pageable.

Pinned memory means the operating system keeps the memory resident and suitable for efficient DMA transfers to the GPU.

In PyTorch:

DataLoader(
    dataset,
    pin_memory=True
)

can improve CPU-to-GPU transfer efficiency.

Then:

x = x.to(
    "cuda",
    non_blocking=True
)

can enable asynchronous transfers in suitable situations.

86. Streams

CUDA streams represent ordered sequences of GPU operations.

Operations inside one stream execute in order.

Different streams may overlap.

For example:

Stream A
compute batch 1

Stream B
transfer batch 2

This lets us overlap:

computation

with:

data transfer

when hardware and dependencies allow.

Modern frameworks use streams extensively behind the scenes.

87. Asynchronous execution

GPU execution is frequently asynchronous.

Consider:

y = model(x)
print("done")

The CPU may submit GPU operations and continue without waiting for every GPU instruction to finish.

This is important for benchmarking.

Bad timing:

start = time.time()
y = model(x)
end = time.time()

may measure mostly CPU launch time.

A correct benchmark may require synchronization:

torch.cuda.synchronize()

start = time.time()

y = model(x)

torch.cuda.synchronize()

end = time.time()

Otherwise GPU execution may still be in progress when the timer stops.

88. Warm-up

GPU programs often run slower the first few iterations.

Reasons include:

kernel initialization
memory allocation
JIT compilation
library autotuning
cache population

Therefore serious benchmarking usually includes:

warm-up iterations

before measuring steady-state speed.

Example:

for _ in range(20):
    model(x)

torch.cuda.synchronize()

# benchmark after warm-up

89. Memory fragmentation

GPU memory may be available overall but split into inconvenient pieces.

Imagine free memory:

1 GB
500 MB
700 MB
300 MB

Total:

2.5 GB

But if an operation needs one contiguous 2 GB block, allocation may fail depending on the allocator and situation.

This is memory fragmentation.

Framework memory allocators attempt to reduce these problems, but long-running dynamic workloads can still encounter them.

90. Out-of-memory errors

When you see:

CUDA out of memory

possible solutions include:

reduce batch size
reduce sequence length
use BF16/FP16
gradient checkpointing
FSDP
ZeRO
CPU offloading
quantization
smaller optimizer state
better memory allocator behavior
free unused tensors

But first determine what is consuming memory.

Do not blindly call:

torch.cuda.empty_cache()

and assume the underlying problem is solved.

91. Quantization

Quantization represents values using fewer bits.

For example:

FP16 → INT8
FP16 → INT4

A 70-billion-parameter model stored at FP16 requires approximately:

70B×2 bytes=140 GB70B \times 2\ bytes = 140\ GB

At INT4:

70B×0.5 bytes≈35 GB70B \times 0.5\ bytes \approx 35\ GB

ignoring metadata and quantization overhead.

That is an enormous difference.

Quantization is therefore crucial for LLM inference.

92. Compute vs bandwidth during LLM inference

LLM inference has two particularly important phases:

prefill
decode

During prefill, many input tokens can be processed in parallel.

Large matrix operations tend to dominate.

During autoregressive decoding, the model often generates one token at a time.

At each step, weights may need to be streamed through the GPU.

Decode can therefore become strongly memory-bandwidth constrained.

This is why:

memory bandwidth
KV-cache design
batching
quantization

matter so much for LLM serving.

93. KV cache

Transformer inference stores attention keys and values from previous tokens.

This is the:

KV cache

Without it, generating token 1000 would require repeatedly recomputing information for tokens 1 through 999.

Instead, prior K/V tensors are cached.

But KV cache grows with:

batch size
sequence length
number of layers
hidden dimension
precision

For long-context models, KV-cache memory can become enormous.

This makes memory management central to LLM inference systems.

94. Continuous batching

Traditional batching might wait for several requests:

Request A
Request B
Request C
Request D

and process them together.

But LLM requests finish at different times.

Modern serving systems often use:

continuous batching

Requests dynamically enter and leave the active batch.

Conceptually:

Step 1:
A B C

Step 2:
A B C

C finishes

Step 3:
A B D

B finishes

Step 4:
A E D

This keeps GPUs highly utilized.

Systems such as modern LLM inference engines heavily exploit this idea.

95. Distributed inference

Very large models may need multiple GPUs even for inference.

Tensor parallelism can split model layers across GPUs.

Pipeline parallelism can split layers into stages.

Expert parallelism can distribute MoE experts.

Serving systems must balance:

latency
throughput
communication
memory usage
batching
fault tolerance

Inference optimization is therefore its own form of HPC.

96. Profiling an AI workload: a practical mental checklist

Suppose training feels slow.

Do not immediately rewrite kernels.

Work from the outside inward.

Ask:

1. Is the GPU busy?

Check:

GPU utilization
memory usage
power usage

2. Is the CPU feeding data fast enough?

Check:

DataLoader
disk I/O
preprocessing
host-to-device transfer

3. Are kernels large enough?

Tiny kernels can lead to poor utilization.

4. Are Tensor Cores being used?

Check:

precision
tensor shapes
library kernels

5. Is the workload compute-bound or memory-bound?

Use profiling and roofline thinking.

6. In distributed training, is communication dominating?

Inspect:

NCCL
AllReduce
AllGather
ReduceScatter

7. Is there load imbalance?

Compare different ranks.

8. Are unnecessary synchronizations happening?

CPU calls such as:

tensor.item()

can sometimes force GPU synchronization.

9. Is memory limiting batch size?

Consider:

mixed precision
checkpointing
sharding

10. Does the architecture map efficiently onto hardware?

Sometimes architecture changes produce bigger improvements than low-level tuning.

97. A concrete training example

Imagine training a transformer on four GPUs.

Each GPU receives:

batch size = 16

So global batch size is:

16×4=6416 \times 4 = 64

Training proceeds roughly like:

Step 1

GPU 0:
forward batch A
backward batch A
gradient g0

GPU 1:
forward batch B
backward batch B
gradient g1

GPU 2:
forward batch C
backward batch C
gradient g2

GPU 3:
forward batch D
backward batch D
gradient g3

Then:

NCCL AllReduce

combines gradients:

g=g0+g1+g2+g34g = \frac{g_0 + g_1 + g_2 + g_3}{4}

All GPUs receive the synchronized result.

Then every GPU runs:

optimizer.step()

Because every model started identical and receives identical updates, they remain identical.

This is ordinary distributed data-parallel training.

98. Now imagine a 500-billion-parameter model

A 500B model cannot simply be copied onto every GPU.

So suppose we build:

8-way tensor parallel groups
8 pipeline stages
64 data-parallel replicas

Then total GPU count is:

8×8×64=40968 \times 8 \times 64 = 4096

Within each model replica:

Tensor parallelism
splits individual layer computation across 8 GPUs.

Pipeline parallelism
splits transformer layers across 8 stages.

Then 64 copies of this distributed model process different training data.

Across those replicas:

data parallelism

synchronizes training state.

This is roughly the kind of multi-dimensional thinking needed for frontier-scale model training.

99. The hidden enemy: communication ratio

Suppose one training step spends:

80 ms computation
20 ms communication

Total:

100 ms

Now double the GPUs.

Maybe computation becomes:

40 ms

but communication rises to:

25 ms

Total:

65 ms

You doubled hardware but only improved performance:

100/65≈1.54×100 / 65 \approx 1.54\times

not:

2×

Scale again.

Eventually communication can dominate.

That is why adding GPUs has diminishing returns unless the system is carefully designed.

100. High-performance AI is a systems problem

At beginner level, AI performance looks like:

buy a faster GPU

At advanced level, performance comes from the interaction of:

model architecture
tensor shapes
numerical precision
kernel implementation
memory hierarchy
batching
parallelism
network topology
data loading
communication
compiler
runtime
storage

The GPU is only one part of the system.

101. The five major bottleneck categories

Most GPU performance problems can be mentally grouped into five categories.

Compute

Are execution units saturated?

Examples:

matrix multiplication throughput
Tensor Core utilization

Memory

Can data reach compute units fast enough?

Examples:

global-memory bandwidth
cache misses
KV-cache traffic

Launch/CPU overhead

Is the GPU waiting because the CPU cannot submit work efficiently?

Examples:

tiny kernels
Python overhead

Communication

Are GPUs waiting for each other?

Examples:

AllReduce
AllGather
network congestion

I/O

Can storage and preprocessing supply data?

Examples:

disk
dataset decoding
tokenization

This classification is extremely useful when debugging performance.

102. Optimization should follow measurement

A common mistake is:

slow training
↓
random optimization
↓
change ten things
↓
no idea what helped

A better approach is:

measure
↓
identify dominant bottleneck
↓
change one meaningful thing
↓
measure again

For example:

Profiler:
47% of iteration is DataLoader waiting

Wrong response:
write custom CUDA kernel

Correct response:
fix input pipeline

Performance engineering should be evidence-driven.

103. Senior-level perspective: think in resource utilization

When reading AI performance papers, stop thinking only in terms of:

layers
parameters
accuracy

Also think:

How many bytes move?

Where do they move?

How often are they reused?

Which device owns them?

What communication collective is needed?

Which hardware unit executes the operation?

What numerical format is used?

Can computation overlap communication?

Is the operation compute-bound or bandwidth-bound?

What happens when GPU count increases?

These questions often reveal why an architecture is fast or slow.

104. Senior-level perspective: algorithms and hardware co-design

The fastest algorithm mathematically is not necessarily the fastest algorithm on real hardware.

For example:

Algorithm A
fewer FLOPs
huge memory traffic

Algorithm B
slightly more FLOPs
excellent memory locality

Algorithm B may run much faster.

Modern AI increasingly practices:

hardware-software co-design

Architectures are influenced by:

tensor-core dimensions
memory hierarchy
communication topology
precision support
parallelism strategy

And hardware is increasingly designed around AI workloads.

The boundary between:

AI researcher
systems engineer
compiler engineer
hardware architect

is becoming increasingly important.

105. Why this matters for AI researchers

Suppose you invent a new attention mechanism that reduces theoretical FLOPs by 20%.

That sounds excellent.

But imagine it:

requires irregular memory access
cannot use Tensor Cores efficiently
creates many tiny kernels
requires expensive synchronization

It may actually train slower than standard attention.

A serious AI researcher should therefore ask both:

Is my algorithm theoretically efficient?

and:

Does it map efficiently onto modern accelerators?

This distinction becomes increasingly important as models scale.

106. A compact mental model to remember

When thinking about a GPU workload, picture this pipeline:

DATA
 ↓
GPU memory
 ↓
caches/shared memory
 ↓
registers
 ↓
CUDA/Tensor cores
 ↓
result

Then for multiple GPUs:

GPU 0
 ↕
high-speed interconnect
 ↕
GPU 1
 ↕
network
 ↕
GPU 2 on another machine

Performance depends on keeping useful data flowing through this system without unnecessary waiting.

107. The three fundamental questions

For almost every GPU optimization problem, ask:

Question 1

Is the GPU doing useful arithmetic?

If not, maybe kernels are too small or the CPU cannot feed it.

Question 2

Is the GPU waiting for memory?

If yes, improve locality, batching, fusion, or reduce precision.

Question 3

Is the GPU waiting for another GPU?

If yes, optimize communication, topology, sharding, or parallelism.

Most HPC reasoning can be traced back to these three questions.

108. Final picture

A modern AI accelerator system looks something like:

                    Training Cluster

            ┌────────────────────────┐
            │       CPU / Host       │
            │ preprocessing / I/O    │
            └───────────┬────────────┘
                        │
                  PCIe / NVLink
                        │
        ┌───────────────▼──────────────┐
        │            GPU               │
        │                              │
        │  Global Memory / HBM         │
        │          ↓                   │
        │       L2 Cache               │
        │          ↓                   │
        │   Streaming Multiprocessors  │
        │                              │
        │   Registers                  │
        │   Shared Memory              │
        │   CUDA Cores                 │
        │   Tensor Cores               │
        └──────────────┬───────────────┘
                       │
              NVLink / NVSwitch
                       │
                Other GPUs
                       │
                  InfiniBand
                       │
                Other Servers

Above all of this sit libraries and frameworks:

PyTorch
TensorFlow
JAX

    ↓

cuBLAS
cuDNN
NCCL
Triton

    ↓

CUDA

    ↓

GPU hardware

And your transformer, diffusion model, robot perception system, RL policy, or multimodal model ultimately runs through this entire stack.

The key things to remember

You do not need to remember every CUDA detail.

The deeper principles are more valuable.

A GPU wins through massive parallelism.

Threads are organized into warps, blocks, and grids.

Warps execute using a SIMT-style execution model.

GPU performance is often limited not by arithmetic, but by memory movement.

Registers and shared memory are tiny but extremely fast.

Global GPU memory is large but relatively expensive to access.

Tensor Cores provide enormous throughput for matrix multiplication using lower-precision numerical formats.

Mixed precision lets AI exploit those Tensor Cores while controlling numerical instability.

Batching creates enough parallel work to keep hardware busy.

Once a model becomes too large or training becomes too slow, work must be distributed.

Data parallelism splits data.

Pipeline parallelism splits layers.

Tensor parallelism splits individual tensor operations.

FSDP and ZeRO split training state.

Large systems combine several kinds of parallelism at once.

And once many GPUs work together, communication becomes part of the algorithm.

That is the real shift from ordinary deep learning into high-performance AI research:

You stop thinking of the neural network as something that simply "runs on a GPU."

You begin thinking of it as a computational graph whose tensors, kernels, memory accesses, numerical formats, and communication patterns must be carefully mapped onto a hierarchy of processors and networks.

That is the systems-level understanding behind modern large-scale AI.

Command Palette

1. CPU vs GPU: the most important intuition

2. Why neural networks love GPUs

3. Simplified GPU architecture

4. CUDA

5. Host and device

6. Kernel

7. Threads, blocks, and grids

8. Why so many threads?

9. Warps

10. SIMD vs SIMT

11. Warp divergence

12. Memory hierarchy

13. Registers

14. Shared memory

15. Global memory

16. Memory bandwidth

17. Arithmetic intensity

18. The roofline model

19. Coalesced memory access

20. Tensor Cores

21. CUDA cores vs Tensor Cores

22. Tensor shapes affect speed

23. Batching

24. Throughput vs latency

Latency

Throughput

25. Batch size and training

26. Microbatching

27. Precision

28. FP32

29. FP16

30. BF16

31. Mixed precision

32. TF32

33. FP8

34. Training memory is much larger than model size

35. Activations

36. Activation checkpointing

37. High-performance computing

38. One GPU is eventually not enough

39. Distributed training

40. Data parallelism

41. Why gradients must synchronize

42. Communication collectives

43. Broadcast

44. Reduce

45. AllReduce

46. AllGather

47. ReduceScatter

48. NCCL

49. Communication is expensive

50. PCIe, NVLink, and networking

51. Data-parallel limitation

52. Model parallelism

53. Pipeline parallelism

54. Pipeline bubbles

55. Tensor parallelism

56. Row and column parallelism

57. Why tensor parallelism requires fast interconnects

58. Sequence parallelism

59. Expert parallelism

60. Hybrid parallelism

61. Fully Sharded Data Parallel

62. ZeRO

63. Scaling efficiency

64. Strong scaling vs weak scaling

Strong scaling

Weak scaling

65. Communication-computation overlap

66. Synchronization

67. Load balancing

68. GPU utilization

69. Profiling

70. PyTorch profiler

71. NVIDIA Nsight Systems

72. Nsight Compute

73. Occupancy

74. Kernel launch overhead

75. CUDA Graphs

76. `torch.compile`