Parallel Computing Fundamentals: CPUs, GPUs, Latency vs Throughput

Serial execution runs one instruction stream end-to-end on one core: step A finishes, then step B starts. Parallel execution splits the work across many workers that run at the same time.

Two flavours that get confused:

• Concurrency: structuring a program as independent tasks that could run in parallel (e.g. async I/O on one thread).
• Parallelism: actually executing them simultaneously on multiple physical cores.

A modern laptop CPU runs ~8-16 threads in parallel. A consumer GPU runs tens of thousands. The interesting question isn't "can I parallelise this?" — it's "will the parallel version actually be faster after data movement and synchronisation?"

If a fraction $p$ of your program is parallelisable and $(1-p)$ is stuck serial, the maximum speedup with $N$ workers is:

$S(N) = \frac{1}{(1-p) + p/N}$

Plug in numbers:

• $p = 0.95$, $N = 100$ → speedup ≈ 17x. Not 100x.
• $p = 0.95$, $N = \infty$ → speedup caps at 20x.
• $p = 0.50$, $N = \infty$ → speedup caps at 2x.

The takeaway: the serial fraction is your hard ceiling. Spending engineer-months parallelising a kernel only matters if the surrounding code (I/O, setup, teardown) doesn't dominate. Profile first, parallelise the hot path.

A CPU core is built to make one thread go fast. The transistor budget is spent on:

• Deep pipelines and out-of-order execution
• Branch prediction and speculative execution
• Large per-core caches (L1/L2 measured in MB)
• Wide SIMD units (AVX-512 = 16 floats per instruction)

A modern x86 core can chew through complex, branchy code with unpredictable memory access patterns and still hit 5+ GHz. The price: most of the die is control logic and cache, not arithmetic units. A 16-core CPU has maybe 64-128 vector lanes total. Great for the operating system, web servers, and serial bottlenecks. Bad for problems that need millions of independent multiply-adds.

A GPU flips the trade-off. It spends transistors on arithmetic units (thousands of them) and almost nothing on per-thread cleverness. Each individual thread is slow — no branch prediction, tiny per-thread cache, in-order execution — but you get tens of thousands of them running in lockstep.

A consumer RTX 4090 has:

• 16,384 CUDA cores across 128 streaming multiprocessors (SMs)
• ~83 TFLOPS FP32 throughput
• ~1 TB/s memory bandwidth (vs ~80 GB/s on a fast desktop CPU)

It cannot run your Python interpreter. It can multiply a 16384×16384 matrix in milliseconds. Different tool, different job.

Cores per die and what they're optimised for.

NVIDIA GPUs execute threads in groups of 32 called a warp. All 32 threads share one instruction pointer and execute the same instruction at the same time on different data. NVIDIA calls this SIMT (Single Instruction, Multiple Threads).

The consequence is brutal: if your code has an if/else that splits a warp (say, 16 threads go left, 16 go right), the hardware runs both branches sequentially, masking off the inactive threads. This is warp divergence and it halves your throughput for that section.

// Divergent — bad on GPU
if (threadIdx.x % 2 == 0) heavy_path();
else                       other_heavy_path();

Keep all 32 lanes doing the same thing whenever you can.

GPUs love workloads that are:

• Massively parallel: millions of independent items.
• Regular: same operation on every element, predictable memory access.
• Arithmetic-dense: many FLOPs per byte read from memory (compute-bound, not memory-bound).

GPUs hate workloads that are:

• Branch-heavy: every thread takes a different path → warp divergence.
• Tiny: kernel launch overhead is ~5-20µs; if the work takes less than that, you've made it slower.
• Pointer-chasing: linked lists, trees, graphs with irregular access patterns.
• Tightly coupled to the CPU: PCIe transfers are slow (~16-32 GB/s); ping-ponging data per step destroys any speedup.

GPUs sit behind a PCIe bus. Moving data from host RAM to device memory is slow compared to GPU compute:

If your kernel runs in 1 ms but moving the input takes 20 ms, you're not GPU-bound — you're PCIe-bound. The winning pattern is: upload once, run many kernels back-to-back, download once. Frameworks like PyTorch and CUDA Graphs are mostly tricks to keep data resident on the GPU between operations.

Before you write CUDA, ask:

1. Is the parallel fraction high? (Amdahl) If less than ~80%, the speedup ceiling isn't worth the effort.
2. Are there enough independent items? As a rough floor: aim for ≥10,000 work items per kernel, ideally ≥1M.
3. Is the work-per-item regular? Same operation, predictable memory access.
4. Does data stay on the GPU? If you'd have to round-trip to the CPU between every step, the GPU is fighting itself.

If any of those is no, a well-tuned multithreaded CPU implementation will usually beat a naive GPU port. If all four are yes, you're in CUDA's sweet spot — expect 10-100x speedups on the right problem.