pipeline-parallelism

1 free lesson tagged pipeline-parallelism across AI. Each one is a short sequence of focused steps with narration and a five-question quiz at the end — take them in any order, no signup required.

advanced

Splitting the Model Itself: Tensor and Pipeline Parallelism

Sharding distributes copies; tensor and pipeline parallelism split the computation. This lesson covers the column-then-row trick that lets a transformer block communicate only twice per layer, sequence parallelism for the parts tensor parallelism cannot reach, the pipeline bubble and why it is (p-1)/m, what 1F1B and interleaving actually fix, and how the three axes compose into a 3D layout.

10 steps·~15 min

Splitting the Model Itself: Tensor and Pipeline Parallelism

Related topics