A systolic array is a 2D grid of tiny processing elements (PEs) that each do one multiply-accumulate per cycle, then pass their operands to their neighbors. Data flows through the array like blood through a heart — hence "systolic." The array's power comes from a brutal insight: DRAM bandwidth is the bottleneck, not multipliers. So load each operand from memory once, then reuse it across many PEs by shuffling it laterally.

Consider multiplying two 3×3 matrices A × B. A naive design fetches 9 elements of A and 9 of B for every output, and does 27 multiplies with 54 memory reads. A 3×3 systolic array does the same work with just 18 memory reads: A rows enter from the left, B columns enter from the top, and each PE holds a running accumulator. On each cycle every PE multiplies its two incoming operands, adds to the accumulator, and passes A one step right and B one step down. After 5 cycles (2N–1), the accumulators hold the full C matrix.

Each PE is dead simple: a multiplier, an adder, an accumulator register, and two pass-through registers for the operands. No fetching, no branching, no cache — just local wires to nearest neighbors. That locality is why systolic arrays scale beautifully: adding a PE adds one MAC per cycle and doesn't lengthen any wire. Contrast with a shared bus, where every new consumer adds capacitance and slows the whole thing down.

Real-world example: Google's TPU v1 is a 256×256 systolic array of 8-bit multipliers running at 700 MHz. That's 65,536 MACs per cycle × 700 MHz × 2 ops per MAC = 92 TOPS from a single chip, using around 40 W. A GPU of the same era needed roughly 10× the power for similar inference throughput because it kept fetching weights from register files instead of shuffling them PE-to-PE.

Rule of thumb: An N×N systolic array multiplies two N×N matrices in 3N–2 cycles (N–1 to fill the pipeline, N to stream data, N–1 to drain). Throughput is one full matrix per N cycles once steady-state hits. Compare to a single MAC unit: N³ cycles. The array trades N² silicon for an N² speedup — a perfect linear scaling that almost nothing else in computer architecture achieves.

The catch: systolic arrays are rigid. They demand rectangular, dense, uniform work. Sparse matrices, non-square shapes, and irregular access patterns all leave PEs idle. That's why TPUs shine at dense convolutions and struggle with sparse transformers unless the array is subdivided or compilers pack the sparsity away.