SIMD code lives and dies by data layout. Your vectors arrived from memory in one shape, but the math you want to do needs them in another. That's where the permute/shuffle unit earns its keep — and where naive code quietly burns cycles.

Every SIMD ISA has a family of these ops: x86's PSHUFB, VPERMD, VPERMPS, VPERM2F128; ARM's TBL/TBX; NEON's VZIP/VUZP. They all move lanes around inside (or between) vector registers, guided by an index vector or an immediate mask.

The catch: not all permutes are equal. There are three tiers on modern x86:

• In-lane shuffles (PSHUFD, SHUFPS): stay within a 128-bit lane. 1 cycle latency, throughput 1/cycle on port 5 (Intel) or ports 1/2 (AMD Zen 4).
• Cross-lane permutes (VPERMD, VPERMPS on 256-bit): traverse the 128-bit lane boundary. 3 cycles latency, still port 5 only — the wiring to swap between lanes physically costs more silicon and time.
• 512-bit two-source permutes (VPERMT2D): read two source vectors plus an index. 3–5 cycles latency, throughput often 1 per 2 cycles because the crossbar is enormous.

Why the lane boundary matters: AVX2 was bolted onto AVX by widening most 128-bit ops to two independent 128-bit halves. Real 256-bit crossbar wiring only exists for the explicit cross-lane instructions. That's why PSHUFB can't shuffle byte 0 of the low lane into the high lane — the wires aren't there.

Concrete example — matrix transpose: A 4×4 float transpose using _mm_shuffle_ps + _mm_unpacklo/hi_ps takes 8 in-lane shuffles at 1c each = ~8 cycles. Extend to 8×8 with AVX2 and you must use VPERM2F128 for the lane-crossing step; each of those is 3 cycles. The transpose cost roughly triples per doubling of width, not doubles — that's the port-5 bottleneck plus cross-lane latency compounding.

Port pressure rule of thumb: On Intel Skylake through Golden Cove, every shuffle competes for port 5. If your loop does 4 shuffles per iteration, port 5 caps you at 4 cycles/iteration regardless of how much math the other ports could handle. Check with perf stat -e uops_dispatched_port.port_5 — if it's near 100%, you're shuffle-bound, and no amount of FMA cleverness will save you.

Design workaround: Precompute layouts. Store data in structure-of-arrays form so lanes align naturally with the operation, and reserve permutes for the unavoidable reduction step.