Here's a bar bet that pays every time: ask a developer to explain why their C++ ray tracer runs 40% faster on their laptop than on the shiny dual-socket Xeon they were told is "twice as fast." The answer is almost always NUMA, and the fix is almost always numactl.

Every modern server with more than one CPU socket — and any AMD chiplet-based part, and even some ARM boxes — is a Non-Uniform Memory Access machine. Each socket owns some DRAM. Reads to your own socket's memory take ~80 ns. Reads to the other socket cost 130–180 ns. The kernel tries to keep processes and their pages co-located, but when a scheduler migrates a busy thread across sockets (which it does, all the time), you keep paying cross-socket tax for the rest of the run.

First, look at what you actually own:

$ numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23
node 0 size: 128824 MB
node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31
node 1 size: 128954 MB
node distances:
node   0   1
  0:  10  21
  1:  21  10

Distance 21 vs 10 is the ratio: remote accesses are 2.1× slower. Now pin a process so it never crosses:

# Run on node 0 CPUs only, allocate memory only from node 0
$ numactl --cpunodebind=0 --membind=0 ./raytracer scene.obj

# Same thing, shorter
$ numactl -N 0 -m 0 ./raytracer scene.obj

For a large process that genuinely wants all RAM (Postgres shared_buffers, a JVM with a 200 GB heap, an in-memory cache) you don't want to bind — you want to interleave, so cache lines are scattered evenly across sockets and both memory controllers share the load:

$ numactl --interleave=all java -Xmx200g -jar bigcache.jar

This is the single most reproducible perf win for JVMs on big boxes. MongoDB, Cassandra, Elasticsearch all recommend it in their tuning docs, and you can just... prepend it to the command. No JVM flag mysticism required.

The companion tool is numastat, which tells you whether any of this is actually working:

$ numastat -p $(pgrep -f raytracer)
Per-node process memory usage (in MBs) for PID 84213
                    Node 0  Node 1  Total
Huge                   0.00   0.00   0.00
Heap                4832.14   3.02  4835.16
Stack                  0.06   0.00   0.06
Private              412.55   0.44   413.00

4832 MB on node 0, 3 MB on node 1. Binding worked. Without numactl, you'd see it 50/50 and every load would be a coin flip.

Two more tricks worth knowing:

• numactl --physcpubind=0-3 --localalloc ./prog — pin to specific cores, then let each thread allocate on whichever node it happens to run on. Better than --membind when you don't know the access pattern.
• numactl --preferred=1 ./prog — try node 1 first, spill to node 0 only if node 1 fills up. Useful when one node is already busy with another workload.

Why is this better than the mainstream alternative? Because there isn't one. taskset pins CPUs but ignores memory. cgroups' cpuset.mems works but requires setting up a cgroup. The kernel's autoNUMA balancer (enabled since 3.8) migrates pages in the background — helpful, but reactive, and it can't beat a process that told the allocator the right answer up front.

Ancient tool. Ships in every distro. Two flags. Sometimes 40% faster.