When you call send() with MSG_ZEROCOPY, the kernel doesn't copy your buffer into a socket buffer — it pins your user pages and hands them directly to the NIC via DMA. This saves a memcpy that can dominate CPU time on 25/100 Gbps links. But it creates a new problem: when is it safe to reuse or free that buffer? The send() call returns immediately, long before the NIC has actually transmitted the data.

The answer is a completion notification queued on the socket's error queue, read via recvmsg() with the MSG_ERRQUEUE flag. Each notification is a struct sock_extended_err with ee_origin = SO_EE_ORIGIN_ZEROCOPY, plus a range [ee_info, ee_data] giving the sequence numbers of completed sends. The kernel assigns each zerocopy send a monotonic 32-bit ID; you track which IDs you've submitted and wait for the matching completion before touching the buffer again.

Two subtle mechanisms make this practical. First, the kernel coalesces contiguous completions — 1000 zerocopy sends usually produce far fewer than 1000 notifications, because the range [N, N+999] fits in one message. Second, the ee_code field carries an SO_EE_CODE_ZEROCOPY_COPIED flag: if the kernel had to fall back to a copy anyway (e.g., because your buffer was too small to amortize the pinning cost, typically under ~10KB), the flag is set. This lets you dynamically decide whether zerocopy is winning.

Real-world example: Cloudflare's HTTP/3 stack uses MSG_ZEROCOPY for response bodies over a threshold. They maintain a ring of in-flight buffer descriptors indexed by send ID. On every event loop tick they drain the error queue, mark buffers as free up to the highest completed ID, and reuse them. The tricky bit: if you close the socket, you must drain the error queue first or leak the pinned pages until the socket's final release.

Rule of thumb: MSG_ZEROCOPY breaks even around 10KB per send on modern x86. Below that, the pinning overhead (get_user_pages + refcount + completion) exceeds the memcpy you saved. A 4KB HTTP response? Just use regular send(). A 1MB file chunk? Zerocopy every time — you'll cut CPU per byte by roughly 30%.

The gotcha that bites everyone: the buffer isn't safe to reuse when send() returns. Modifying it in flight will corrupt the on-wire bytes with whatever you wrote, because the NIC reads directly from your pages. Always wait for the completion.