A normal send() copies your buffer into a fresh kernel sk_buff. That copy is fine for a 1500-byte packet, but for a 1 MB TLS record it burns memcpy bandwidth and pollutes the L2 cache. MSG_ZEROCOPY (Linux 4.14+) lets the kernel pin your user pages and reference them directly from the skb's fragment array — no copy, ever.

The catch is you don't own the buffer anymore. The NIC may DMA from it minutes later (retransmits, TCP small queues, qdisc backlog). Reusing or freeing it would corrupt the wire. So the kernel returns a completion notification asynchronously via the socket's error queue, told to you as an SO_EE_ORIGIN_ZEROCOPY message.

The protocol:

• setsockopt(fd, SOL_SOCKET, SO_ZEROCOPY, &one, 4) — opt in per socket.
• send(fd, buf, len, MSG_ZEROCOPY) — returns immediately as usual, but the kernel stamps this send with a monotonically-increasing 32-bit ID.
• Later, recvmsg(fd, &msg, MSG_ERRQUEUE) returns a sock_extended_err whose ee_info..ee_data fields give the ID range of sends now safe to reuse.
• The kernel also sets ee_code & SO_EE_CODE_ZEROCOPY_COPIED if it had to fall back to a copy — silent zero-copy failure, which you'll only notice if you check.

Concrete example — an HTTPS proxy sending large TLS records: a naive send(4 MB) costs ~1.3 ms of pure memcpy at 3 GB/s. With MSG_ZEROCOPY it costs a page-pin walk (get_user_pages_fast) plus the error-queue drain — measured on production kernels at roughly ~40% CPU savings above 10 KB payloads, but ~5% worse below 4 KB because the notification bookkeeping outweighs the saved copy.

Rule of thumb: only enable MSG_ZEROCOPY when your average send is larger than two pages (~8 KB). Below that, the notification overhead and the extra syscall to drain the error queue dominate.

Sharp edges:

• UDP got MSG_ZEROCOPY only in 5.0; before that, TCP only.
• Notifications are batched by ID range — one errqueue message can complete thousands of sends. Poll for EPOLLERR alongside your normal EPOLLOUT.
• The pages are pinned, so they count against RLIMIT_MEMLOCK. A misbehaving app that never drains the errqueue will accumulate pinned pages and eventually get ENOBUFS back from send().
• If your buffer straddles a fork/COW boundary, the kernel silently falls back to copy — check SO_EE_CODE_ZEROCOPY_COPIED.