Every TCP recv() call has a hidden assumption: bytes you read are gone from the socket buffer. If you want to look ahead without consuming, you pass MSG_PEEK. But MSG_PEEK always peeks from byte zero of the receive queue — the next byte the kernel would hand you. If you want to peek at byte 4096 without first reading (and discarding) 4095 bytes into a throwaway buffer, classical POSIX has no answer.

SO_PEEK_OFF (Linux 3.4+, TCP and Unix sockets) is the answer. You set a per-socket offset with setsockopt(fd, SOL_SOCKET, SO_PEEK_OFF, &off, sizeof(off)). From then on, every recv(fd, buf, len, MSG_PEEK) reads starting at that offset, not at the head. And critically: the kernel advances SO_PEEK_OFF by the number of bytes peeked, so successive peeks walk forward through the queue automatically. A non-peek recv() also decrements the offset (never below zero), keeping it consistent as bytes drain.

Concrete example — an HTTP/1.1 pipelining parser. A client has pipelined three requests into one TCP segment. You need to find the \r\n\r\n terminator of each request's headers before deciding whether to fully consume it (some requests may need to be forwarded byte-for-byte to a backend, avoiding a user-space copy). Without SO_PEEK_OFF, each successive peek re-reads bytes you already scanned — O(N²) in queue depth. With SO_PEEK_OFF, you scan each byte exactly once, still non-destructively, and only issue the actual consuming recv() once you know the framing.

The subtle bit: it's not thread-safe. SO_PEEK_OFF is a single integer on the socket. Two threads peeking concurrently will race — one thread's peek advances the offset the other thread expected. Use it only from a single reader thread (which is the norm for socket I/O anyway) or serialize with your own lock.

Rule of thumb: if your protocol parser ever does recv(MSG_PEEK) in a loop with an increasing buffer size to find a delimiter, you're doing O(N²) copies. Switching to SO_PEEK_OFF-based incremental peeking makes it O(N) — for a 64 KB HTTP header scan, that's ~2 GB of avoided memcpy versus ~64 KB.

One more gotcha: SO_PEEK_OFF is only meaningful for stream sockets (TCP, Unix stream). On datagram sockets it's silently ignored — datagrams have record boundaries and the "peek at offset" model doesn't apply.