How to yield a reference to internally pinned data in a custom Stream implementation? (Lifetime mismatch with GATs)?

2026-04-24

Stack Overflow: View Question

Tags: rust, async-await, rust-pin

Score: 1 | Views: 73

The asker is building a zero-copy asynchronous parser in Rust. The idea is straightforward in concept: read bytes from an async reader into an internal buffer, parse them in place, and then yield references to the parsed data through a Stream. The critical constraint is "no cloning"—they want true zero-copy semantics where consumers borrow directly from the parser's internal buffer.

This is one of Rust's hardest problems at the intersection of three notoriously tricky areas: self-referential structs, pinning, and async iteration.

The fundamental tension is this: the standard Stream trait (from futures) defines poll_next to return Poll<Option<Self::Item>>, where Item is an associated type with no lifetime tied to self. This means you cannot return a reference that borrows from the stream's own internal buffer. The item must be owned or must borrow from something with an independent lifetime. This is the same limitation that prevents Iterator from yielding references to its own state—often called the "lending iterator" problem.

The asker mentions GATs (Generic Associated Types) as a potential solution. In theory, a lending stream trait could look like:

trait LendingStream {
    type Item<'a> where Self: 'a;
    fn poll_next(self: Pin<&mut Self>, cx: &mut Context)
        -> Poll<Option<Self::Item<'_>>>;
}

This would let Item borrow from self. However, there are significant practical obstacles:

A pragmatic approach is to sidestep the lending problem entirely: use a shared buffer behind Bytes (from the bytes crate), which provides zero-copy slicing through reference counting. Each yielded chunk is a Bytes slice into the original allocation—owned, cheaply cloneable, and no lifetime entanglement. This is the pattern used by hyper, tonic, and most production Rust async I/O.

If true lending semantics are essential, the lending-stream or gat-lending-iterator crates offer experimental traits, but be prepared to write your own combinators.

The challenge: Rust's Stream trait fundamentally cannot express items that borrow from the stream itself, making zero-copy async parsing a collision between the borrow checker, pinning invariants, and the absence of a standardized lending stream abstraction.

All newsletters