stdout Buffering Trap: The Debug Log That Vanishes Right Before the Crash2026-08-31
A junior on your team is chasing a segfault in a batch record processor. They added a printf to trace which record trips the crash, then ran the tool in production redirected to a log file. The tool crashes as expected — but the log is empty. Nothing. Not even the records that processed cleanly.
#include <stdio.h>
int process_record(const char *record) {
printf("processing: %s\n", record);
// ... expensive parsing, validation, DB writes ...
if (record[0] == 'X') {
int *p = NULL;
return *p; // triggers SIGSEGV
}
return 0;
}
int main(int argc, char **argv) {
for (int i = 1; i < argc; i++) {
process_record(argv[i]);
}
return 0;
}
Interactive run: ./proc A B X C prints three "processing:" lines before the crash. Perfect. Piped run: ./proc A B X C > run.log 2>&1 → run.log is zero bytes. The junior swears the printf is being skipped. It isn't.
C's stdio buffering mode depends on what stdout is connected to, decided once at program start. When stdout is a terminal (TTY), it's line-buffered — every \n flushes. When stdout is a file, pipe, or anything non-interactive, it's fully block-buffered (typically 4 KiB or 8 KiB). Three short "processing:" lines don't come close to filling the block.
Normal program exit calls exit(), which runs registered handlers that flush all open FILE* streams. But SIGSEGV — or any signal that causes abnormal termination — skips that. The process dies; the buffered pages in userspace never make it to write(2). Your logs die with the process.
This is the "why did my debug output vanish exactly when I needed it most?" trap. The very act of redirecting to a file to preserve the output is what makes it disappear. Same code, same input, same crash — the presence or absence of a TTY silently changes the buffering discipline, and a null-pointer dereference erases everything you were counting on to diagnose it.
It gets worse under process supervisors (systemd, Docker, Kubernetes) where stdout is always a pipe. Local development "works." Production loses the last N seconds of every crash log.
Force line buffering (or no buffering) at startup, before any output:
int main(int argc, char **argv) {
setvbuf(stdout, NULL, _IOLBF, 0); // line-buffered regardless of destination
setvbuf(stderr, NULL, _IONBF, 0); // stderr is unbuffered on TTY but not pipes
for (int i = 1; i < argc; i++) {
process_record(argv[i]);
}
return 0;
}
Options ranked from safest to most invasive:
setvbuf(stdout, NULL, _IOLBF, 0) — line-buffered; cheap, matches TTY behavior. Best default for logging.setvbuf(stdout, NULL, _IONBF, 0) — unbuffered; every character is a syscall. Fine for low-volume logs, wasteful for hot loops.fflush(stdout) after each meaningful log — surgical, but easy to forget.stdbuf -oL ./proc — no code change, but relies on the operator remembering.Also install a signal handler for SIGSEGV/SIGABRT that calls fflush(NULL) (flushes all streams) before re-raising — belt-and-braces for the crashes you didn't anticipate.
