strtok Internal State Trap: The Nested Tokenizer That Devours Itself2026-06-16
This function parses a multi-line CSV by splitting on newlines, then splitting each line on commas. Looks textbook. Run it on a three-line input and you'll process exactly three fields — then it falls silent.
#include <stdio.h>
#include <string.h>
static void emit(const char *f) { printf("[%s] ", f); }
void parse_csv(char *input) {
char *line = strtok(input, "\n");
while (line != NULL) {
char *field = strtok(line, ",");
while (field != NULL) {
emit(field);
field = strtok(NULL, ",");
}
line = strtok(NULL, "\n");
}
}
int main(void) {
char data[] = "a,b,c\nd,e,f\ng,h,i";
parse_csv(data);
putchar('\n');
return 0;
}
Expected: [a] [b] [c] [d] [e] [f] [g] [h] [i]. Actual: [a] [b] [c].
strtok is one of C's oldest sins: it holds a single static pointer inside the library for "where to resume next time you pass NULL." There is exactly one of these — globally, for the whole process. The moment you nest two strtok walks, the inner one stomps the outer one's bookmark.
Trace what happens:
strtok(input, "\n") writes a \0 over the first newline, returns "a,b,c", and parks its bookmark at "d,e,f\ng,h,i".strtok(line, ","). That re-initializes the bookmark to point inside "a,b,c". The outer bookmark — pointing at d — is gone forever.a, b, c, eventually returning NULL with the bookmark sitting on the terminator at the end of the first line.strtok(NULL, "\n") resumes from that position, finds only \0, returns NULL, and the loop ends. Lines two and three are never seen.This is the kind of bug that survives review because the code reads like prose. Nothing in the signature warns you that strtok is secretly a stateful object — a singleton with no name. Threads make it worse (each call races on the static), but you don't need threads to break it; nested calls in a single thread are enough.
Use strtok_r (POSIX) or strtok_s (C11 Annex K). They take an explicit char **saveptr, so each tokenizer carries its own bookmark:
void parse_csv(char *input) {
char *outer, *inner;
char *line = strtok_r(input, "\n", &outer);
while (line != NULL) {
char *field = strtok_r(line, ",", &inner);
while (field != NULL) {
emit(field);
field = strtok_r(NULL, ",", &inner);
}
line = strtok_r(NULL, "\n", &outer);
}
}
Two save pointers, two independent walks, no shared global state. As a rule, treat plain strtok as deprecated: it's safe only at the bottom of the call stack, in a single thread, with no callee ever touching it. That's a precondition you cannot enforce — and the day someone adds a log_debug(line) that itself tokenizes a format string, your parser quietly stops at line one.
strtok stores its resume position in a single hidden global, so any nested or concurrent call silently hijacks every walk in progress — always reach for strtok_r with an explicit save pointer.
