qsort Comparator Subtraction Trap: When a - b Overflows the Sort2026-06-01
This program sorts five integers — some positive, some negative, all comfortably inside int range. The output should be the array in ascending order. Spot the bug:
#include <stdio.h>
#include <stdlib.h>
int cmp_int(const void *a, const void *b) {
int x = *(const int *)a;
int y = *(const int *)b;
return x - y; /* negative if x < y, zero if equal, positive if x > y */
}
int main(void) {
int arr[] = { 1500000000, -1500000000, 0, 2000000000, -2000000000 };
size_t n = sizeof arr / sizeof arr[0];
qsort(arr, n, sizeof(int), cmp_int);
for (size_t i = 0; i < n; i++) printf("%d ", arr[i]);
putchar('\n');
return 0;
}
You expect -2000000000 -1500000000 0 1500000000 2000000000. What you actually get is something like 1500000000 2000000000 -2000000000 -1500000000 0 — the negative numbers ended up on the right, and the order changes capriciously across compilers and optimization levels.
The "clever" one-liner return x - y; is the most copy-pasted comparator in C, and it is broken. When x is large and positive and y is large and negative (or vice versa), the mathematical difference exceeds INT_MAX and the subtraction signed-overflows.
Concretely, with x = 1500000000 and y = -1500000000, the true difference is 3,000,000,000 — well past INT_MAX (2,147,483,647). The result wraps around to a negative number, so cmp_int reports that the bigger value is "less than" the smaller one. qsort dutifully believes the lie and shuffles the array into garbage.
Worse: signed integer overflow in C is undefined behavior. Modern optimizers are entitled to assume it never happens. Under -O2, a compiler may decide that x - y preserves the sign of x > y and elide bounds checks elsewhere — meaning a debug build could "pass" while the release build silently corrupts data. The bug is invisible in unit tests that only use small numbers, and lurks until the day production data contains a value near INT_MAX.
The fix is to compare, not subtract:
int cmp_int(const void *a, const void *b) {
int x = *(const int *)a;
int y = *(const int *)b;
return (x > y) - (x < y); /* -1, 0, or +1 — never overflows */
}
Each subexpression is a _Bool promoted to int, so the result is always one of -1, 0, +1. No subtraction of operands ever occurs. An explicit ladder works equally well:
if (x < y) return -1;
if (x > y) return 1;
return 0;
The same trap hides in any "subtract to compare" idiom: timestamps (time_t differences across the epoch), hash values, file offsets, even pointer arithmetic compared as integers. strcmp sidesteps it by returning differences of unsigned char, which can't overflow int. Your int comparator has no such luxury.
Static analyzers (clang-tidy's cert-int31-c, MISRA C 10.1) flag this pattern, but only if you turn them on. Java's Comparator.comparingInt dodges it via Integer.compare; Go's slices.SortFunc takes a cmp.Compare that does the same. If your language's stdlib gives you a comparison helper, use it — and never write a - b in a sort callback again.
a - b silently overflows on operands of opposing sign and invokes undefined behavior that optimizers love to weaponize.
