You already know kmalloc(): it hands back a pointer to memory that is contiguous in both virtual and physical address space, carved from the buddy/slab allocators. But kmalloc(16 * 1024 * 1024, GFP_KERNEL) often fails on a long-running server even with gigabytes free. Why? Because the buddy allocator needs an unbroken run of order-12 pages (16MB), and after weeks of uptime, physical memory is Swiss cheese. This is where vmalloc() earns its keep.

What vmalloc actually does. It grabs N individual physical pages (order-0 allocations, which almost never fail), then patches together kernel page-table entries in a dedicated virtual address range (VMALLOC_START to VMALLOC_END, typically 32TB on x86-64) so the pages appear contiguous to your kernel code. The pointer you get back is a normal-looking kernel virtual address, but there is no linear offset from physical — virt_to_phys() is undefined behavior on it.

The costs you pay.

• Every access is a TLB lookup on a fresh mapping, unlike the kernel's direct-map region which is backed by 1GB huge pages already loaded in the TLB. First touch of a vmalloc region is slow.
• You cannot DMA to it — the physical pages are scattered. Drivers doing bus-mastering use kmalloc, the DMA API, or scatter-gather lists.
• Freeing costs a TLB flush — historically a synchronous IPI to every CPU (vmalloc_sync_all), now lazy-batched via the vmap purge list, but still visible in perf traces of module unload.
• Metadata overhead: each allocation costs a struct vm_struct plus a red-black tree node in the vmap area tree.

Real-world example. When you insmod a kernel module, the loader calls module_alloc(), which is vmalloc under the hood (in a special sub-range that keeps modules within ±2GB of the kernel text so call rel32 instructions can reach). A 400KB module = 100 order-0 pages stitched together. It doesn't matter that physical RAM is fragmented; the module still loads.

Rule of thumb. Use kmalloc up to about 8KB freely, up to 128KB with care, and switch to vmalloc (or kvmalloc, which tries kmalloc first and falls back automatically) anywhere above that — unless you need DMA or hot-path performance, in which case reserve at boot with alloc_pages(GFP_KERNEL, order) while memory is still un-fragmented.

The kernel's own filesystems demonstrate this: XFS uses kvmalloc for its extent buffers precisely because a 64KB allocation might succeed as kmalloc on a fresh boot and only need vmalloc after weeks of uptime.