Does MASM x64 (ml64.exe) generate incorrect assembly in regards to where function arguments should be located (ABI)?

2026-07-20

Stack Overflow: View Question

Tags: assembly, x86-64, masm, calling-convention, abi

Score: 6 | Views: 169

The asker is porting a function originally written using GCC inline assembly into an MSVC x64 project. Because there's no intrinsic for ltr (Load Task Register), they've placed the routine in a standalone .asm file and are assembling with ml64.exe. They suspect MASM is generating code that violates the Windows x64 calling convention — placing arguments in the wrong registers.

Why this is interesting: The Windows x64 ABI is quite specific — the first four integer/pointer arguments go in RCX, RDX, R8, R9, with 32 bytes of "shadow space" reserved on the stack by the caller. Any asymmetry between what the C compiler emits at the call site and what the assembler emits inside the callee will silently corrupt state. The question probes whether ml64.exe's PROC directive with typed parameters actually honors this ABI, or whether it does something unexpected (like using fastcall-style register mapping that mirrors 32-bit conventions).

Approach:

Gotchas:

The challenge: Diagnosing whether MASM's high-level PROC abstraction actually respects the Windows x64 ABI, or whether hand-rolled register access is the only reliable path.

All newsletters