You're building a webhook signer. Each outbound request is HMAC-signed over its parameters, and the receiver verifies with the shared secret. Locally, tests pass. In staging, tests pass. In prod, roughly half of retried requests fail signature verification — but only after a pod restart. Same params, same secret, different signature.

func SignRequest(params map[string]string, secret []byte) string {
    h := hmac.New(sha256.New, secret)
    for k, v := range params {
        h.Write([]byte(k))
        h.Write([]byte("="))
        h.Write([]byte(v))
        h.Write([]byte("&"))
    }
    return hex.EncodeToString(h.Sum(nil))
}

func main() {
    secret := []byte("s3cr3t")
    params := map[string]string{
        "amount":   "100",
        "currency": "USD",
        "user_id":  "42",
    }
    fmt.Println(SignRequest(params, secret))
    fmt.Println(SignRequest(params, secret))
    // Often the same within one run — always different across runs.
}

The Bug

Go deliberately randomizes map iteration order. Since Go 1.0 the spec has said the order is unspecified, and since 1.3 the runtime actively perturbs it — a seed derived from a per-map hash randomizer is mixed in at map creation so that iteration order varies from run to run (and sometimes within a run after growth).

That means h.Write gets fed amount=100&currency=USD&user_id=42& on one invocation and user_id=42&amount=100&currency=USD& on the next. HMAC is order-sensitive: different byte sequences produce different digests. Your signature depends on the runtime's random seed.

Why does it seem to "work" in tests? Small maps (≤ 8 entries, one bucket) often iterate in insertion order for the duration of a single process — the randomization mostly kicks in across processes or after the map grows past one bucket. So a test suite that creates a fresh map inside each test can hash consistently for hours, then a pod restart re-seeds the runtime and every signature changes.

This class of bug hides in anything that turns a map into an ordered byte stream: HMAC signing, canonical JSON, cache keys built from concatenated pairs, log-line fingerprints for dedup, or an idempotency key hashed from a request body.

The Fix

Never iterate a map when the output order matters. Extract keys, sort them, then iterate in a defined order:

func SignRequest(params map[string]string, secret []byte) string {
    keys := make([]string, 0, len(params))
    for k := range params {
        keys = append(keys, k)
    }
    sort.Strings(keys)

    h := hmac.New(sha256.New, secret)
    for _, k := range keys {
        h.Write([]byte(k))
        h.Write([]byte("="))
        h.Write([]byte(params[k]))
        h.Write([]byte("&"))
    }
    return hex.EncodeToString(h.Sum(nil))
}

A few adjacent traps worth naming:

• encoding/json sorts map keys when marshaling map[string]T — so json.Marshal(params) is deterministic. But json.Marshal on a struct preserves field order, and swapping between the two representations silently changes the signed bytes.
• Separators matter for security. With key=value& concatenation, {"a":"b&c","d":"e"} and {"a":"b","c":"d","e":""} produce colliding inputs. Length-prefix each field, or sign a canonical JSON encoding.
• Don't rely on any Go container's iteration order for correctness unless the docs guarantee it — sync.Map.Range is also unspecified, and range over a channel is FIFO only for a single producer.

If you want a lint that catches this class of bug, go vet won't — but a review rule of "no range over a map inside a hash, hmac, Marshal, or io.Writer pipeline without a preceding sort" catches it every time.