Every developer knows md5sum and sha256sum. Flip one bit, the hash changes completely. That's the whole point of a cryptographic hash — but it's spectacularly useless the moment you want to answer "are these two files similar?" Different by one appended byte? Totally different hash. A 40 GB VM image with one changed sector? Totally different hash.

ssdeep (Jesse Kornblum, 2006, based on Andrew Tridgell's spamsum) implements Context Triggered Piecewise Hashing. It chunks the file at content-defined boundaries (a rolling hash rings the bell), hashes each chunk into one base64 character, and concatenates them. Two files that share most content produce hashes that share most characters, and ssdeep gives you a similarity score from 0 to 100.

Basic use — the format is blocksize:hash1:hash2,filename:

$ ssdeep report_v1.pdf
ssdeep,1.1--blocksize:hash:hash,filename
6144:8Nl3fT...JXaB:8Nl3fT...aB,"report_v1.pdf"

$ ssdeep -b report_v1.pdf report_v2.pdf
6144:8Nl3fT...JXaB:8Nl3fT...aB,"report_v1.pdf"
6144:8Nl3fT...JXaC:8Nl3fT...aC,"report_v2.pdf"

Now the interesting part — directly compare two files and get a score:

$ ssdeep -d report_v1.pdf report_v2.pdf
report_v2.pdf matches report_v1.pdf (94)

94% similar. A single-byte change to a text file typically scores 99. A cover-page swap on a PDF might land at 70. Under about 40, the match is likely coincidence.

Match a suspect against a known-bad corpus. This is the malware analyst's day job:

# Build signature file from known-bad samples once
$ ssdeep -rl ./malware_zoo/ > known_bad.sigs

# Later, screen incoming files against it
$ ssdeep -rlm known_bad.sigs ./quarantine/
./quarantine/invoice.exe matches known_bad.sigs:trickbot_2024_var3.bin (81)
./quarantine/loader.dll matches known_bad.sigs:emotet_dropper.bin (63)

Cluster all similar files in a directory — great for finding near-duplicate documents, dumps, or captures:

$ ssdeep -rlp ./document_dumps/
./document_dumps/draft_final.docx matches ./document_dumps/draft_v7.docx (89)
./document_dumps/draft_final.docx matches ./document_dumps/draft_v6.docx (72)
./document_dumps/spec_A.pdf      matches ./document_dumps/spec_B.pdf   (54)

The -t 70 flag suppresses matches below a threshold. Pipe through sort -k5 -n for a ranking.

Why not just diff or rsync's rolling checksum?

• diff tells you what changed but gives no compact scalar score; useless on binaries.
• cmp is boolean.
• rsync --checksum uses rolling hashes internally but doesn't expose "how similar" — it only decides which blocks to ship.
• MinHash/simhash libraries exist but require you to write code. ssdeep is a stable CLI you can drop into a shell pipeline today.

Where ssdeep shines and where it doesn't. It's excellent on files 4 KB – 100 MB where you care about content-level similarity: documents, executables, memory dumps, PCAPs, spam email. It's poor on very small files (too few chunks) and on rearranged content (moving a section to the front tanks the score — for that, look at sdhash or Trend Micro's TLSH, both worthy siblings). It's not cryptographic — you cannot use it as a tamper-evidence hash. And blocksize must match to compare, so ssdeep automatically probes adjacent block sizes; if you script it, use -b and let the tool handle the arithmetic.

Available in every distro as ssdeep, plus Python bindings (python-ssdeep) and a libfuzzy for embedding.