GeekCoding101 make your way to geek

Find and Remove Duplicate Photos with a Python Script

A Python script, written with ChatGPT, that finds duplicate photos by xxhash checksum and cleans them up, with cached checksums for fast reruns.

6 min read
How I Automated My Photo Cleanup
On this pageOn this page

get_rid_of_dup.py is a free, open-source Python command-line script that finds duplicate photos, or any duplicate files, by comparing xxhash checksums. It then deletes the copies and leaves your originals alone. It works inside one folder or across two folders, and it can save checksums so later runs go much faster.

I wrote it over Thanksgiving with a lot of help from ChatGPT, after giving up on cleaning my WD NAS photo library by hand. Below are the commands and how the script decides what to delete. After that comes what building it with an LLM actually took.

GeekCoding101 · Published Dec 2, 2024 · 5 min read

Get the script: github.com/geekcoding101/get_rid_of_dup

On this page

  • Quick start
  • How the script works
  • Commands at a glance
  • Before you delete anything
  • Performance
  • Why I wrote it
  • Building it with ChatGPT: what it really took
  • FAQ

Quick start

  1. Clone or download the repo from GitHub.

  2. Make sure you have Python 3 and the xxhash library. The script uses xxhash for its checksums. Install it with pip install xxhash if you don’t have it.

  3. Find duplicates inside one folder:

    Terminal window
    python get_rid_of_dup.py dedup --base-dir ./photos --max-width 50 --verbose
  4. Or compare a folder of originals against a second folder, such as an old backup, and clean up the second one:

    Terminal window
    python get_rid_of_dup.py search --base-dir ./originals ./backup --max-width 50 --verbose --exclude "*.DS_Store"
    python get_rid_of_dup.py checksum --base-dir ./originals ./backup
    python get_rid_of_dup.py delete --base-dir ./originals ./backup

How the script works

The script reads each file, computes an xxhash checksum of its contents, and groups files with identical checksums as duplicates. xxhash is a fast non-cryptographic hash. That makes it a good fit for comparing tens of thousands of files, where speed matters more than cryptographic strength.

It has two ways of working.

Single-folder mode (dedup)

This mode finds duplicates within one folder. When several files are identical, the script treats the one with the shortest file name as the original. That keeps IMG_1234.jpg instead of IMG_1234 (1).jpg or IMG_1234 copy.jpg.

Terminal window
python get_rid_of_dup.py dedup --base-dir ./photos --max-width 50 --verbose
runtime screenshot 01

Cross-folder mode (search, checksum, delete)

This mode compares two folders. The first folder after --base-dir is the reference, and the script never touches it. Duplicates are cleaned out of the second folder only. That makes it a good fit for “I have my real library here and three old copies over there.”

Terminal window
python get_rid_of_dup.py search --base-dir ./test ./others --max-width 50 --verbose --exclude "*.DS_Store"
python get_rid_of_dup.py checksum --base-dir ./originals ./backup
python get_rid_of_dup.py delete --base-dir ./originals ./backup
runtime screenshot 02 runtime screenshot 03 runtime screenshot 04

Commands at a glance

CommandWhat it doesExample
dedupFinds duplicates inside one folder and keeps the shortest-named file as the originalpython get_rid_of_dup.py dedup --base-dir ./photos --verbose
searchFinds files in the second folder that duplicate files in the base folderpython get_rid_of_dup.py search --base-dir ./photos ./comparison --max-width 100
checksumGenerates and saves checksums so later runs can reuse thempython get_rid_of_dup.py checksum --base-dir ./photos ./backup
deleteDeletes the duplicates from the second folder and leaves the base folder untouchedpython get_rid_of_dup.py delete --base-dir ./photos ./backup

Useful options:

OptionUse
--base-dirThe folder (or folders) to work on. In cross-folder mode, the first one is the reference.
--verbosePrints more detail while it runs.
--exclude "*.DS_Store"Skips files matching a pattern, such as macOS .DS_Store files.
--skip-existingHelps fit the script into repeat runs and different workflows.
--max-widthUsed in the examples above with values like 50 and 100.

Before you delete anything

  • It finds exact copies, not look-alikes. Checksums match only when two files are byte-for-byte identical. A resized, re-edited or re-exported version of a photo has a different checksum, so the script won’t flag it.
  • Your base folder is safe in cross-folder mode. Only the second folder is cleaned.
  • In single-folder mode, the shortest name wins. Check that this rule suits your naming before you remove anything.
  • Run search first and read the output. Delete only when the list looks right.
  • Keep a backup. Deleting files is permanent, especially on a network share.
  • It isn’t limited to photos. The script compares file contents, so it works on any file type.

Performance

On my library, scanning 30,000+ files, including large images, took under a minute. That’s faster than making a coffee. Because the checksum command saves checksum data, later runs can skip recomputing it and finish much faster.

A tip from experience: I first worked on the NAS through an SMB mount on my MacBook, and it was painfully slow. Copying the files to local disk first made everything far quicker.

Why I wrote it

Thanksgiving usually means food, family and laughter. This year it also meant facing a huge library of duplicate photos on my WD NAS.

I mounted the NAS SMB share on my MacBook and spent two days copying files, then started reviewing them by hand. My eyes hurt, my patience ran out, and I knew there had to be a better way.

The existing “remove duplicate photos” tools didn’t help much. Most were paid, overly complex, or just didn’t fit what I needed. Even the free options expected me to stitch together commands like find. So I did what any tech enthusiast does: I wrote my own, with a “little” help from ChatGPT.

Building it with ChatGPT: what it really took

I’d tried scripting this years ago and gave up because of the time it needed. This time I had ChatGPT (no marketing here… I’m a paid user, though).

Warning: I thought I could finish this script in two hours with ChatGPT’s help. I spent almost the entire day on it! So those posts claiming you can build an iOS app in two hours? Be sceptical.

Still, I’m grateful LLMs exist. Judging by how much code there is and how good it is, I’d estimate this would have taken me 10 to 15 days alone. That’s about a tenfold speed-up for me, and it saved me from plenty of dead ends.

What stood out:

  • It teaches as it writes. For non-coders, ChatGPT makes programming less mysterious. For experienced developers, it speeds up the workflow and suggests ideas you might not have tried.
  • “Too complex to script” became doable. With its help I handled fiddly logic, performance tuning and error handling much faster than I would have alone.
  • It takes the repetitive parts. That leaves you free to focus on the design decisions, like which file counts as the original.

If you’re stuck with file clutter, or any repetitive chore, Python plus an LLM can turn it into a satisfying win.

FAQ

Does it find similar photos, or only exact duplicates? Only exact duplicates. It compares file checksums, so a resized or edited copy of a photo has a different checksum and won’t be flagged.

Will it delete my original photos? In cross-folder mode, no. The first folder after --base-dir is the reference and is never cleaned. In single-folder mode, it keeps the file with the shortest name as the original. Either way, run search first and keep a backup.

Can I use it on a NAS? Yes. I used it on photos from a WD NAS. Working over an SMB mount was very slow for me, so copying the files to local disk first is worth it.

Why xxhash instead of MD5 or SHA? xxhash is built for speed. For spotting identical files on your own disk, you don’t need cryptographic strength, and a fast hash makes large libraries quick to scan.

Is the script free? Yes. It’s open source on GitHub: geekcoding101/get_rid_of_dup.


Thanks for reading! Grab the script on GitHub, and feel free to browse the other posts on GeekCoding101.

Thanks for reading my blog post! Feel free to check out my other posts or contact me via the social links in the footer.

Keep reading

All posts