In short: it uses Rust's standard library BTreeMap for deduplication, but with a twist that allows it to compare files lazily, reading only as little file content as necessary. Nerding out about the fast deduplication algorithm You can add -s flag to dedupe small files, too. This program will only deduplicate files larger than a single disk block (4KB, usually), because in many filesystems hardlinking tiny files may not actually save space. Hardlinking will make all duplicates of a file have the same file permissions. Editing a hardlinked file edits it in all places at once (except in some applications that delete & create a new file, instead of overwriting existing files). Deleting one of hardlinks leaves other hardlinks unchanged. Unlike symlinks, the hardlinks behave like real files. Instead, litreally the same file will exist in two or more directories at once.
How does hardlinking work?įiles are deduplicated by making a hardlink. You can also use this program as a Rust library for seamless integration. Add -json option if you want machine-readable output. Symlinks, special device files, and 0-sized files are always skipped.ĭon't try to parse program's usual output. It prints both paths on the same line with the difference between them highlighted as. dupe-krill -d # find dupes without doing anythingĭupe-krill # find and replace with hardlinks
#CRMFUSION SINGLE TABLE DEDUPLICATOR INSTALL#
If you have the latest stable Rust (1.42+), build the program with either cargo install dupe-krill or clone this repo and cargo build -release.
Gracefully handles symlinks and special files.ĭownload binaries from the releases page.It's aware of existing hardlinks and supports merging of multiple groups of hardlinks.Replaces files atomically and it's safe to interrupt at any time.Deduplicates incrementally as soon as duplicates are found.It's very fast and reasonably memory-efficient.Useful for reducing sizes of multiple backups, messy collections of photos and music, countless copies of node_modules, macOS app bundles, and anything else that's usually immutable (since all hardlinked copies of a file will change when any one of them is changed). Replaces files that have identical content with hardlinks, so that file data of all copies is stored only once, saving disk space.
#CRMFUSION SINGLE TABLE DEDUPLICATOR DOWNLOAD#
# 52 in Filesystem Download history 5/week 2/week 43/week 47/week 132/week 38/week 33/week 82/week 1/week 3/week 3/week 2/week 20/week 13/week 24/week Ĩ54 lines Dupe k rill - a fast file deduplicator
0 kommentar(er)
