content-defined-chunking

Fast and efficient content-defined chunking for data deduplication. Java implementation of FastCDC as library.

DedupBench is a benchmarking tool for content-defined chunking techniques used in data deduplication. It currently supports eleven unique CDC techniques and five different vector instruction sets.

Steadlith reuses unchanged RAG chunks with content-defined identities, cache-aware planning, and transactional indexing.

Fast content-defined chunking in Go.

CDCZip is a deduplication filter meant for use as long range match finder preprocessor for use alongside other compression algorithms.

Find (partial content) duplicate files.

Print FastCDC rolling hash chunks and checksums.

A Rust library for Content-Defined Chunking (CDC).

Synclet is a content-defined, chunk-based file synchronization and backup tool that detects and syncs only the actual changes—efficiently, intelligently, and reliably.

🧬 The 21st Century Data Format for AI. CDC deduplication, Zero-Download cloud streaming, O(1) random access. Built in Rust.

Turn x86-64 hardware into a high-performance POSIX dedup appliance with SMB/FUSE, multi-stage data reduction, and an embedded WebUI.

Open-source, self-hostable file sync for people and AI agents, with a Rust daemon and CLI, native macOS Finder integration, scoped sharing, and S3-compatible storage.

ChunkShift Core — deterministic content-defined chunking, streaming manifests, verification, and bounded-memory scanning for .NET.

A lightweight, local, self-hosted S3-compatible object store written in Go with zero third-party dependencies, featuring content-addressed storage, deduplication, and delta replication.

For Python apps that ship data to devices: each one checks it's genuine, downloads only changes, and searches offline.

Find where your disk space is duplicated — even inside files. FastCDC content-defined chunking + BLAKE3, parallel, one static Zig binary, interactive HTML treemap report.

Content-defined chunking with deduplication, integrity checking, and resumable ingest. Standard library only.

Content-defined chunk boundaries so a one paragraph edit re-embeds two chunks, not the document

Kernel-safe Rust algorithms for hardware-accelerated byte-stream analysis, content-defined chunking, matching, and runtime dispatch.

Open-source toolkit for lossless, reference-based reconstruction of large binary artifacts using deterministic CDC and compact SBD1 seeds.