Storage Engine

Architecture

• Append-only file -- Documents stored as [status:u8][length:u32 LE][payload]. Soft-delete flips the status byte in place.
• Write-Ahead Log (WAL) -- CRC32 checksums per entry. Transaction ID tagging. 3-fsync protocol: WAL → data → checkpoint.
• Zstd compression -- Level 3 by default. Transparent per-document. Thread-local compressor/decompressor reuse.
• AES-256-GCM encryption -- Optional. Random 12-byte nonce per document. Applied after compression.
• LRU document cache -- Per-collection in-memory cache. JSON deserialized once, then Arc-refcounted. Configurable capacity.
• Lock-free reads -- Separate read-only file handle uses pread. Writes are serialized via Mutex.
• Lazy sync mode -- Background thread batches fsyncs at a configurable interval. Reduces write latency at the cost of durability window.

Collection Isolation

Each collection has its own storage file, WAL, indexes, and cache. Per-collection RwLock enables concurrent reads across different collections and concurrent reads within the same collection.