The document engine is OxiDB's default engine: JSON documents in collections, MongoDB-style queries, no schema. This article walks through what actually happens under an insert and a find — the write path and its fsync contract, how queries pick indexes, where document bytes physically live, how the WAL stays bounded while the server runs, what the transactions really guarantee — and the crash-test discipline that keeps every one of those claims honest. Companion article for the relational side: How the SQL Engine Works.
One engine instance per database. Collections are created implicitly on first insert and are independent of each other: each owns its WAL, its storage, its indexes, and its caches, so writes to different collections never contend. Within a collection, storage is a concurrent map with bucket-level locking — multiple threads insert and read in parallel without a collection-wide writer lock.
An acknowledged write has exactly one meaning: it is on disk. The path that enforces it:
A query is a small operator tree — $eq, $gt, $in, $regex, $elemMatch, $and/$or/$nor, $expr and friends. Execution is mechanical, not cost-based:
Null < Bool < Number < DateTime < String — with ISO-8601 / RFC 3339 / date-like strings auto-detected and stored as epoch milliseconds, so a date range query works no matter how the application serialized its timestamps. Equality and range predicates on an indexed field collapse to B-tree range scans; composite indexes serve multi-field prefixes.update_one/delete_one stop at the first match.$elemMatch, $not, $all, $size, $type, $mod, $nor, $expr) — they run as filters over the candidate set, and explain tells you exactly which ones did.The aggregation pipeline ($match, $group, $lookup, $facet, $setWindowFields, and the time-series trio $ohlcv → $densify → $fill) executes stage by stage over the same machinery; $match heads use the same index paths a find does.
Since v0.38, disk-first is the default: document bytes live in an append-only, zstd-compressed, memory-mapped data file, and RAM holds only a ~24-byte-per-document offset index plus a bounded LRU cache of hot decoded documents. Resident memory tracks your working set, not your dataset — measured on a live 48-hour ingest workload, 489 MB resident became ~35 MB. Updates and deletes leave dead space that automatic compaction reclaims once it crosses a dead-space threshold. OXIDB_DISK_FIRST=0 restores the fully-resident mode (fastest point reads, RAM proportional to data), and the choice is per collection, recorded next to the data — the on-disk format is authoritative over the environment, so flipping the flag never reinterprets existing collections.
Either way, the durable tree snapshot is written the same way: serialize to a temp file, fsync, atomically rename over the old snapshot. A crash mid-write leaves the previous snapshot intact. (A side effect you can watch on a dashboard: during the write, old and new coexist, so a disk-usage graph "breathes" by up to the snapshot size and settles at the rename.)
A WAL that only truncates at graceful shutdown is unbounded for a server that runs for months. But truncating a live WAL naively loses writes: a writer appends its record before it applies, so a snapshot taken concurrently can miss a document whose record is about to be erased — measured, that cost ~3 in 2000 acknowledged writes. The engine therefore seals instead of truncating:
seal(): the live WAL is atomically renamed to a numbered segment; a fresh empty one takes its place. Every record in the sealed segment is now, by construction, already applied.Every crash point is safe because recovery replays sealed segments as well as the live WAL, idempotently by document id: crash before the persist and the segment replays onto the old snapshot; crash between persist and retire and it replays onto the new one, harmlessly. This fires automatically past OXIDB_WAL_CHECKPOINT_BYTES (default 64 MiB).
One honest war story: the segment scanner once had a fast path whose sentinel file the checkpoint itself deleted — after which later segments were silently never retired and never replayed. It was caught because a demo dashboard's disk graph refused to behave, and it is why the sentinel is now a maintained invariant with a crash-replay regression test that was red before the fix. Claims here are only as good as the tests that pin them.
Transactions run OCC — optimistic concurrency control — with a three-phase commit:
(document, version) pairs into a read set. Nothing touches shared state.OCC retries hurt exactly one workload: a hot document everyone updates (an exchange's order book account, an inventory counter). For that there is find_for_update — real pessimistic per-document locks taken inside a transaction, acquired in sorted id order, re-reading after acquisition so the recorded versions are the locked ones, held to commit, with lock-timeout as the deadlock answer. Contenders queue instead of retry-storming.
What this adds up to, stated precisely: committed transactions are serializable with respect to the items they read and wrote; phantoms and torn reads for non-transactional observers are admitted for plain find/count. The exact guarantee and its anomaly scorecard live in docs/isolation.md — and a characterization test suite pins the document so the code cannot drift from it silently.
One anomaly used to slip through the model above: a report scanning while a transaction commits could see account A after the transfer and account B before it — a sum that was never true at any moment. Since 0.38.1 that cannot happen: every aggregate runs against a single commit instant. The mechanism is snapshot visibility for the read path only — the write path (OCC, find_for_update, group commit) is untouched, and writers never wait on a reader.
It works by remembering the past only while someone is looking at it. Opening a snapshot pins a commit sequence S; from then on, any write that displaces a document's bytes drops the prior bytes into an in-memory version map on its way through. A snapshot read resolves each live document against that map — rolled back to its state at S, later inserts invisible, later deletes resurrected. When no snapshot is open (the normal state) the map does not exist and a write pays one atomic load to find that out. Aggregations are cheaper still: they run optimistically against the latest state, and if the version map proves no write raced the scan — the overwhelmingly common case — the fast result was the snapshot and is returned as-is; only a raced scan pays the resolve-and-recheck fallback.
Multi-query consistency is explicit: snapshot_begin returns a token, snapshot_find / snapshot_count / snapshot_aggregate read through it (all at the Read role tier), snapshot_end releases it. A snapshot held open forever would make the version map a slow leak, so snapshots expire after OXIDB_SNAPSHOT_MAX_SECS (default 300): reads through a dead snapshot fail loudly, writers never stall, and the map prunes to the oldest live snapshot. The pinning test is red-first in the strictest sense — money moves between 400 accounts in OCC transactions while an aggregation sums balances; on the pre-feature tree it observes a torn sum within two seconds, with the feature it cannot.
A TTL index (create_ttl_index(collection, field, expireAfterSeconds)) is a normal field index plus a rule. A per-database sweeper thread ticks every second, computes now − expireAfterSeconds, and range-scans the index — O(expired + log n), never a table scan. One subtlety worth naming: the range's lower bound is the DateTime type floor, not unbounded — in a total order where numbers sort below dates, an unbounded range would sweep in documents whose field is a number and delete them regardless of expiry. Eviction removes the real documents: storage, caches, every index, full-text entries. RAM frees immediately; disk follows at the next snapshot. Deletion isn't WAL-logged — if a crash intervenes before the snapshot, recovery resurrects the expired documents and the next sweep, one second later, removes them again.
Full-text indexing is asynchronous by design: writes hand indexing jobs to a background worker over a bounded channel, so ingest latency never pays for tokenization. The index handles HTML, XML, JSON, PDF, DOCX, XLSX — and images via OCR when compiled in — with TF-IDF ranking. Vector indexes ride the same collection machinery for similarity search. Both are additional indexes over the same documents, not separate stores.
At-rest encryption (AES-GCM) sits at the storage layer, below everything above: data files, snapshots, and the WAL all encrypt when a key is supplied, and cost nothing when one is not. The test for it does not check a file — it scans every file the engine writes for plaintext, so a new file format cannot silently ship unencrypted.
PITR is opt-in (OXIDB_PITR) and zero-cost when off. On, every durable WAL write receives a global, monotonic, wall-clock-stamped sequence number, carried in the WAL's v2 record format. Sealed segments — the same ones online checkpointing produces — become the archive stream: a background archiver copies them, byte-verbatim with a trailer, into _archive/ under a self-healing manifest, and owns their retirement. restore_to_point extracts a base backup and replays the archive forward to a GSN, a timestamp, or latest, cutting at transaction boundaries so a restore is never half a transaction.
Multiple isolated databases live under one server (use_db, per-database TTL and alert threads, per-database SQL engines). In cluster mode, writes replicate through Raft. And unlike SQL — which routes whole statements to one backend — document operations carry a collection and a shard key, so oxipool can genuinely shard them: scatter, gather, and merge across shard groups, aggregations included.
The engine's durability claims are enforced by a specific discipline: crash tests SIGKILL a subprocess, never drop a handle — because a graceful shutdown persists everything and hides exactly the bugs that matter. Where a race window is microseconds wide, a test hook widens it so the test fails deterministically without the guard rather than passing by luck. Fault-injection suites (fsync failures, partitions, process kills mid-commit) have each found at least one real bug; each is fixed and pinned by a regression test. The same engine, same defaults, runs a published 24-of-24 win against MongoDB on a 1M-document benchmark — and the crash suites run against the same configuration that benchmark does.
The query surface itself — operators, aggregation stages, index types — lives on the Queries, Aggregation and Indexes pages; the transaction wire API is on Transactions.