How OxiDB is tested

A database vendor saying “trust our tests” is worth little. This page lists the evidence instead: tests other people wrote, tests that were proven to fail before the feature existed, crash tests that kill the process for real, and every known divergence — documented, not hidden.

MongoDB’s own specification tests

The MongoDB unified spec tests (mongodb/specifications) are the language-neutral JSON files every official MongoDB driver is validated against. We run all 189 CRUD files against the OxiDB document engine:

The suite runs with scripts/run_mongo_spec.sh. The spec files are CC BY-NC-SA licensed, so they are cloned at run time rather than vendored — and the adapter's one shim (engine-assigned _id) is documented in the runner's header, not smuggled past you.

On the SQL side the same philosophy already ran its course: the EF Core provider passes all 3832 of Microsoft's official EF Core relational specification tests.

Crash tests that actually crash

Graceful shutdown hides durability bugs: Drop flushes in-memory state and papers over a lost write. Every OxiDB crash suite therefore SIGKILLs a subprocess mid-write and verifies acknowledged writes after recovery — WAL replay, sealed-segment replay, checkpoint interleavings, multi-collection atomicity, encrypted stores.

Two findings that shaped the method:

• A race window can be too small to test honestly. The online-checkpoint barrier protects a µs-wide window between WAL append and B-tree apply; the naive 5-round SIGKILL test passed with the barrier deleted. The suite now widens the window with a hidden stall hook so the test fails in seconds without the fix.
• Red-first proofs. Before MVCC-lite shipped, its torn-read test was extracted and run against the engine without the feature — and had to be strengthened until it reliably failed (observed sum 40005 vs 40000 within 2s). A test that never went red proves nothing.

Fault injection

• fsync faults: injected write/sync failures must poison durability — the engine refuses to persist state containing a rejected transaction and rebuilds from the durable snapshot + WAL.
• Jepsen-style bank workload: concurrent transfers with SIGKILL rounds; total balance must survive every crash.
• Raft partitions: symmetric and asymmetric (deaf-node, reply-loss) partition disruptors against 3-node clusters — each round found or ruled out a real bug, mutation-verified.
• p99 soak: 1.6M ops at ~5.4k ops/s on Linux; read p99 3.9 ms, no drift, RSS plateau.

Fuzzing — with trophies published

An unauthenticated fuzz harness against the wire protocol found 4 real denial-of-service bugs (fixed in 0.28.3; servers older than that are vulnerable). A fuzzer that has never caught anything is a fuzzer that isn't trying; ours has a trophy list and keeps running.

Honest semantics: the isolation scorecard

OxiDB documents exactly which anomalies its OCC model admits and which it excludes — a per-anomaly scorecard in docs/isolation.md, pinned by a characterization suite so the docs cannot drift from the engine. Aggregations are snapshot-consistent by default (MVCC-lite): a concurrent aggregate can never observe half a transfer.

Benchmark methodology — including the losses

The 1M-document MongoDB comparison runs both engines natively, same machine, same indexes, client outside Docker (an earlier setup measured a Docker port-forward artifact and flattered nobody — we documented it). Current standing: OxiDB faster in 12 of 18 operations, including every aggregation and bulk updates. The losses are printed too, with their causes:

• UpdateOne latency: OxiDB fsyncs every commit with F_FULLFSYNC (~4 ms on Apple SSDs, measured); MongoDB's default acknowledges before its journal reaches disk. At equal durability settings the two engines are within 2µs of each other — the gap is a durability policy, not engine speed.
• Index build / DeleteMany: the cost of maintaining three compound indexes that in exchange answer $group aggregations 5–8× faster than MongoDB without reading a single document.

One harness lesson worth stealing: in an earlier EF Core benchmark, the more OxiDB won a round, the worse its next number looked — the idle server paid recovery inside the measured window. Benchmarks get root-caused here like bugs do.

The numbers today

What’s next on the evidence ladder

• Exhaustive crash-point testing: a fault-injecting storage shim that kills at every fsync boundary, not at random ones — the SQLite discipline.
• Differential testing: millions of random operation sequences replayed against a real MongoDB, every divergence either fixed or added to the documented list.
• Jepsen + Elle with published histories, so the linearizability claim is something you can re-check yourself rather than believe.