Scaling out is two different problems, and OxiDB keeps them apart. Replication makes copies of the same data, so losing a machine doesn't lose the data. Sharding splits the data, so no machine has to hold all of it. They solve different things, they are configured separately, and they compose.
| Raft replication | oxipool sharding | |
|---|---|---|
| Buys you | Survival — a node can die | Capacity — data won't fit on one node |
| Each node holds | The same data | A slice of the data |
| Costs you | Write latency (a quorum must agree) | Cross-shard queries fan out |
| Doesn't help with | Data bigger than one node | A shard dying |
| Turned on by | OXIDB_NODE_ID on the server | Running oxipool in front |
Pick replication if you fear machines failing. Pick sharding if you fear the data outgrowing a machine. Most people who need one eventually need both — which is why they are separate layers rather than one setting.
Every node holds a full copy. One node is the leader; it is the only one that accepts writes. A write is not acknowledged until a majority of nodes has durably stored it — which is what lets any minority of them fail without losing an acknowledged write.
client
│ insert / update / delete / SQL write
▼
┌───────────┐ the follower does not apply it;
│ node 2 │──┐ it forwards to the leader
│ (follower)│ │
└───────────┘ │
▼
┌───────────┐
│ node 1 │ 1. append to own log
│ (LEADER) │ 2. replicate to followers
└─────┬─────┘ 3. wait for a MAJORITY (here: 2 of 3)
┌────────┴────────┐ 4. commit + apply + ACK the client
▼ ▼
┌───────────┐ ┌───────────┐
│ node 2 │ │ node 3 │
│ (follower)│ │ (follower)│
└───────────┘ └───────────┘
ACK only after a majority has it → any single node may now die
without the write ever being lost.Reads are served locally by whichever node you ask, which is why reads scale with the cluster and writes do not: a write costs a round trip to a quorum, every time.
A 3-node cluster survives 1 failure; a 5-node cluster survives 2. The rule is floor(N/2), and it is why cluster sizes are odd — a 4-node cluster also survives only 1 failure, so the 4th node buys nothing but another machine to pay for.
The majority rule is also what makes split-brain impossible. Cut a 5-node cluster into 3 and 2: only the side of 3 can form a majority. The side of 2 keeps running and keeps refusing writes — it cannot accept them, because it can never reach a quorum. Two halves can never both commit, because two majorities of the same cluster must overlap.
# three machines, each with its own id and Raft address
OXIDB_NODE_ID=1 OXIDB_RAFT_ADDR=10.0.0.1:4445 OXIDB_ADDR=10.0.0.1:4444 ./oxidb-server
OXIDB_NODE_ID=2 OXIDB_RAFT_ADDR=10.0.0.2:4445 OXIDB_ADDR=10.0.0.2:4444 ./oxidb-server
OXIDB_NODE_ID=3 OXIDB_RAFT_ADDR=10.0.0.3:4445 OXIDB_ADDR=10.0.0.3:4444 ./oxidb-serverThen bootstrap once, from node 1 — it starts as a one-node cluster, the others join as learners (they catch up without voting), and a membership change promotes them to voters:
oxidb --host 10.0.0.1 --port 4444
> raft_init
> raft_add_learner node_id=2 addr=10.0.0.2:4445
> raft_add_learner node_id=3 addr=10.0.0.3:4445
> raft_change_membership members=[1,2,3]
> raft_metrics # state, term, leader, per-follower progressThe learner step matters: a fresh node joining a cluster with a large log has to copy it, and until it has, counting it toward a quorum would stall every write. As a learner it catches up silently and only then starts voting.
| Engine | Replicated? | How |
|---|---|---|
| Document | Yes | Every mutating command goes through the log. |
| SQL | Yes — writes | Each statement is parsed to decide. A statement that only reads runs locally; anything that can mutate replicates. Stored-procedure CALLs count as writes, which is why Cobra procedures are validated as deterministic at CREATE: a procedure that could read the clock or the network would apply differently on each node. |
| Time-series | No — node-local | Replicate by ingesting into more than one node. |
| OxiMem / MQTT | No — by design | An in-memory cache and a message bus; neither wants a consensus round trip. |
The SQL classification is a parse, not a prefix match, which is the only way to get it right: WITH t AS (DELETE … RETURNING *) SELECT * FROM t starts with WITH and is unmistakably a write.
Replication cannot help with data that does not fit. Sharding splits a collection across independent OxiDB servers by a shard key you choose per collection. oxipool sits in front and makes the split invisible: clients speak the ordinary wire protocol and never learn a shard exists.
Keys are not hashed straight onto shards. They land on one of 256 virtual chunks, and a chunk map assigns chunks to shards:
doc = { "region": "eu-west", "bal": 120 }
│
│ shard key for this collection is "region"
▼
crc32("eu-west")
│
│ % 256
▼
chunk 173
│
│ chunk_map[173]
▼
shard 1
┌──────────┬──────────┬──────────┐
│ shard 0 │ shard 1 │ shard 2 │
│ chunks │ chunks │ chunks │
│ 0..85 │ 86..170 │ 171..255 │
└──────────┴──────────┴──────────┘The indirection is the point. Hash straight to shard = hash % 3 and adding a fourth shard re-maps almost every key — a total reshuffle. With a chunk map you move chunks: to add a shard, hand it a quarter of the chunks and only that data moves. The map is data, not arithmetic, so it can also be lopsided on purpose — a bigger machine can simply own more chunks.
ROUTED — the query carries the shard key: one shard answers.
find({region: "eu-west", bal: {$gt: 100}})
│
oxipool ──────────────► shard 1 (shards 0 and 2 idle)
SCATTER-GATHER — no shard key: ask everyone, merge.
count({bal: {$gt: 100}})
│
oxipool ──┬──► shard 0 ──┐
├──► shard 1 ──┤ merge
└──► shard 2 ──┘ │
▼
sum of the threeA routed query costs the same as a single-server query no matter how many shards there are — so choose the shard key to match how you actually query, not to look evenly distributed. A key that never appears in your queries turns every one of them into a fan-out.
| Command | Merge |
|---|---|
find, aggregate, search | Concatenate the documents |
count | Sum the counts |
update, delete | Sum the modified/deleted counts |
find_one | First shard with a match |
| DDL | Broadcast to every shard |
Two of those are subtler than they look.
find with skip/limit cannot be sent to the shards as written. A per-shard skip would drop up to skip × (shards−1) documents that belong in the answer — each shard would skip its own rows independently. So skip is removed on the way out and limit becomes skip + limit; the global window is a subset of the union of the per-shard windows, and the merge re-sorts and slices it globally.
update_one / delete_one without a shard key must not fan out at all. Sent to every shard, each one dutifully modifies one local document — up to N changes for a command that promised one, with the merge merely choosing which reply to show. Instead the shards are probed serially and it stops at the first that actually changed something.
# three plain OxiDB servers — they don't know they're shards
OXIDB_ADDR=10.0.0.1:4444 OXIDB_DATA=/data ./oxidb-server # … and .2, .3
# the router in front
OXIPOOL_LISTEN=0.0.0.0:4445 \
OXIPOOL_SHARDS=10.0.0.1:4444,10.0.0.2:4444,10.0.0.3:4444 \
OXIPOOL_SHARD_KEYS="accounts:region,events:tenant_id" \
OXIPOOL_NUM_CHUNKS=256 \
OXIPOOL_REQUEST_TIMEOUT=30 \
./oxipoolClients now connect to :4445 and use OxiDB exactly as before. Collections without a shard key live on shard 0 — sharding is opt-in per collection.
oxipool also runs in a plain master/replica mode, where it classifies each request and sends reads to replicas and writes to the master:
OXIPOOL_MASTER=10.0.0.1:4444 \
OXIPOOL_REPLICAS=10.0.0.2:4444,10.0.0.3:4444 \
./oxipoolClassification reads the request's cmd field — never the payload. Scanning the raw bytes for words like insert is the obvious shortcut and a real bug: a document that merely contains the word would reroute a read to the master, and one containing begin_tx would pin a pooled connection per occurrence. What a user stores must never steer routing.
The layers compose: shard for size, replicate each shard for survival.
clients
│
┌─────────┐
│ oxipool │ splits by shard key
└────┬────┘
┌──────────────┼──────────────┐
▼ ▼ ▼
┌───────────┐ ┌───────────┐ ┌───────────┐
│ shard 0 │ │ shard 1 │ │ shard 2 │ ← a third of the data each
│ ┌─────┐ │ │ ┌─────┐ │ │ ┌─────┐ │
│ │ ldr │ │ │ │ ldr │ │ │ │ ldr │ │
│ └──┬──┘ │ │ └──┬──┘ │ │ └──┬──┘ │ ← each shard is its own
│ ┌──┴──┐ │ │ ┌──┴──┐ │ │ ┌──┴──┐ │ Raft group: 3 copies,
│ │f f│ │ │ │f f│ │ │ │f f│ │ its own leader, its own
│ └─────┘ │ │ └─────┘ │ │ └─────┘ │ majority
└───────────┘ └───────────┘ └───────────┘
9 machines · 3× the capacity of one · any one machine may dieEach shard elects its own leader and reaches its own quorum; a shard losing a node is that shard's problem and nobody else's. What the layers do not give you is a transaction across shards — each shard commits independently, so cross-shard atomicity is not on offer. Pick a shard key that keeps things which must change together on the same shard (a tenant, an account, a region), and cross-shard writes stop being something you need.
The SQL and time-series engines are not sharded — oxipool splits on a collection's shard key, and a SQL statement has no collection to split on. SQL scales by replication (reads across replicas, writes through the leader), not by sharding.
The interesting question isn't whether a cluster survives a clean crash — it's what it does when the network lies. These are OxiDB's tested behaviours, each pinned by a test that injects the fault deterministically at the transport.
╔═══════════════════╗ ╳ ╔═══════════════╗
║ n1 n2 n3 ║ ╳ ║ n4 n5 ║
║ majority (3) ║ ╳ ║ minority (2) ║
║ elects a leader ║ ╳ ║ cannot elect ║
║ keeps committing║ ╳ ║ refuses writes║
╚═══════════════════╝ ╳ ╚═══════════════╝
heal ──► minority discards nothing it
committed (it committed nothing)
and catches up. All 5 converge.A clean split is polite — both ends agree the other is gone. Real networks drop one direction and leave the two ends disagreeing about whether the peer is alive.
A node that hears nothing but can still speak gets no heartbeats, so it calls an election — and since its own messages get through, it simply wins and leads. No harm done.
The dangerous one is a node whose requests land but whose replies never come back. It asks for votes, they are granted, it never learns, so it times out and asks again at a higher term — forever. This is Raft's classic "disruptive server": each campaign could force the real leader to step down, and the cluster would lose its leader over and over to a node that can never win. It doesn't happen here, because a follower that has heard from a live leader recently refuses a vote request without adopting its term. Under test, the disruptor climbed to term 16 while the healthy cluster sat at term 5 and committed every single write.
For a sharded cluster the failure is different in kind — and the danger is not an error, it's an answer. A fan-out has every shard's reply in hand and must fold them into one. Fold only the ones that came back and the client gets ok: true and a perfectly plausible number that is quietly missing a third of the data — a count of 40 where the truth is 60. That is worse than an outage, because an outage is visible.
So every merge fails loudly if any shard did not answer. Even find_one: "not found" is only true if every shard was asked, and the document may be living on precisely the shard that is down. Meanwhile a query that carries the shard key still routes to a live shard and answers normally — one dead shard must not take down the healthy ones.
shard 1 down:
count({}) → ok:false "…failed on one or more shards"
(NOT 40 — the plausible wrong answer)
find({region:"eu"}) → ok:true (routed to shard 0, which is fine)A shard that has crashed is the easy case — the connection breaks and everything errors at once. A shard that is partitioned is worse: it accepts the request, reads it, and never answers. Nothing fails; the call simply never returns, and the pooled connection it borrowed is never given back — repeat that and the pool drains until the whole router hangs. OXIPOOL_REQUEST_TIMEOUT (seconds, 30 by default) is what turns that silence into an error.
More: Server & configuration · Transactions & consistency · SQL engine