In-Memory Systems with Durable Backing

Learning / Data Systems / Data Architecture and Platforms / 022

LESSON

Data Architecture and Platforms

022 30 min advanced

Day 470: In-Memory Systems with Durable Backing

The core idea: An in-memory system is only trustworthy when its write path makes a deliberate durability promise through logs, snapshots, and replica handoff.

Today's "Aha!" Moment

After 021.md, PayLedger has a solid plan for compressed historical segments and can prove archived payout bytes are intact. That helped the audit path, but it did not fix the hot path. During payroll cutoff, the balance-gate service must reserve funds for each payout instruction in under 5 ms so the UI can confirm or reject a transfer immediately. Querying the relational ledger on every reservation keeps durability obvious, but it pushes p99 latency high enough to stall payroll operators at exactly the worst moment.

The team moves the working set into RAM instead. Each merchant's available balance, open holds, and payout sequence counters live on a single shard owner in memory. Durable backing comes from an append-only log on local SSD, a replica stream to a standby, and periodic snapshots to object storage. The easy mistake is to call this "just a cache in front of the database." Once balance-gate tells the caller that a hold exists, that answer changes money movement. The real question is not whether the latest bytes happen to be in memory or on disk. The real question is what durability barrier has been crossed before the system says "success."

That is the useful shift. The log defines operation order. The snapshot defines restart speed. Replica acknowledgment defines the failover loss envelope. If those choices are vague, the system can be fast and still recreate already-spent balance after a crash. In-memory systems with durable backing are not fast because they ignore storage; they are fast because they reduce storage to a disciplined write-ahead and recovery path.

Why This Matters

Teams reach for memory-first systems when the working set fits comfortably in RAM and the product pays heavily for disk round-trips. Rate limiters, fraud counters, session stores, market data books, and balance reservation engines all fit this pattern. The production pressure is that the same system still needs restart recovery, replay, auditability, and controlled failover. That creates the core trade-off of the lesson: every step that tightens durability usually adds latency, write amplification, or operational complexity.

For PayLedger, the failure is concrete. If balance-gate mutates RAM and returns success before its log record is durable, a node power loss can erase an approved hold and let the same funds be reserved again. If it waits for a local fsync on every single request, latency spikes. If it also waits for a replica to acknowledge the write, the data-loss window gets smaller, but the failover procedure must fence the old primary carefully or two owners may believe they control the same merchant shard. The right design is the one whose crash window, recovery time, and latency envelope are all stated explicitly.

The previous lesson established that durable bytes are only useful if the system can detect corruption and trust recovered data. This lesson moves that same integrity model into the live state path: logs and snapshots are now the recovery substrate for state that usually lives in memory. The next lesson, 023.md, picks up the next pressure point. Once the hot state is fast enough, a few large merchants can dominate one shard and create hotspots.

Core Walkthrough

Part 1: Grounded Situation

PayLedger partitions balance-gate by merchant_id. At any moment, exactly one shard owner is allowed to accept writes for a merchant. That avoids multi-writer races in the low-latency path and gives the service a deterministic command stream. A reservation request such as reserve_funds(merchant_id=1842, reservation_id="run-2026-03-31-447", amount_cents=125000) is routed to the owner for shard 1842, which checks the in-memory balance map and decides whether the command is admissible.

The naive alternative is a dual write: update an in-memory cache for speed, then persist to a database for durability. That looks simple until one write succeeds and the other does not. A retry can then double-apply the hold, or a failover can reconstruct an older balance than the API already exposed. PayLedger avoids that split-brain behavior by treating the ordered command stream as the source of truth and the in-memory map as the materialized state used for fast reads.

The data flow is narrow on purpose:

client
  -> router
      -> shard owner
          -> in-memory balance map
          -> append-only WAL segment
          -> replica stream
          -> periodic snapshot writer

That layout tells you where the guarantees live. The RAM map makes reads cheap. The WAL is the durable history of accepted commands. The snapshot shortens replay. The replica gives a second copy for failover and recovery.

Part 2: Mechanism

On the write path, PayLedger treats each reservation as a command against a shard-local state machine. The owner first checks an idempotency index keyed by reservation_id. If the caller is retrying a timed-out request, the service must return the same answer instead of attempting a second reservation. If the request is new, the owner evaluates it against the current in-memory balance and emits a command record that includes the decision, the delta, and a monotonically increasing log sequence number (LSN).

That record is appended to the WAL before the result is exposed externally. PayLedger uses 4 ms group commit rather than fsync on every request, because the extra batching keeps the hot path within budget while still giving a crisp durability point. Commands whose WAL records are flushed are considered committed. Only then does the owner publish the updated balance and return success to the caller.

def reserve_funds(cmd, shard):
    if cmd.reservation_id in shard.decisions:
        return shard.decisions[cmd.reservation_id]

    decision = evaluate(shard.state, cmd)
    entry = wal.append(encode(cmd, decision))
    wal.flush_group_until(entry.lsn)

    if decision.approved:
        shard.state.apply(decision.delta)

    shard.decisions[cmd.reservation_id] = decision
    replica.publish(entry.lsn, cmd, decision)
    return decision

The exact sequence is more important than the syntax. Appending to the log before acknowledging the write means the service can recover committed commands in order after a crash. Recording the decision under the idempotency key means retries can be answered deterministically. Streaming the committed LSN to a replica means failover can be expressed as "which node has applied up to LSN N?" rather than "which node feels freshest?"

Snapshots solve a different problem. A pure append-only log is enough for correctness, but recovery time grows with log length. PayLedger writes a snapshot of each shard every 30 seconds or every 256 MiB of committed WAL, whichever comes first. Each snapshot stores the full balance map, active holds, idempotency window, and the snapshot_lsn it represents. On restart, the node loads the newest verified snapshot, then replays only log records with LSNs greater than snapshot_lsn. If the latest snapshot is at LSN 18,240 and the crash left 746 newer committed records, recovery is seconds long. Without snapshots, replay could take millions of commands and miss the restart SLO.

The previous lesson matters here because logs and snapshots are only useful when their bytes can be trusted. PayLedger stores checksums on WAL frames and snapshot blocks. Recovery replays until the last valid record, truncates any torn tail, and pulls missing durable data from the replica if needed. Durability is not "we have a backup somewhere." Durability is "every acknowledged command can be reconstructed in order from verified bytes."

Part 3: Implications and Trade-offs

Once the mechanism is explicit, the trade-offs become clear instead of ideological. A service that acknowledges after writing to RAM but before flushing the WAL gets excellent latency, but the crash window includes acknowledged writes. A service that acknowledges only after local fsync gives stronger single-node durability, but p99 latency rises with storage jitter. A service that waits for both local durability and replica acknowledgment shrinks the failover loss window further, but it now depends on replica lag, election rules, and fencing to avoid split brain. None of these is universally correct; the right choice depends on whether the product can tolerate a few milliseconds of extra latency more easily than it can tolerate replay gaps.

Snapshot cadence has its own trade-off. Frequent snapshots reduce replay time and make failover calmer, but they add I/O load and serialization cost while the system is already busy. Sparse snapshots minimize background work, but replay becomes long and promotion can stall under pressure. The operational answer is to measure the boundary directly. PayLedger watches wal_flush_p99_ms, replay_seconds, snapshot_age_seconds, replica_apply_lag, and the share of traffic carried by the busiest shard. Those metrics reveal whether the next problem is storage durability, recovery time, or emerging shard hotspots.

The important design consequence is that an in-memory system with durable backing is not "memory plus some backups." It is a state machine with three carefully chosen artifacts: a log for order, a snapshot for restart time, and a replication boundary for failover semantics. Once those are explicit, performance tuning becomes honest because the team can say exactly what is being traded away and what is being protected.

Failure Modes and Misconceptions

Connections

Resources

Key Takeaways

PREVIOUS Compression, Checksums, and Corruption Handling NEXT Hotspot Mitigation Strategies

← Back to Data Architecture and Platforms

← Back to Learning Hub