Introduction
When you run points across two systems (POS and EC), the balances will drift apart sooner or later. A connection drops, processing fails on only one side, a resend runs twice. The causes vary, but what will happen, will happen.
What matters is less "making sure it never drifts" and more "being able to return to the correct state when it does." In this article, I'll explain how to use the audit log for reconciling and recovering balances, based on the design I actually built. How you hold the log completely changes how painless failure response becomes.
Why Balances Drift
Crossing Systems Creates Two "Correct Answers"
The POS and the EC site each have their own balance. As long as the two always match, there's no problem — but when one update succeeds and the other fails, two "correct answers" are born in that instant.
The in-store spend of 500 points succeeds
Only the reflection to the EC side fails on a connection error
Since both numbers are "correct within their own system," comparing the systems against each other won't settle the answer. This is the root of what makes reconciliation hard.
Resends and Reordering
Another cause is the resending and reordering of processing. When the network is unstable, the same instruction can arrive twice, or an instruction issued later can arrive first.
If you hold only "the current value" of the balance, you can't absorb this kind of disorder. But if you have an event log that records each change one by one, you can re-sort by time, reject duplicates, and redo the correct stacking. The log is like a script for setting a disordered reality straight again.
Rebuilding the Correct State from Logs
Derive the Balance as the Sum of Events
The basic recovery policy is simple. Rather than a human guessing at the broken balance, you re-sum the event log from the beginning to compute the balance as it ought to be. That's the whole of it.
Pull out all of that customer's grants, spends, and conversions
Put them in the correct order by occurred_at
Collapse events with the same unique key into one
The total of the changes is the balance as it ought to be
You treat the "balance as it ought to be" as authoritative and overwrite the number on the system that had drifted. Because the answer comes from the fact of the history rather than human intuition, you can respond with confidence.
Speed Up Reconciliation with balance_after
If you record "the balance after the operation (balance_after)" for each event, reconciliation gets dramatically faster. You don't have to sum all events — you can detect an anomaly just by checking "does the balance_after of the last event match the current balance?"
Balance snapshots as checkpoints
Summing everything each time gets heavy as events pile up. If you stamp the balance onto each event, it becomes a checkpoint along the way, letting you quickly pinpoint "which event the drift started from."
Once you know the point where the difference appears, you can narrow the investigation to just that spot. Being able to say "it went wrong from this grant" rather than "it drifted somewhere" directly translates into response speed.
Idempotency Makes Recovery Safe
The recovery process itself can fail and be re-run. If that process isn't idempotent, it might move the balance twice again — the balance you just fixed.
That's exactly why enforcing duplicate prevention with a unique key at the event-recording stage also supports the safety of recovery. The property that "running the same operation any number of times yields one result" is what makes re-running failure responses nothing to fear.
Building Reconciliation into Operations
Regular Automatic Reconciliation
You never know when a failure will strike. So I set up a mechanism that periodically — overnight, for instance — cross-checks the POS and EC balances and detects any drift. The goal is to create a state where the system notices before a human does.
Make "Log Is Authoritative" the Only Recovery Rule
As an operational rule, I insisted on "when in doubt, treat the log as authoritative." Start debating whether the POS or the EC is right, and recovery goes nowhere. Decide in advance that the audit log — where every change remains — is the single source of truth.
Thanks to this consistent policy, you can restore the correct state mechanically, without panic, during a failure. Designing so that you can trust the log is, in the end, what helps the front line the most.
Conclusion
Balances drifting is unavoidable. That's exactly why it pays to hold the audit log as a "ledger for restoring the correct state."
- Recompute the balance as the sum of events — fix it from fact, not intuition
- Speed up reconciliation with balance_after — quickly pinpoint where drift began
- Make recovery re-runs safe with idempotency — the same operation yields one result no matter how many times it runs
The event schema that underpins this log is explained in "Designing Event Records for Grants, Spends, and Conversions," and the big picture in the hub article "Point Audit Logs."