What metrology taught me about payment gateways

How error accumulates

For independent contributions, the combined uncertainty is the sum in quadrature:

u_c = sqrt(u1² + u2² + u3² + u4²)

The practical consequence is that the largest contribution dominates the result. With u1 = 0.1, u2 = 0.1, u3 = 0.1 and u4 = 1.0, the combined value is 1.015. Eliminating the first three links entirely would bring the total to 1.0. All that effort for a 1.5% gain.

That holds for independent errors. If the contributions are correlated (every link using the same biased standard, or all of them measuring in the same environment with the temperature out of range), they add linearly:

u_c = u1 + u2 + u3 + u4

The accumulation gets much worse. Correlated error is the nastiest problem in the chain, because you think you’re adding noise when you’re actually adding bias.

Industrial calibration has a rule of thumb called TUR (test uncertainty ratio): the standard you use must be at least 4 times more precise than the tolerance you want to guarantee. For a tolerance of ± 0.1 mm, the standard needs an uncertainty better than 0.025 mm. Measuring with a precision close to the tolerance is practically not measuring at all.

Calibration doesn’t eliminate error

Calibrating doesn’t make the instrument perfect. Calibrating means determining its error at specific points in its range and, when possible, correcting the bias. A calibrated instrument isn’t one that never errs. It’s one whose error is known and documented.

That changes three things in day-to-day operation: you know in which range to trust the instrument, you know it drifts over time and has to go back to the lab, and you know whether its current error is acceptable for the decision you’re going to make with that number.

Composite availability

If each link has independent availability, end-to-end availability is their product:

A_total = A1 × A2 × A3 × A4

With plausible numbers:

Each extra layer costs roughly 43 minutes of downtime per month, even if each one looks excellent on its own. It’s the same phenomenon as uncertainty accumulating: you can’t be more reliable than the product of your links.

The worst link dominates here too. A gateway at 99.5% in front of a participant at 99.95% gives you 99.45%. In that scenario, improving your application from 99.99% to 99.999% changes nothing in the number your customer sees. It’s the fast station being optimized while the bottleneck stays put.

The math above assumes independent failures. If two different gateways settle through the same direct participant, they aren’t redundant: when the participant goes down, both go down together. That’s exactly metrology’s correlated error, which adds linearly instead of in quadrature.

State isn’t the same everywhere

This is the quietest problem in the chain. Each link keeps a copy of the payment’s state, and those copies don’t change at the same time.

sequenceDiagram
  participant SPI as SPI (Bacen)
  participant PD as Direct participant
  participant GW as Gateway
  participant App as Your application

  SPI->>PD: settled at T0
  Note over PD: state updated at T0 + 200ms
  PD->>GW: notification
  Note over GW: state updated at T0 + 900ms
  GW-->>App: webhook
  Note over App: state updated at T0 + 2s<br/>(or never, if the webhook fails)

Between T0 and the moment your application updates, there’s a window where you hold an outdated measurement without knowing it. If the webhook gets lost, that window never closes.

Anyone who works with payments knows the failure modes of this window:

• Lost webhook: the payment settled, the customer paid, and your system never found out. The customer shows up at support with the receipt in hand.
• Duplicate webhook: the same notification arrives twice because of a retry. Without idempotency, you credit twice.
• Out-of-order webhook: CONFIRMED arrives after EXPIRED because they took different routes.
• Ambiguous timeout: you asked for a charge to be created, it timed out, and you don’t know whether it exists on the other side. It’s the worst kind of error, the one that doesn’t tell you whether it happened.

Notice that none of this is random error. It’s systematic error. Querying the same gateway again doesn’t help, because if it has the wrong state it will return the wrong state very consistently. To correct bias you have to measure against a higher-level standard.

Regulation and compliance

Metrology has a risk that isn’t technical: the lab that calibrates your standard can lose its accreditation. Overnight, every certificate it issued becomes questionable and your traceability chain becomes invalid, even with physically perfect instruments.

The same thing happens in the payments chain, and it’s the category of risk that shows up least in gateway comparisons:

• The direct participant is the one accountable to Bacen. If it gets sanctioned, has its authorization suspended or goes under special administration, everyone hanging off it stops operating. You did nothing and you’re still down.
• The contract your operation rests on isn’t yours. The agreement between the direct participant and the intermediary PSP can change without you being part of the conversation. A risk policy change at a link you don’t even know about takes down your flow.
• New requirements travel down the chain late. A Bacen requirement reaches the direct participant, which passes it to the PSP, which passes it to the gateway, which has to expose it in the API so you can comply. Each translation loses information, and the compliance deadline stays the original one.
• Compliance becomes a black box. When your customer is blocked by a risk review, who made the decision? In a four-link chain, finding out turns into days of email, over a rule that directly affects your revenue.
• The data surface grows. Transaction data and personal data pass through one more third party. Under LGPD, each intermediary is one more processor in the flow, one more contract, one more place to audit and one more possible leak point.
• Custody of the money gets murky. Between settlement and the payout to you, whose account exactly holds the funds? In a long chain the answer is usually “someone in the middle”, and that link’s credit risk is yours.

Pick the reference standard

You need to define which source is the truth for your system, and it can’t be your own database. In practice the closest you can get is the direct participant’s statement, which is the link nearest to the SPI that you have access to.

Everything else (API responses, webhooks, cache) is measurement, is subject to error and needs to be checked against that standard periodically.

Reconciliation is calibration

Reconciling isn’t an accounting chore, it’s the system’s calibration. It answers exactly the metrological question: what is the current error of my measurement relative to the standard?

flowchart LR
  Extrato["Participant statement<br/>(reference standard)"]
  DB["State in your database<br/>(instrument)"]
  Diff["Diff"]
  Metric["Divergence metric<br/>+ alert"]

  Extrato --> Diff
  DB --> Diff
  Diff --> Metric

And the result has to become a number, just as a calibration certificate does:

# HELP payments_reconciliation_divergence_total Divergências entre estado local e extrato
# TYPE payments_reconciliation_divergence_total counter
payments_reconciliation_divergence_total{type="missing_locally",provider="gw_a"} 12
payments_reconciliation_divergence_total{type="missing_upstream",provider="gw_a"} 0
payments_reconciliation_divergence_total{type="amount_mismatch",provider="gw_a"} 1

# HELP payments_state_lag_seconds Atraso entre liquidação na origem e atualização local
# TYPE payments_state_lag_seconds histogram
payments_state_lag_seconds_bucket{le="1",provider="gw_a"} 8420
payments_state_lag_seconds_bucket{le="5",provider="gw_a"} 9910
payments_state_lag_seconds_bucket{le="30",provider="gw_a"} 9998
payments_state_lag_seconds_bucket{le="+Inf",provider="gw_a"} 10000

With the provider label you can compare providers against each other and alert when one of them starts to drift. Without these metrics you don’t know the instrument’s error, you just trust it.

A webhook is not a source of truth

A webhook is push: no guarantee of delivery, ordering or uniqueness. Its job is to cut latency by telling you something changed. Correctness comes from querying and reconciling.

async function onWebhook(event: GatewayEvent) {
  // without validating the signature, the sensor accepts any input
  if (!verifySignature(event)) return reject()

  // idempotency by the provider's key, not by a locally generated id
  if (await alreadyProcessed(event.endToEndId)) return ack()

  // don't trust the payload: query the state again at the source
  const state = await gateway.getCharge(event.chargeId)

  // transitions only move forward, never back
  await applyStateTransition(event.chargeId, state)

  return ack()
}

The three points that save the most headaches:

• Idempotency by the provider’s business key. In Pix the endToEndId is stable across the whole chain, so it works as a key even if the payment passes through several links.
• Querying again instead of trusting the webhook body, which may be stale even relative to the gateway that sent it.
• A monotonic state machine. A charge that is already CONFIRMED doesn’t go back to PENDING because an old event arrived. It’s the equivalent of rejecting a reading outside the instrument’s range.

Measure each link separately

In metrology, breaking down the uncertainty tells you which contribution to go after. Same thing here: measuring only end-to-end latency doesn’t tell you where to invest.

Instrumenting per hop, with provider as a label, surfaces things like “the gateway responds in 80 ms but the participant behind it takes 1.2 s”. Without that breakdown you don’t know which link is the weakest, and without knowing that, every optimization is a gamble.

Do the math before choosing

Before signing with a gateway, you can do the same calculation a metrologist would, which is checking whether the total uncertainty fits within the tolerance:

• Is this provider a direct participant? If not, how many links are there between it and the SPI? It’s worth asking this explicitly, because it’s common for a product to describe itself as a “Pix connection” without making clear that it settles through three layers.
• What availability do I need to deliver to my customer?
• Multiplying the SLAs of the whole chain, do I reach that number?
• Who is the direct participant at the end? Is it the same one as my other providers?
• What does this intermediary give me that makes up for the links it adds?

The TUR logic fits well here. If you need to deliver 99.9% and pick a path that delivers 99.9% in the best case, it’s like measuring a 0.1 mm tolerance with an instrument that has 0.1 mm of uncertainty. You need slack.

Redundancy only counts if it’s independent

Having two gateways looks like redundancy, but if both settle through the same direct participant, they share a failure mode. Real redundancy requires paths that split as close to the source as possible.

flowchart TB
  SPI["SPI"]
  P1["Participant A"]
  P2["Participant B"]
  G1["Gateway 1"]
  G2["Gateway 2"]
  App["Your application"]

  SPI --> P1
  SPI --> P2
  P1 --> G1
  P2 --> G2
  G1 --> App
  G2 --> App

If the two paths converge on a single participant before Bacen, you have two integrations to maintain and a single point of failure.

The cost of running several providers

This is the most common way out for people who use gateways. Since no single provider delivers the availability you need, you sign up two or three and put a routing layer in front that switches routes when one starts failing. In reliability terms this is parallel redundancy, and it works: with two independent paths at 99.75%, the chance of both being down at the same time drops to 0.0006%.

The problem is that the availability math gets better while the complexity math gets worse, and nobody puts the second one in the spreadsheet before deciding:

• Each provider has its own model. Different state names, different error codes, different webhook formats, different retry policies. You end up writing a translation layer to a canonical internal state, and that layer becomes where the bugs show up.
• Reconciliation multiplies. Each provider has its own statement, its own cutoff time and its own way of identifying the transaction. Reconciling two is more than twice the work of reconciling one, because now there’s the case of a charge created on one and paid through the other.
• Routing needs state. Deciding which provider to use requires knowing each one’s health in real time, with health checks, a circuit breaker and criteria for switching back. A poorly calibrated router drops good traffic or keeps insisting on a dead route.
• Failover on top of in-flight operations is the hard case. Switching routes is simple for a new charge. For a charge already created on the provider that went down, it isn’t: it exists there, it can be paid at any moment and it’s still your responsibility.
• The test surface grows with it. Each provider’s sandbox, each one’s behavior on timeout, and the failover scenario, which is precisely the hardest to reproduce and so almost never gets tested.

The upshot is that you trade a reliability problem for an engineering problem, and you end up maintaining in-house something that looks a lot like a payment orchestrator. For a large operation that pays off. For a small team it usually costs more than it seems, and it’s worth noticing that much of this work exists to make up for the instability the extra links introduced.

The same caveat about correlated error applies: two providers only improve availability if they’re truly independent. If both settle through the same direct participant, you paid all that complexity cost and still have a single point of failure.