Cutting payment latency with a Rust-first backend pod

The situation

The company processes card and UPI payments for small and mid-sized merchants. Its authorisation path, the code that decides in real time whether a payment goes through, lived inside a large service that had grown over several years. Most of the time it was fine. During sale seasons, when traffic spiked, tail latency climbed, some payments timed out and merchants noticed.

The in-house team knew the problem well but was stretched across product work. Hiring senior systems engineers through the usual channels was taking months, and the next peak season was a quarter away.

The brief

The CTO gave us a brief of a few sentences: keep p99 latency on the authorisation path inside a defined budget at peak traffic, without a risky big-bang rewrite, and leave the team stronger than it found it. The engagement had to start within weeks, not months.

What we did

We shortlisted four candidates per role from engineers who had already cleared our screening, weighted towards people who had worked on latency-sensitive systems. The company interviewed eight people over a week and chose three senior backend engineers. We also proposed a graduate of the Fellowship's engineering track to work alongside them, focused on testing and tooling.

The pod's first rule was to measure before changing anything. Two weeks of profiling showed that most of the tail latency came from a small number of causes: lock contention in a shared cache, garbage-collection pauses under load and a synchronous call that could be moved off the critical path.

• Carve out, don't rewrite. The authorisation hot path was extracted into a small Rust service behind the existing gateway, with the same API contract.
• Prove it in the shadows. The new service ran on mirrored production traffic for several weeks, with its decisions compared against the old path.
• Safety first. Idempotency keys, strict timeouts and a one-switch fallback to the old path were in place before any real traffic moved.
• Roll out gradually. Traffic moved in stages, merchant segment by merchant segment, with a written go or no-go at each step.

How it went

By the fourth month the new path was handling all authorisation traffic. Tail latency at peak dropped substantially and stayed inside the budget through the following sale season. Infrastructure cost for that path also fell, because the new service needed far fewer resources for the same load.

Two of the three senior engineers converted to full-time roles at the end of the engagement. The Fellowship graduate built the load-testing harness the team still uses, and was hired as well.

What we'd tell another team

• Profile before you rewrite. The fix was smaller than anyone expected.
• Use Rust where it earns its keep: small, hot, well-defined paths. Not everything needs to move.
• Shadow traffic turns a scary migration into a boring one. Boring is the goal.

The pod did the least glamorous thing first: they measured. That discipline is what we screen for, and it is why the migration was uneventful.
Saurav Kumar Jha, Founder & CEO