Idempotency

Last time, the takeaway was that retrying a request without reading what the failure actually means is how a rare, deterministic error turns into unnecessary load. This one is the flip side of that same problem - not "should this be retried," but what happens when something does get sent twice, on purpose or by accident, and the operation it triggers can't safely run more than once.

This wasn't hypothetical for us. During a peak-traffic window, the reports started coming in - users getting force-logged-out mid-session, no obvious pattern to point to. First guess was a frontend glitch, some lag or race condition in the client. The frontend code checked out clean, nothing there explained it. Backend looked the same way at first pass, no obvious logic error jumping out either. It was only once we traced the idempotency of the access-token refresh endpoint specifically that the actual picture showed up. The same defect turned out to be quietly breaking MFA setup too, the exact same race making that flow hard to finish for anyone unlucky enough to hit it. By the time we'd actually pinned it down, finding it felt less like a clean trace and more like we'd gotten there on luck - the kind of debugging session that ends with someone joking we should've just lit some incense and asked for help instead.

Two endpoints in our auth flow, refresh token exchange and MFA verification, both consume a single-use token and exchange it for a new session. Both were marked idempotent: false in the API docs. That annotation was accurate, and also completely useless - it was documentation text, nothing in the code actually enforced it. A duplicate request carrying the same token (double-click, a client retrying on timeout, a background refresh racing a foreground one) could race the same token through two concurrent executions. Whichever request lost surfaced as a hard failure. For token refresh specifically, that means the user gets logged out mid-session with no idea why.

Why this class of bug is different from the retry bug

Last article's bug was about reading an error correctly before deciding to retry. This one is about a different property entirely: some operations are safe to repeat, some aren't, and consuming a single-use token is firmly in the second group. There's no way to make "consume a token" naturally idempotent the way POST /block is (call it 100 times, end state is always the same) - it needs an artificial guard bolted on, because by design the second call is supposed to fail.

The other piece worth naming: HTTP clients default to at-least-once delivery. Timeout-then-retry is standard behavior in most mobile SDKs, not an edge case. Exactly-once processing isn't something you get from the network - it's at-least-once delivery plus a consumer that knows how to deduplicate. Before this fix, the gateway assumed one HTTP request equals one logical operation. That assumption just doesn't hold once retries are in the picture, and nothing here was accounting for it.

What this looks like for a real user

The bug above is real - this is what led us to it. The blow-by-blow timing below is reconstructed, not a literal transcript of any one report:

An access token is close to expiring, so the mobile app fires a background refresh over a degraded cellular connection (an elevator, basement parking, transient 3G packet drop). The first request isn't dead, just delayed in transit. After an aggressive 800ms client timeout, the frontend's standard retry interceptor fires a second identical request carrying the exact same single-use refresh token.

Both requests end up racing into the auth service within milliseconds of each other. Without a deduplication guard, whichever request reaches the database first consumes and rotates the token, issuing a new session. The second request arrives a split second later, finds the token already marked consumed, and gets rejected with a hard 401 Unauthorized (INVALID_TOKEN). The client's global error handler reads that 401 as "the refresh token was revoked," wipes local secure storage, and force-logs the user out in the middle of their active session.

The failure race: how uncoordinated retries break a valid session

Both requests were completely legitimate, emitted by the exact same user with a valid token. The only thing that turned the second request into a fatal error was a few milliseconds of network jitter - which is why reproducing this bug in manual QA is close to impossible, but in production at scale it generates a steady trickle of unexplained logout complaints.

The fix I expected to be simple

Before writing anything custom, I checked whether we already had reusable lock infrastructure - we did. The internal Redis library ships a distributed-lock decorator: retries, TTL, fencing tokens, auto-renew, all there. Nobody in this codebase was using it on any route.

Reading through the compiled source to see how to wire it in, the acquisition call looked like this:

await this.redisStringService.set(lockKey, lockValue, {
  nx: true,
  ttl: Math.floor(ttl / 1000),
});

And the set() implementation underneath it:

if (options?.ttl) {
  await this.redisClient.set(key, strValue, 'EX', options.ttl);
}
else if (options?.nx) {
  await this.redisClient.set(key, strValue, 'NX');
}
else if (options?.xx) {
  await this.redisClient.set(key, strValue, 'XX');
}
else {
  await this.redisClient.set(key, strValue);
}

Those branches are if / else if, so only one of them ever runs. Which means the options you pass in aren't combined - they compete, and ttl wins:

The third row is the one that matters, because it is the only row the lock decorator ever produces: it defaults ttl to 30 seconds when you don't specify one, so every acquisition call passes both. The nx is silently dropped, and what runs is an unconditional overwrite.

So the lock always "succeeds." Not sometimes, not under contention - always, for everyone, including the caller who is stomping on the lock someone else is currently holding. A mutual-exclusion primitive that can never fail to acquire is not a weak lock, it's not a lock at all, and it fails in exactly the shape of a check that cannot fail: every call returns success, so nothing downstream ever looks wrong until two things run at once.

That ruled it out as the implementation vehicle for this fix - and it's worth flagging separately to whoever owns that library, since it's not something fixable from this repo, and anyone else who reaches for that decorator expecting mutual exclusion is getting none.

What I built instead

Constraints going in:

• No new client-facing contract - no new header, no new field, zero frontend changes.
• Key derived from a hash of the token itself, not a client-generated identifier. The token is already unique and single-use; there's nothing new to generate.
• Short TTL (5 seconds) - this only needs to outlive one request's round trip, not act as a long-lived lock.
• Fail-open on Redis errors. This is a UX/consistency guard, not a security control - if Redis is down, real logins and refreshes should not get blocked by a broken dedup check.
• 409 Conflict on a rejected duplicate.

async tryAcquire(scope: string, token: string, ttlSeconds = 5): Promise<boolean> {
  const key = this.buildKey(scope, token);
  const probe = randomUUID();
  try {
    // Deliberately omitting ttl here - the internal lock library's set()
    // takes the ttl branch and silently drops nx whenever both are passed.
    await this.redisString.set(key, probe, { nx: true });
    const owner = await this.redisString.get(key);
    if (owner !== probe) return false;
    await this.redisMain.expire(key, ttlSeconds);
    return true;
  } catch (error) {
    this.logger.warn(`Lock check failed, failing open: ${error}`);
    return true; // not a security control - must not block real logins
  }
}

Wired into both endpoints right after existing input validation, before the command executes - reject with 409 Conflict if tryAcquire returns false.

The gap in that two-step, and why it's worse than it looks

There's a window between the NX set succeeding and the follow-up EXPIRE, and the obvious reading is "if the process dies in there, the key never expires." True, and rare enough to shrug at. But process death isn't the only way to land in that window, and the other way is much more ordinary: the get() or the expire() simply throws. A momentary Redis blip, a connection reset, a timeout - any of those lands in the catch, which returns true and fails open. The request proceeds, which is correct. What's left behind is not: a key that was successfully set, with no TTL attached, that nothing will ever remove.

And that key isn't inert. The next request carrying the same token finds it, reads an owner that doesn't match its own probe, and returns false - so the user gets a 409 on a request that deserved to succeed, permanently, for that token. The blast radius is one token, which is small; the failure is silent and unrecoverable, which is not.

The two-step is what the vendored wrapper forces, because its set() cannot express NX and TTL at the same time. But the underlying client can - Redis has done this in one atomic command for years:

// One command, no window, no cleanup path to get wrong.
const acquired = await this.redisMain.set(key, probe, 'EX', ttlSeconds, 'NX');
if (acquired !== 'OK') return false;
return true;

Which is the actual lesson from this whole section, and it isn't about Redis. The vendored library was broken, so the fix routed around its set() for the NX part - but kept using it, and inherited a two-step where one step existed. Once you've established that an abstraction can't be trusted for the thing you need, the move is to go under it, not to build a careful workaround on top of it.

What's deliberately not done yet

This was scoped as a minimal, contract-preserving change: reject the race, nothing else. Two things are proposed but intentionally left for later: making the API docs actually describe the mitigation instead of just saying "not idempotent," and logging rejected duplicates so a future "why was I logged out" ticket has something concrete to check instead of a guess.

There's a third piece that didn't make it into this fix at all, and it's a different kind of problem, not a race between duplicate requests anymore, but what it means when the same token shows up again well after the fact. That's not a network retry. That's a signal worth treating differently, and it deserves its own post rather than a rushed paragraph at the end of this one.

Runnable Reproduction

A sanitized, standalone reproduction demonstrating the race condition, the defective vendored lock, and the atomic dedup guard is available in the lab repository:

• engineering-notebook-labs/idempotency