Zero-Trust Cryptographic Key Engine

Have you ever looked at how a JWT actually gets signed across a real platform and wondered where the trust really lives? On paper it's solved - pull in a crypto library, call sign(), done. But once a few dozen services all need to verify tokens on their own, the question stops being "how do I sign a token" and becomes "who holds the key, and what happens the day that key has to change."

That was roughly where we were. Authentication ran through one auth service using a single symmetric HMAC secret. Any downstream service that wanted to verify a token by itself needed a copy of that secret - and handing the same secret to twenty services means one compromised container is enough to compromise all twenty. So most services skipped local verification and just called the auth service over the network on every request instead, which is the same chatty-microservices problem the permission-cache case study ran into, just showing up on the auth path this time.

We already had a Rust cryptographic service sitting nearby - built originally for signing blockchain-style transactions, not for web session tokens, and already using secp256k1 for that. The redesign was really about pointing that service at a second job instead of standing up a whole new one: move the master key behind a real hardware boundary, sign JWTs with the same curve the service already had audited and battle-tested, and publish the public half through a JWKS endpoint that can rotate without breaking anyone mid-flight. (What that original blockchain use case actually was stays out of this write-up - the curve choice and the reasoning behind reusing it are the part that's actually mine to talk about.)

Two options, and neither one was actually enough

Before this redesign there were really only two paths on the table, and both had a real cost attached.

The first is the shared-secret path above - HS256, one key, and the choice is either distribute it everywhere (blast radius becomes the whole platform) or keep it centralized and pay a network round-trip on every single authorization check. Neither side of that is free.

The second option is signing directly inside the HSM on every request. That does solve the key-exposure half of the problem - the private key genuinely never leaves the hardware boundary - but hardware crypto chips are built for tamper resistance, not throughput. Signing on the chip caps out at some fixed number of operations per second dictated by the silicon, not by how many logins your platform actually needs to handle a second. I won't quote a specific vendor number here - the real figure is under NDA, and it also isn't really the point. The point is that it's a per-call cost: every single signature pays the hardware round-trip again, and no amount of application-side tuning changes that, because the bottleneck isn't in your code.

So the real question wasn't "software or hardware." It was: how do you keep the key inside the hardware boundary without paying a hardware round-trip for every signature?

The fix: wrap the key once, sign on the host many times

The answer is envelope encryption, and it's not a new idea - it's the same shape as the outbox pattern in the permission-cache write-up, just applied to a key instead of a document. The HSM holds one long-lived Master Key Encryption Key (KEK) and never signs anything with it directly. Instead:

1. Generate a short-lived signing key pair (ES256K, secp256k1) inside the HSM boundary - it's never plaintext outside it during creation.
2. Ask the HSM to wrap that private key's private half with the Master KEK - one hardware call, done once per key's lifetime, not once per signature.
3. Hand back only the wrapped ciphertext blob and the (non-secret) public key. The HSM-side copy of the private key is destroyed once it's wrapped.
4. The first time a token needs signing under that key, unwrap the blob back into a host-side key object and cache it - every signature after that reuses the same host-side key rather than unwrapping again.

The HSM still does real work here - every key that's ever used gets provisioned and wrapped through it, and the Master KEK, marked non-extractable, never leaves it - but that cost is paid once per key, not once per login. Everything downstream of the unwrap runs at ordinary host CPU speed, and only stays that way because the unwrap itself doesn't repeat on the hot path (more on that below - getting this wrong the first time is exactly what happened).

A signature format mismatch nobody warns you about

Something this migration surfaced that wasn't obvious going in: the ECDSA signatures coming out of Rust/OpenSSL and the ones the JWT spec expects aren't the same shape on the wire. An ECDSA signature is mathematically just two numbers, R and S, each 32 bytes for a 256-bit curve. Both formats carry exactly those two numbers. They disagree entirely on how to write them down.

ASN.1 DER - what OpenSSL emits. Every value is wrapped in a tag and a length. Walking a real 71-byte signature byte by byte:

IEEE P1363 - what RFC 7515 wants. No tags, no lengths, no structure:

  R (32 bytes) ‖ S (32 bytes)
  = always exactly 64 bytes

The 0x00 above is the part that catches people. ASN.1 integers are signed, so when R's first byte has its high bit set, a zero byte gets prepended to stop it reading as negative - which makes R 33 bytes instead of 32. That happens for R or S independently, about half the time each, so the same key signing the same kind of payload produces a different-length signature from one call to the next. Measured over 20,000 real secp256k1 signatures:

That variability is the whole reason a fixed-width format exists, and it's also why this bug is so unpleasant to meet in production: it isn't deterministic. Feed DER to a verifier expecting P1363 and it just returns "invalid signature" - no hint that the problem is encoding rather than cryptography - and it does so for every signature, so you can burn an afternoon suspecting the wrong key before you think to look at byte lengths.

The fix is a plain bitwise conversion at the gateway: unwrap the structure, read the two integers, strip the pad byte if it's there, and left-pad each into a fixed 32 bytes.

export function derToRawSignature(derBuffer: Buffer): Buffer {
  let offset = 2; // Skip 0x30 [len]
  if (derBuffer[0] !== 0x30) throw new Error('Not an ASN.1 DER sequence');

  // Read R integer
  if (derBuffer[offset++] !== 0x02) throw new Error('Expected integer tag 0x02 for R');
  const rLength = derBuffer[offset++];
  let rBytes = derBuffer.subarray(offset, offset + rLength);
  offset += rLength;

  // Read S integer
  if (derBuffer[offset++] !== 0x02) throw new Error('Expected integer tag 0x02 for S');
  const sLength = derBuffer[offset++];
  let sBytes = derBuffer.subarray(offset, offset + sLength);

  // Strip leading zero if 33 bytes (ASN.1 signed padding)
  if (rBytes.length === 33 && rBytes[0] === 0x00) rBytes = rBytes.subarray(1);
  if (sBytes.length === 33 && sBytes[0] === 0x00) sBytes = sBytes.subarray(1);

  const rawR = Buffer.alloc(32, 0);
  const rawS = Buffer.alloc(32, 0);
  rBytes.copy(rawR, 32 - rBytes.length);
  sBytes.copy(rawS, 32 - sBytes.length);

  return Buffer.concat([rawR, rawS]); // Exactly 64 bytes
}

The compact format pays for itself twice over, once you count it: a P1363 ES256K signature is 64 bytes, an RSA-2048 signature is 256 bytes - 75% smaller, sitting in an Authorization: Bearer header on every single request. It's not the reason secp256k1 got picked - that reason was reuse, not byte-counting - but it's a real side benefit either way, since P-256 would have given the same 64 bytes.

Rotating keys without logging anyone out

A signing key can't live forever, and rotating it is a distributed-state problem, not a flag flip. If the auth service switches from kid_1 to kid_2 while a large chunk of users are holding tokens signed under kid_1, deleting kid_1 from the JWKS the moment rotation happens turns every one of those tokens into a 401 at once.

The fix is a four-phase lifecycle instead of an on/off switch:

I'll be honest about how I first tried to prove this worked, because the first attempt didn't actually prove anything. I signed a token under kid_1, verified it once, rotated to kid_2, then verified the same token again - on the same gateway instance, which already had kid_1's public key sitting warm in its cache from the first check. Of course it verified. It would have verified even if the JWKS logic had a bug that dropped retired keys entirely, because the cache never needed to ask. That's not a test, that's a tautology.

The fix was to verify the second time on a fresh gateway instance that had never seen the token before - forcing it to actually fetch the post-rotation JWKS and find kid_1 still there. And the other half needed testing separately too: revoke kid_1 for real, hit it with yet another fresh instance, and confirm it's rejected. One half doesn't mean anything without the other - "still valid during the grace period" and "actually gone after revocation" are two different claims, and only checking one of them proves neither.

Defending the JWKS endpoint against forged key IDs

If a gateway gets a JWT header with a kid it doesn't recognize, the naive move is to immediately call the security service to ask about it. Fine for one request. Not fine if someone floods the gateway with thousands of forged tokens carrying random, made-up kid values - now every one of those triggers its own round-trip, and the security service's connection pool is the thing that falls over.

The fix is a negative cache sitting next to the normal one: the first time a kid comes back unresolvable, remember that for sixty seconds and reject everything with that kid locally, no round-trip at all.

async verifyJwt(rawJwt: string): Promise<VerifyResult> {
  // ...parse header, extract kid...

  const blockedUntil = this.negativeKeyCache.get(kid);
  if (blockedUntil && Date.now() < blockedUntil) {
    return { valid: false, keyId: kid, source: 'NEGATIVE_CACHE_REJECT', latencyMs: /* ... */ };
  }

  let keyObj = this.publicKeyCache.get(kid);
  if (!keyObj) {
    await this.refreshJwks();
    keyObj = this.publicKeyCache.get(kid);
    if (!keyObj) {
      this.negativeKeyCache.set(kid, Date.now() + this.NEGATIVE_TTL_MS);
      return { valid: false, keyId: kid, source: 'COLD_JWKS_MISS', latencyMs: /* ... */ };
    }
  }
  // ...verify signature...
}

Worth flagging the mistake I made writing this the first time: that first unresolvable lookup - the one that actually pays for the real fetch - was tagged with the same NEGATIVE_CACHE_REJECT label as every fast-rejected request after it. Which meant the printed count of "cache rejections" was always one too high, and the fix at the time was subtracting 1 in the logging code instead of fixing the label. That's patching where the number gets printed instead of where it goes wrong. The real fix is a separate COLD_JWKS_MISS label for that first call, so the count that comes out the other end is just correct, not adjusted to look correct.

What the lab actually measured, and what it can't tell you

The lab linked at the end of this article talks to a real HSM now - SoftHSM2 over real PKCS#11, not a JavaScript class pretending to be one. The Master KEK is generated inside the token with CKA_EXTRACTABLE=FALSE actually enforced by SoftHSM, not asserted in a comment; wrapping and unwrapping are genuine C_WrapKey/C_UnwrapKey calls. What it still isn't is physical hardware - SoftHSM is a shared library loaded into the same process as everything else, so there's no bus, no network, no chip on the other end of those calls. The production version - Rust, a real chip - handles real production numbers, and those are under NDA.

Getting this far surfaced two real problems, and neither was the kind you'd plan for.

The first version that talked to real SoftHSM2 showed direct hardware signing beating envelope signing - roughly 0.3ms against 2.2ms, the opposite of everything this article has argued so far. Not a bug in the crypto - a wrong assumption about what SoftHSM actually is. It's software, loaded in-process, so calling it costs about what calling OpenSSL directly costs. It structurally cannot reproduce a physical HSM's actual bottleneck, because that bottleneck is specifically the round trip to a separate device, and there is no separate device here. The fix keeps everything else genuinely real - the sign operation, the PKCS#11 call, the key material - while adding back a fixed delay for the one piece no software-only PKCS#11 implementation can manufacture: a hardware round trip that doesn't exist in this lab.

The second problem was worse, because it was a real design mistake, not a measurement artifact. Once unwrapping went through an actual PKCS#11 call, signJwt() doing that on every single signature - which this lab, and this article before this rewrite, had always assumed was fine - turned into a genuine bottleneck: host-side throughput fell below what signing directly on the chip achieved. The whole architecture in this article rests on paying the HSM cost once per key's lifetime, not once per signature - and the code had never actually enforced that, it just hadn't been expensive enough to notice. The fix is a small cache: unwrap once per key, hold onto the resulting host-side key for every signature after that, the same data-key-caching pattern AWS KMS's own SDK uses for the identical reason.

With both of those fixed, on ordinary hardware with no tuning, across seven consecutive runs:

Roughly 10-12x, and that gap - not any vendor's datasheet number - is the actual argument for wrapping instead of signing on the chip every time.

The curve costs far more than I first reported

The host-side figure above deserves a second look, because almost all of it is one thing. Signing the same payload directly through Node's crypto, with no JWT assembly and no DER conversion around it, measured on the same machine:

P-256 is about 15x faster, and secp256k1 accounts for roughly 1.07ms of the envelope path's 1.15ms - the JWT assembly, base64 and DER-to-P1363 conversion together cost less than a tenth of what the curve does. OpenSSL ships a heavily optimised constant-time implementation for the NIST curves because that is what essentially all of TLS runs on; secp256k1 gets the generic path.

This correction matters more than a number usually would. An earlier version of this article reported the same comparison as "roughly 4x," which understated the real cost of reusing the blockchain-signing pipeline by nearly a factor of four - and understated it in the direction that made my own decision look better. Reusing that service was still the right call, because standing up a second audited signing path costs more than 1ms per token ever will at this volume. But that argument has to be made against the real number, and the real number is 15x.

What's still open

Two things worth naming rather than glossing over. Zeroing memory and locking it against swap protects against the ordinary failure modes - a core dump, a page written to disk - but not against a kernel-level compromise or someone with physical access to the machine while a key is unwrapped. If the threat model includes that, the next step is confidential computing (encrypted RAM at the hardware level), not another layer of software hygiene.

And elliptic-curve signatures, secp256k1 included, are exactly the kind of thing a sufficiently large quantum computer breaks via Shor's algorithm. Nothing in this design is quantum-resistant, and nothing here claims to be - it's a problem for a later migration, not one this architecture tries to solve today.

Lessons

Using an HSM to sign every request directly is trying to make a key locker do a web tier's job - it wasn't built for that throughput, and asking it to be would always lose. Wrapping a short-lived key with it instead is what actually lets the hardware boundary and internet-scale request volume coexist.

Never assume two libraries implementing the same signature algorithm agree on the wire format. DER and P1363 are both "just" ECDSA signatures, and they still don't interoperate without an explicit bridge.

And a rotation without a grace period isn't rotation, it's an outage with extra steps - the four-phase state machine exists because the alternative is an instant, avoidable wave of 401s.

Reusing an existing key pipeline isn't free just because it saves you from building a second one - secp256k1 signs about 15x slower than P-256 in software, and that only became visible once the reproduction actually used the same curve as production instead of a more convenient one. The first time I measured it I got the direction right and the magnitude wrong by a factor of four, in my own favour, which is its own lesson about who benchmarks tend to flatter.

A software HSM is real cryptography with no real round trip, and treating those as the same thing inverts a benchmark. Making the key-wrapping boundary in this lab genuinely real (SoftHSM2, real PKCS#11) also, briefly, made direct hardware signing look faster than envelope signing - because software running in-process has nothing resembling a hardware round trip to pay. Being more honest about the crypto and being honest about the physical bottleneck turned out to be two different problems that needed two different fixes.

And making one thing more real can quietly break an assumption somewhere else that was never actually tested. This whole architecture rests on unwrapping a key once and signing with it many times - but nothing in the code enforced that until unwrapping became a real, non-free operation and the cost of calling it on every signature stopped being invisible.

Runnable reproduction

A complete, runnable lab demonstrating hardware key wrapping, the DER-to-P1363 bridge, zero-downtime JWKS rotation, and negative caching against forged key IDs is available here:

• labs/cryptographic-trust-engine