Azure B2C: "The provided id_token failed signature validation" - azure-ad-b2c

Question

I am creating an identity provider, and have plumbed it into a B2C custom policy.
Discovery happens fine, and the user gets sent to my idp fine.
However, when I return the user back to B2C, using the id_token response type (the only response type I am supporting at the moment) B2C gives me the error in the title. The full error is:
error_description=AADB2C90239: The provided id_token failed signature validation. Please provide another token and try again.
Everything else I am validating my token against (jwt.io, various node jwt/jwk verification methods) seems to be happy with what I am returning, only B2C isn't.
Here is an example of my id_token response
I am just running it on my local machine, hence the issuer being ngrok, but my openid-configuration looks like this:
{
"issuer": "https://28b5fe46.ngrok.io",
"authorization_endpoint": "http://localhost:3000/authentication/auth",
"jwks_uri": "https://28b5fe46.ngrok.io/.well-known/openid-configuration/keys",
"response_modes_supported": [
"query"
],
"response_types_supported": [
"id_token"
],
"scopes_supported": [
"openid"
],
"subject_types_supported": [
"public"
],
"id_token_signing_alg_values_supported": [
"RS256"
],
"claims_supported": [
"sub"
]
}
and my keys look like this:
{
"keys": [
{
"kid": "kiddy123",
"alg": "RS256",
"kty": "RSA",
"use": "sig",
"n": "ALdnY_dWrQgjaqWuqUFpr-__p62xsWSWGiVH9CEWlxOTGrR8jhfG_C_xxGkkptWrVWwJgpmJ7zOMFXjqc9HqGCitl9Czl-X68Ld2xnZ_HdmikRTv-Witn8V-5QiQhSEGwvA2Xek8OVWKWOZ7Z4L9_SjNar33E8zGGFtz77_pmzJ6_zompGQSkLwAMTU3buI8TUKFIOBLGtc_eFCfgcbHFjJuYYp_200A4Xz2a0lNvvHSZ8PLKmwoRIBbFhkhe0zFd0mwq97SavTUXrltZnq3ghTbc6QzJC4T3_4J1LMTEQIFpaK3jK9VPFrLVESy8Ovz4CdxsfY3_bv2QX2HfPOxJziDIOsoztXUvfPPH-tu5nZ2JpKuE5ftAL3W7LbR5SCe2fHqV8aSsJbyDP2fHKbjy1pBhx_MQ2ty17YJRcqHER0l2GEhRf7AhdewIdv2_LdMIUr1tYuYWLuiZVN682fgYZZGl_TAxhBSyi65uzXbziG4STpENkX8KvBPjkMJc-EfwRcegJzS2kJVd-fnE5fCF2lHuo0hC93piViUhtzo6_1R5AXKk2JKxs_kWRd30E7DE8LZPRw-2hM3zrEQ6X5VL7q-UvHLR6SUKdjHXPYUX2FJAuj8EhQlqhovf26_pwO26wHhBl8mkJo9T8c8MQSVz3y12AJbP99-lo0We5umk4uP",
"e": "AQAB"
}
]
}
The B2C journey recorder doesn't give me any extra information, only the aforementioned error.
Does anyone know how I might debug this further?
Here is how I am creating the id_token. You can see near the bottom I am validating the token against the pem that was used to create the jwks.
const { session } = common.getServices(request)
const sessionValues = session.get(request)
const pem = fs.readFileSync(path.join(__dirname, '..', '..', '..', '..', '..', 'jwtRS256.key'))
const {
nonce,
redirectUri,
state,
databucketItems: { sub },
clientId
} = sessionValues
const jwk = njwk.JWK.fromJSON(JSON.stringify({
'kid': 'kiddy123',
'alg': 'RS256',
'kty': 'RSA',
'use': 'sig',
'n': 'ALdnY_dWrQgjaqWuqUFpr-__p62xsWSWGiVH9CEWlxOTGrR8jhfG_C_xxGkkptWrVWwJgpmJ7zOMFXjqc9HqGCitl9Czl-X68Ld2xnZ_HdmikRTv-Witn8V-5QiQhSEGwvA2Xek8OVWKWOZ7Z4L9_SjNar33E8zGGFtz77_pmzJ6_zompGQSkLwAMTU3buI8TUKFIOBLGtc_eFCfgcbHFjJuYYp_200A4Xz2a0lNvvHSZ8PLKmwoRIBbFhkhe0zFd0mwq97SavTUXrltZnq3ghTbc6QzJC4T3_4J1LMTEQIFpaK3jK9VPFrLVESy8Ovz4CdxsfY3_bv2QX2HfPOxJziDIOsoztXUvfPPH-tu5nZ2JpKuE5ftAL3W7LbR5SCe2fHqV8aSsJbyDP2fHKbjy1pBhx_MQ2ty17YJRcqHER0l2GEhRf7AhdewIdv2_LdMIUr1tYuYWLuiZVN682fgYZZGl_TAxhBSyi65uzXbziG4STpENkX8KvBPjkMJc-EfwRcegJzS2kJVd-fnE5fCF2lHuo0hC93piViUhtzo6_1R5AXKk2JKxs_kWRd30E7DE8LZPRw-2hM3zrEQ6X5VL7q-UvHLR6SUKdjHXPYUX2FJAuj8EhQlqhovf26_pwO26wHhBl8mkJo9T8c8MQSVz3y12AJbP99-lo0We5umk4uP',
'e': 'AQAB'
}))
const time = new Date().getTime()
const jwt = njwt.create({
iss: 'https://28b5fe46.ngrok.io',
name: 'Cheese man',
aud: clientId,
nonce,
redirectUri,
state,
sub
}, pem, jwk.alg)
jwt.setHeader('kid', 'kiddy123')
jwt.setExpiration(time + (12 * 60 * 60 * 1000))
jwt.setNotBefore(time - (2 * 60 * 60 * 1000))
const compacted = jwt.compact()
const returnUrl = `${sessionValues.redirectUri}?${qs.stringify({
id_token: compacted,
state
})}`
/** Just for verifying the id_token**/
const verifier = njwt.createVerifier().withKeyResolver((kid, next) => {
return pem.toString()
})
const parsedJwt = verifier.verify(compacted)
console.log(parsedJwt) // This prints the token correctly - meaning it is valid
return h.redirect(returnUrl)