Confidential Rest-Api w/ Permissions - Always 403s - What Am I Doing Wrong? - node.js

Question

I've tried for many hours now and seem to have hit a wall. Any advice/help would be appreciated.
Goal: I want to authorize the express rest-api (ex client-id: "my-rest-api") routes (example resource: "WeatherForecast") across various HTTP methods mapped to client scopes (examples: "create"/"read"/"update"/"delete"). I want to control those permissions through policies (For example - "Read - WeatherForecast - Permission" will be granted if policy "Admin Group Only" (user belongs to admin group) is satisfied.
Rest-api will not log users in (will be done from front end talking directly to keycloak and then they will use that token to talk with rest-api).
Environment:
Keycloak 15.1.1 running in its own container, port 8080, on docker locally (w/ shared network with rest-api)
"my-rest-api": Nodejs 16.14.x w/ express 4.17.x server running on its own container on docker locally. Using keycloak-connect 15.1.1 and express-session 1.17.2.
Currently hitting "my-rest-api" through postman following this guide: https://keepgrowing.in/tools/kecloak-in-docker-7-how-to-authorize-requests-via-postman/
What Happens: I can login from keycloak login page through postman and get an access token. However when I hit any endpoint that uses keycloak.protect() or keycloak.enforce() (with or without specifying resource permissions) I can't get through. In the following code the delete endpoint returns back 200 + the HTML of the keycloak login page in postman and the Get returns back 403 + "Access Denied".
Current State of Realm
Test User (who I login with in Postman) has group "Admin".
Client "my-rest-api" with access-type: Confidential with Authorization enabled.
Authorization set up:
Policy Enforcement Mode: Enforcing, Decision Strategy: Unanimous
"WeatherForecast" resource with uri "/api/WeatherForecast" and create/read/update/delete client scopes applied.
"Only Admins Policy" for anyone in group admin. Logic positive.
Permission for each of the client scopes for "WeatherForecast" resource with "Only Admins Policy" selected, Decision Strategy: "Affirmative".
Current State of Nodejs Code:
import express from 'express';
import bodyParser from 'body-parser';
import session from "express-session";
import KeycloakConnect from 'keycloak-connect';
const app = express();
app.use(bodyParser.json());
const memoryStore = new session.MemoryStore();
app.use(session({
secret: 'some secret',
resave: false,
saveUninitialized: true,
store: memoryStore
}));
const kcConfig: any = {
clientId: 'my-rest-api',
bearerOnly: true,
serverUrl: 'http://localhost:8080/auth',
realm: 'my-realm',
};
const keycloak = new KeycloakConnect({ store: memoryStore }, kcConfig);
app.use(keycloak.middleware({
logout: '/logout',
admin: '/',
}));
app.get('/api/WeatherForecast', keycloak.enforcer(['WeatherForecast:read'],{ resource_server_id: "my-rest-api"}), function (req, res) {
res.json("GET worked")
});
app.delete('/api/WeatherForecast', keycloak.protect(), function (req, res) {
res.json("DELETE worked")
});
app.listen(8081, () => {
console.log(`server running on port 8081`);
});
A Few Other Things Tried:
I tried calling RPT endpoint with curl using token gotten from postman and got the RPT token perfectly fine, saw permissions as expected.
I tried calling keycloak.checkPermissions({permissions: [{id: "WeatherForecast", scopes: ["read"]}]}, req).then(grant => res.json(grant.access_token)); from inside an unsecured endpoint and got "Connection refused 127.0.0.1:8080".
I tried just disabling Policy Enforcement Mode just to see, still got Access Denied/403.
I tried using keycloak.json config instead of object method above - same exact results either way.
I tried openid-client (from another tutorial) and also got connected refused issues.
I've tried using docker host ip, host.docker.internal, the container name, etc. to no avail (even though I don't think it is an issue as I obviously can hit the auth service and get the first access token).
I really want to use Keycloak and I feel like my team is so close to being able to do so but need some assistance getting past this part. Thank you!
------------------- END ORIGINAL QUESTION ------------------------
EDIT/UPDATE #1:
Alright so a couple more hours sank into this. Decided to read through every line of keycloak-connect library that it hits and debug as it goes. Found it fails inside keycloak-connect/middleware/auth-utils/grant-manager.js on the last line of checkPermissions. No error is displayed or catch block to debug on - chasing the rabbit hole down further I was able to find it occurs in the fetch method that uses http with options:
'{"protocol":"http:","slashes":true,"auth":null,"host":"localhost:8080","port":"8080","hostname":"localhost","hash":null,"search":null,"query":null,"pathname":"/auth/realms/my-realm/protocol/openid-connect/token","path":"/auth/realms/my-realm/protocol/openid-connect/token","href":"http://localhost:8080/auth/realms/my-realm/protocol/openid-connect/token","headers":{"Content-Type":"application/x-www-form-urlencoded","X-Client":"keycloak-nodejs-connect","Authorization":"Basic YW(etc...)Z2dP","Content-Length":1498},"method":"POST"}'
It does not appear to get into the callback of that fetch/http wrapper. I added NODE_DEBUG=http to my start up command and was able to find that swallowed error, which appears I am back to the starting line:
HTTP 31: SOCKET ERROR: connect ECONNREFUSED 127.0.0.1:8080 Error: connect ECONNREFUSED 127.0.0.1:8080
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1157:16)
at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17)
I then saw something that I thought may be related due to my docker network set up (Keycloak and Spring Boot web app in dockerized environment) and tried change host name dns so I could use something other then local host but it didn't work either (even added to redirect uri, etc.).
UPDATE #2:
Alright so I got the keycloak.protect() (pure authentication) endpoint working now. I found through reading through the keycloak-connect lib code more options and it seems that adding "realmPublicKey" to the keycloak config object when instantiating keycloak-connect fixed that one. Still no luck yet on the authorization keycloak.enforce side.
const kcConfig: any = {
clientId: 'my-rest-api',
bearerOnly: true,
serverUrl: 'http://localhost:8080/auth',
realm: 'my-realm',
realmPublicKey : "MIIBIjANBgk (...etc) uQIDAQAB",
};

Answer 1

So my team finally figured it out - the resolution was a two part process:
Followed the instructions on similar issue stackoverflow question answers such as : https://stackoverflow.com/a/51878212/5117487
Rough steps incase that link is ever broken somehow:
Add hosts entry for 127.0.0.1 keycloak (if 'keycloak' is the name of your docker container for keycloak, I changed my docker-compose to specify container name to make it a little more fool-proof)
Change keycloak-connect config authServerUrl setting to be: 'http://keycloak:8080/auth/' instead of 'http://localhost:8080/auth/'
Postman OAuth 2.0 token request Auth URL and Access Token URL changed to use the now updated hosts entry:
"http://localhost:8080/auth/realms/abra/protocol/openid-connect/auth" -> "http://keycloak:8080/auth/realms/abra/protocol/openid-connect/auth"
"http://localhost:8080/auth/realms/abra/protocol/openid-connect/token" ->
"http://keycloak:8080/auth/realms/abra/protocol/openid-connect/token"