We've set up our deployments to run using Workload Identity. Each deployment has its own technical identity, a GCP service account set up with the required IAM roles, and a Kubernetes service account to match (both linked as described in the Workload Identity documentation).
This all works just fine, with one small problem: almost every time a pod starts it gives this error:
Error: Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information. at GoogleAuth.getApplicationDefaultAsync (/home/node/node_modules/google-auth-library/build/src/auth/googleauth.js:155:19) at processTicksAndRejections (internal/process/task_queues.js:97:5) at async GoogleAuth.getClient (/home/node/node_modules/google-auth-library/build/src/auth/googleauth.js:486:17) at async GrpcClient._getCredentials (/home/node/node_modules/google-gax/build/src/grpc.js:88:24) at async GrpcClient.createStub (/home/node/node_modules/google-gax/build/src/grpc.js:213:23)
It then crash-loops and typically comes up later, after 2 or 3 retries. Very very odd...
To make matters worse, every now and then we get this error instead:
Error: 403 undefined: Getting metadata from plugin failed with error: Could not refresh access token: A Forbidden error was returned while attempting to retrieve an access token for the Compute Engine built-in service account. This may be because the Compute Engine instance does not have the correct permission scopes specified: Could not refresh access token: Unsuccessful response status code. Request failed with status code 403 at Object.callErrorFromStatus (/home/node/node_modules/#grpc/grpc-js/build/src/call.js:31:26) at Object.onReceiveStatus (/home/node/node_modules/#grpc/grpc-js/build/src/client.js:176:52) at Object.onReceiveStatus (/home/node/node_modules/#grpc/grpc-js/build/src/client-interceptors.js:342:141) at Object.onReceiveStatus (/home/node/node_modules/#grpc/grpc-js/build/src/client-interceptors.js:305:181) at /home/node/node_modules/#grpc/grpc-js/build/src/call-stream.js:124:78 at processTicksAndRejections (internal/process/task_queues.js:79:11)
And when we see this, it's basically game over until we try all kinds of black voodoo to try and get the service back up (usually it boils down to us deleting the entire auth stuff for that service and recreating things from scratch).
Anyone has any idea what could be going on here?
Our cluster is running 1.16.13-gke.1, and has Istio 1.6.7 installed (using the Istio operator).
So... the mystery has been solved.
There was an obscure but stupid but in our operator that provisions both the IAM and Kubernetes service accounts, including the necessary bindings.
There was a situation with a namespace being deleted that could end up in removing all the bindings to any other namespace :-( So this explains the error above, the moment this happened we got 403 errors (obviously, looking back on it now).
Bug has been fixed today, so all should be well now.
Related
I am working with implementing an azure event hub and already created the required resources in function app and Event Hub NS, Event hub and consumer group. However when I tried to run the function app locally I get the following error and it does not allow me to send messages to the Event Hub. I already configured the Listen,Send,Manage policies at the NS level. Does anyone encountered this issue?
The listener for function 'IngestEvents' was unable to start. System.Private.CoreLib: One or more errors occurred. (InvalidIssuer: Token issuer is invalid. TrackingId:00000xxxxx0000xxxxx, SystemTracker:NoSystemTracker, Timestamp:2022-08-01T07:29:33 (events-00)) (InvalidIssuer: Token issuer is invalid. TrackingId:00000xxxxx0000xxxxx, SystemTracker:NoSystemTracker, Timestamp:2022-08-01T07:29:33 (events-00)). Azure.Messaging.EventHubs: InvalidIssuer: Token issuer is invalid. TrackingId:00000xxxxx0000xxxxx, SystemTracker:NoSystemTracker, Timestamp:2022-08-01T07:29:33 (events-00).
I already tried out solutions in Azure Functions: There was an error performing a read operation on the Blob Storage Secret Repository and The listener for function was unable to start. Why? but no success result.
I am using chain credentials. So mostly it authorized me through VS credentials as default.
I deployed an nft collection on solana using metaplex and candymachine. It was working fine for 4-5 days now suddenly I am receiving this error. I was able to connect wallet and mint fine for a couple of days. I am not sure what's wrong now what I am getting 403 from rpc pool?
Uncaught (in promise) Error: failed to get balance of account
: Error: 403 Forbidden: {"jsonrpc":"2.0","error":{"code":
403, "message":"Access forbidden, contact your app developer or
support#rpcpool.com."}, "id": "" }
So Metaplex and NFT calls are not allowed on the free RPCs. You'll have to use a paid provider or go with https://api.metaplex.solana.com/
If you're doing any sort of NFT drop, it is still recommended to get a paid RPC.
You shouldn't have to re-upload again, as long as you did successfully on the other RPC.
I was trying webhook to send email in chatbot (dialogflow). Getting following error (log entry) .
What may be the reson?
message: "Permission 'cloudfunctions.functions.setIamPolicy' denied on resource 'projects/rare-shadow-276706/locations/us-central1/functions/dialogflowFirebaseFulfillment' (or resource may not exist)."
The issue
This is an issue with the proper permissions, and the service account permissions. What's happening is the dialogflowFirebaseFunction needs to have the setIamPolict role.
The fix
You can achieve this when you (the user) have the roles/iam.securityAdmin role. Check this out for more info. Other roles may also work, but this role will suffice. I would suggest also deleting the existing cloud function, get the new role, and then create the cloud function again.
Solved
After setting Project Owner role, no error.
Thanks
I am not able to access "https://testdev01.azurewebsites.net"
Its showing "The service is unavailable"
_errorData: undefined _sourceErrorLevel: undefined baseTypes: ["t","MsPortalFx.Errors.Error"] code: undefined data: 1 errorLevel: 2
extension: fx handled: undefined innerErrors: ["message: Failed to
retrieve the blade definition for 'ActivityLogBlade' from the
server.\r\nmessage:Manual require of the following modules failed;
["_generated/Blades/ActivityLogBlade" : {Error: Couldn't load
"_generated/Blades/ActivityLogBlade" at
"https://afd.hosting.portal.azure.net/websites/Content/5.12.34.475/Scripts/_generated/Blades/ActivityLogBlade.js?retryAttempt=1.0371554125639133";
error code 404, message: Not Found}];\r\nstack:
RPC_Exception ##\nError: Failed to retrieve the blade definition for 'ActivityLogBlade' from the server.\r\nmessage:Manual require of
the following modules failed; ["_generated/Blades/ActivityLogBlade" :
{Error: Couldn't load "_generated/Blades/ActivityLogBlade" at
"https://afd.hosting.portal.azure.net/websites/Content/5.12.34.475/Scripts/_generated/Blades/ActivityLogBlade.js?retryAttempt=1.0371554125639133";
error code 404, message: Not Found}];\n at new r
(https://portal.azure.com/Content/Dynamic/qLnxN0oZGQC0.js:18:790)\n at
u (https://portal.azure.com/Content/Dynamic/qLnxN0oZGQC0.js:11:1296)\n
at Object.tt [as QReject]
(https://portal.azure.com/Content/Dynamic/qLnxN0oZGQC0.js:11:1405)\n
at
https://portal.azure.com/Content/Dynamic/qDdhDKs5uKzm.js:4:5443\r\nFrom
RPC: fx -> WebsitesExtension
(MsPortalFx.Internal.Constants.RpcMethods.entryPointGetBladeDefinition)\r\n(Callstack
capturing is not enabled. Use ?trace=diagnostics to enable it.)\r\n"]
message: Unable to locate blade 'ActivityLogBlade'. Search
path:'[0]WebsitesExtension-[1]ActivityLogBlade'.
The error (503)- 'The service is unavailable' may be caused by the application level issues, requests taking a long time; application using high memory/CPU or application crashing due to an exception. Firstly, review the application from this front to isolate the problem.
To fetch more details on the error, you could access Kudu console to check the log. You could access http://yourwebappname.scm.azurewebsites.net and review the debug logs.
To access the Application Event Log, use the Diagnose and solve problems blade in the Azure portal.
1. In the Azure portal, open the app in App Services.
2. Select Diagnose and solve problems.
Also, to fetch more details on the error, you can always enable logging. Please check out the documentation 'Enable diagnostics logging for web apps in Azure App Service' for a detailed steps on enabling & fetching the logs.
You may also scale-up your App Service Plan and then check to see if that makes any difference.
If feasible, you may restart the WebApp and then check to see if that helps.
I am trying to use node.js to access the azure resource manager with the following example code:
msRestAzure.interactiveLogin(function(err, credentials) {
if (err) console.log(err);
var client = new resourceManagement.ResourceManagementClient(credentials, 'token');
client.resources.list(function(err, result) {
if (err) console.log(err);
console.log(result);
});
});
I get the following error when I run it:
{ Error: The access token is from the wrong issuer 'https://sts.windows.net/token/'. It must match the tenant 'https://sts.windows.net/token/' associated with this subscription. Please use the authority (URL) 'https://login.windows.net/token' to get the token. Note, if the subscription is transferred to another tenant there is no impact to the services, but information about new tenant could take time to propagate (up to an hour). If you just transferred your subscription and see this error message, please try back later.
at client.pipeline.error (D:\azure-arm\node_modules\azure-arm-resource\lib\resource\operations\resources.js:496:19)
at retryCallback (D:\azure-arm\node_modules\ms-rest\lib\filters\systemErrorRetryPolicyFilter.js:89:9)
at retryCallback (D:\azure-arm\node_modules\ms-rest\lib\filters\exponentialRetryPolicyFilter.js:140:9)
at D:\azure-arm\node_modules\ms-rest\lib\filters\rpRegistrationFilter.js:59:14
at handleRedirect (D:\azure-arm\node_modules\ms-rest\lib\filters\redirectFilter.js:39:9)
at D:\azure-arm\node_modules\ms-rest\lib\filters\formDataFilter.js:23:14
at Request.defaultRequest [as _callback] (D:\azure-arm\node_modules\ms-rest\lib\requestPipeline.js:125:16)
at Request.self.callback (D:\azure-arm\node_modules\request\request.js:185:22)
at emitTwo (events.js:106:13)
at Request.emit (events.js:191:7)
Any place you see token, I've replaced the actual value that was there. The subscription ID I'm providing is correct. I have tried logging in directly rather than using the key above and it had the same effect. Ideally we'd want an application in azure to have access to the arm API but it doesn't look like that's possible in active directory, and I can't seem to get this to work. Any help would be appreciated.
I have used ms-rest-azure versioned 2.6.0 and azure-arm-resource versioned 7.3.0 and it was working fine for me with the same code.
I think its secure to authenticate using service principal authentication way or basic authentication way rather than by interactive login authentication way. We have the steps of these other ways of authentication in here -> https://github.com/Azure/azure-sdk-for-node/blob/master/Documentation/Authentication.md#using-authentication-in-your-nodejs-script which you may try if interested.
Also, as you are ideally looking for application in Azure to have access to the ARM API so I think its worthwhile to give a read of this article -> https://learn.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-api-authentication
Hope this helps!!