AWS node.js SDK error - SignatureDoesNotMatch: Signature expired - node.js

Node.js version 0.10.25
AWS SDK Version latest - 2.0.23
I have an app that is continuously listening to a Queue (SQS) and if there are messages posted in that queue the app will read the message and process it and save some data to S3. When I start the app after about 20 minutes I am getting the following error continuously.
Potentially unhandled rejection [160] SignatureDoesNotMatch: Signature expired: 20141104T062952Z is now earlier than 20141104T062952Z (20141104T064452Z - 15 min.)
at Request.extractError (/myproject/node_modules/aws-sdk/lib/protocol/query.js:39:29)
at Request.callListeners (/myproject/node_modules/aws-sdk/lib/sequential_executor.js:100:18)
at Request.emit (/myproject/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/myproject/node_modules/aws-sdk/lib/request.js:604:14)
at Request.transition (/myproject/node_modules/aws-sdk/lib/request.js:21:12)
at AcceptorStateMachine.runTo (/myproject/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /myproject/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/myproject/node_modules/aws-sdk/lib/request.js:22:9)
at Request.<anonymous> (/myproject/node_modules/aws-sdk/lib/request.js:606:12)
at Request.callListeners (/myproject/node_modules/aws-sdk/lib/sequential_executor.js:104:18)
It is not an issue with my system time. My system time is in sync with the time of my EC2 instance. Why am I getting this error? Is it related to SQS or S3?

I know this is an old question, but I experienced it myself today.
Fortunately, and the AWS NodeJS SDK now has this configuration option called correctClockSkew, which will fix the system clock offset once an error occurs:
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Config.html#correctClockSkew-property

It's likely the time on your computer is incorrect
The signature mismatch is because authentication process depends on clock syncronisation.
Check the time on your machine.
On Linux / WSL you can run sudo hwclock -s to fix. Other OSs will require different commands.

Thanks to Loren Segal (Amazon) for his quick response. Refer https://github.com/aws/aws-sdk-js/issues/401 for more details. In short, the SDK is not retrying signature errors for which the work around is
AWS.events.on('retry', function(resp) {
if (resp.error.code === 'SignatureDoesNotMatch') {
resp.error.retryable = true;
}
});
This is not a regression i.e, it is not a bug introduced in 2.0.23

Related

Timeout when communicating with Web3 Provider

Background
I am trying to investigate and solve problems with a containerized NodeJS web service app that communicates with Avalanche Test Network (https://api.avax-test.network/ext/bc/C/rpc) in order to Mint NFTs.
//...
provider = new HDWalletProvider({
privateKeys: [walletPrivateKey],
providerOrUrl: "https://api.avax-test.network/ext/bc/C/rpc",
pollingInterval: 4000,
});
const myContract = Contract(Contract_NFT);
myContract.setProvider(provider);
const _instance = await myContract.at(process.env.CONTRACT_NFT_ADDRESS);
_instance.mint(provider.getAddress(), mintURL, { from: provider.getAddress() })
//...
Note: I am not the developer of app.
Issue
After starting the container, first request completes successfully. However subsequent requests fail with code: -32603 message: 'ETIMEDOUT' error
at new NodeError (node:internal/errors:387:5)
at Web3ProviderEngine.emit (node:events:502:17)
at /app/node_modules/web3-provider-engine/index.js:54:14
at afterRequest (/app/node_modules/web3-provider-engine/index.js:148:21)
at /app/node_modules/web3-provider-engine/index.js:174:21
at /app/node_modules/web3-provider-engine/index.js:232:9
at /app/node_modules/async/internal/once.js:12:16
at replenish (/app/node_modules/async/internal/eachOfLimit.js:61:25)
at /app/node_modules/async/internal/eachOfLimit.js:71:9
at eachLimit (/app/node_modules/async/eachLimit.js:43:36)
at /app/node_modules/async/internal/doLimit.js:9:16
at end (/app/node_modules/web3-provider-engine/index.js:211:5)
at Request._callback (/app/node_modules/web3-provider-engine/subproviders/rpc.js:36:21)
at self.callback (/app/node_modules/request/request.js:185:22)
at Request.emit (node:events:513:28)
at Timeout.<anonymous> (/app/node_modules/request/request.js:851:16)
Error: PollingBlockTracker - encountered an error while attempting to update latest block:
undefined
at PollingBlockTracker._performSync (/app/node_modules/eth-block-tracker/src/polling.js:51:24)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
When the app is used outside a container and directly on server OS, all requests successfully complete.
What I have tried
I tried searching the internet forums and boards. Similar issue is mentioned in some posts but suggested solution is to tweak Timeout/Interval values which did not help with my issue and still got the error.
I tried using the same NodeJS and NPM versions installed on server OS. Also tried using different versions of NodeJS( 14.XX, 16.XX). Still got the same error.
Also I have tried changing and configuring Docker network( different type of network driver, different IPAM config, passing extra hosts etc.). Still got the same error.
I have confirmed that mentioned URL is accessible from inside container using ping, nc and dig tools. Also have confirmed that successful HTTPS connection is being established.
Questions
What is causing this behavior?
What I can do to reliably use this app inside container?
Have I missed something?

How do I set my AWS Lambda provisioned capacity to zero in off hours?

I'm using the Serverless framework (serverless.com) and it looks like I can save some pretty serious money by only scaling up the provisioned capacity of my Lambda functions during business hours (9am-5pm, 40 hour per week) and then down to zero during off hours. My application isn't used very much in off hours, and if it is, my users are okay with cold starts and it taking longer.
In my serverless.yml, I have functions declared such as:
home_page:
handler: homePage/home_page_handler.get
events:
- http:
path: homePage
method: get
cors: true
authorizer: authorizer_handler
provisionedConcurrency: 1
I also have a Lambda that runs on a regular basis to set the provisioned capacity of the other Lambdas in the account depending on the time of day. In that Lambda if I call:
await lambda.putProvisionedConcurrencyConfig({
FunctionName: myFunctionName,
ProvisionedConcurrentExecutions: 0,
Qualifier: "provisioned",
}).promise();
Then I get this error:
Received error: ValidationException: 1 validation error detected: Value '0' at 'provisionedConcurrentExecutions' failed to satisfy constraint: Member must have value greater than or equal to 1
at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)
at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:55:8)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:688:14)
at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:690:12) {
code: 'ValidationException',
time: 2021-08-14T17:45:48.932Z,
requestId: '8594fad6-d5dd-4a00-adca-c34f9d38b25e',
statusCode: 400,
retryable: false,
retryDelay: 77.81562781029108
}
On the other hand, if I try to delete the provisioned capacity entirely, like this:
await lambda.deleteProvisionedConcurrencyConfig({
FunctionName: myFunctionName,
Qualifier: "provisioned",
}).promise();
That works fine, but when I try to deploy my function again in off hours (which is the norm), CloudFormation errors out with:
No Provisioned Concurrency Config found for this function (Service: AWSLambdaInternal; Status Code: 404; Error Code: ProvisionedConcurrencyConfigNotFoundException; Request ID: 75dd221b-35d2-4a49-80c5-f07ce261d357; Proxy: null)
Does anybody have a neat solution to turn off provisioned capacity during some hours of the day?
After some thinking, if you're still committed to Lambda as the compute solution, I think your best option is to manage provisioned concurrency outside of the Serverless Framework entirely.
You've already got an orchestrator function which will enable provisioned concurrency, you could try removing provisionedConcurrency from your serverless.yml file, adding another method in your orchestrator to disable provisioned currency in the evenings, and verifying that you can deploy when your orchestrator has set your functions to either state.
If you're willing to throw away your orchestrator function, AWS suggests using Application Auto Scaling, which is very useful for exactly what you're doing. (hat tip to #mpv)
That being said, Lambda isn't particularly well suited to predictable, steady-state traffic. If cost is a concern, I'd suggest exploring Fargate or ECS and writing a few autoscaling rules. Your Lambda code is already stateless, and likely is portable and has pretty limited networking rules. There are other forms of compute which would be dramatically cheaper to use.

how to debug error from stripe API call? need a node.js stacktrace across event emitter?

I have a node webapp that makes various API calls to Stripe using the stripe npm package. Sometimes I get errors like the one below. Notice that the stacktrace is truncated so that I cannot see which stripe API call causes the error and I also cannot see where in my app this API call is made.
Is there anything I can do to get better error stacktraces?
Error: Missing required param: customer.
at Function.generate (/home/molsson/dv/foobar/node_modules/stripe/lib/Error.js:39:16)
at IncomingMessage.<anonymous> (/home/molsson/dv/foobar/node_modules/stripe/lib/StripeResource.js:175:33)
at Object.onceWrapper (events.js:299:28)
at IncomingMessage.emit (events.js:215:7)
at IncomingMessage.EventEmitter.emit (domain.js:476:20)
at endReadableNT (_stream_readable.js:1183:12)
at processTicksAndRejections (internal/process/task_queues.js:80:21)
Note: the error itself is just an example. I have already fixed it. I just want to get better stacktraces or a better method of debugging these types of errors quickly.
I'm pretty sure my node version has async stacks by default:
$ node --version
v12.12.0
$ node -p process.versions.v8
7.7.299.13-node.12
I tried running with NODE_OPTIONS='--trace-warnings --stack-trace-limit=9999' but it didn't help.
Does it exists some kind of "async stacktraces across event emitters" debugging thing?
I found a good answer to this question myself. The stripe library fires and event before making a new API request, so you can print a stacktrace from there:
stripe.on('request', request => {
const currentStack = (new Error()).stack.replace(/^Error/, '')
console.log(`Making Stripe HTTP request to ${request.path}, callsite: ${currentStack}`)
})
I added a STRIPE_API_TRACING option to my app that I can turn on if I experience errors without stacks. With the tracing on I can just scroll a bit up in the log and see what API calls was dispatched just before the error happened.

Node.js on Google Cloud Platform Pub/Sub tutorial worker is failing with "TypeError: Cannot call method 'on' of null"

I'm getting an error while working through https://cloud.google.com/nodejs/getting-started/using-pub-sub. (I've successfully completed previous tutorials in the series.)
With the command "SCRIPT=worker.js PORT=8081 npm start", I get this error related to background.js:
TypeError: Cannot call method 'on' of null
at /Users/xke/Documents/node.js/6-pubsub/lib/background.js:57:20
at /Users/xke/Documents/node.js/6-pubsub/node_modules/gcloud/lib/pubsub/index.js:256:7
at /Users/xke/Documents/node.js/6-pubsub/node_modules/gcloud/lib/common/util.js:684:11
at Object.handleResp (/Users/xke/Documents/node.js/6-pubsub/node_modules/gcloud/lib/common/util.js:192:5)
at Request._callback (/Users/xke/Documents/node.js/6-pubsub/node_modules/gcloud/lib/common/util.js:679:12)
at Request.self.callback (/Users/xke/Documents/node.js/6-pubsub/node_modules/request/request.js:354:22)
at Request.EventEmitter.emit (events.js:98:17)
at Request.<anonymous> (/Users/xke/Documents/node.js/6-pubsub/node_modules/request/request.js:1207:14)
at Request.EventEmitter.emit (events.js:117:20)
at IncomingMessage.<anonymous> (/Users/xke/Documents/node.js/6-pubsub/node_modules/request/request.js:1153:12)
Thoughts / solutions?
Thanks!
It appears this is a bug in the gcloud-node library. The library is supposed to automatically create the topic for you, but it appears that there is an edge case that the topic isn't automatically created when you try to subscribe to it.
In the meantime, there is a simple way around this: run the bookshelf app and create / edit a book. It'll cause the library to create the topic. This is because the library does correctly auto-create the topic when you try to publish to it. After that, you should be able to run the worker without error.
Alternatively, you can modify background.js#subscribe to check for the topic's existence and create it if necessary.

Yahoo OpenID returning invalid signature with passport-yahoo

As of today, Dec 10th, 2013, it seems that passport-yahoo is unable to correctly complete the openid workflow. We are getting an invalid signature. My knowledge of openid is quite limited, and I am still looking into the issue. Can anyone provide some insight into this problem? Is anyone else running in to the same problem?
Here is the stack trace from node:
{
"message":"Failed to verify assertion",
"stack":"Failed to verify assertion (message: Invalid signature)
at app/node_modules/passport-yahoo/node_modules/passport-openid/lib/passport-openid/strategy.js:184:36
at app/node_modules/passport-yahoo/node_modules/passport-openid/node_modules/openid/openid.js:1063:12
at app/node_modules/passport-yahoo/node_modules/passport-openid/node_modules/openid/openid.js:1188:14
at app/node_modules/passport-yahoo/node_modules/passport-openid/node_modules/openid/openid.js:1269:7
at Object.openid.loadAssociation (app/node_modules/passport-yahoo/node_modules/passport-openid/node_modules/openid/openid.js:107:5)
at _checkSignatureUsingAssociation (app/node_modules/passport-yahoo/node_modules/passport-openid/node_modules/openid/openid.js:1231:10)
at _checkSignature (app/node_modules/passport-yahoo/node_modules/passport-openid/node_modules/openid/openid.js:1221:5)
at _verifyAssertionAgainstProvider (app/node_modules/passport-yahoo/node_modules/passport-openid/node_modules/openid/openid.js:1184:3)
at app/node_modules/passport-yahoo/node_modules/passport-openid/node_modules/openid/openid.js:1155:16
at app/node_modules/passport-yahoo/node_modules/passport-openid/node_modules/openid/openid.js:678:7
at app/node_modules/passport-yahoo/node_modules/passport-openid/node_modules/openid/openid.js:533:11
at done (app/node_modules/passport-yahoo/node_modules/passport-openid/node_modules/openid/openid.js:254:9)
at IncomingMessage.<anonymous> (app/node_modules/passport-yahoo/node_modules/passport-openid/node_modules/openid/openid.js:258:32)
at IncomingMessage.EventEmitter.emit (events.js:117:20)
at _stream_readable.js:920:16
at process._tickCallback (node.js:415:13)",
"name":"InternalOpenIDError",
"openidError":{"message":"Invalid signature"}
}
Yes - we started having intermittent issues starting this past Friday, then complete failure today. From Y! developer forums, it seems others are having the same problem - http://developer.yahoo.com/forum/OpenID-General-Discussion/OpenID-login-with-Yahoo-broke-today/1386713784256-49fd81b6-7752-4c9f-8b7c-728235c46bae
And in general testing a number of OpenID sites, Yahoo login doesn't work

Resources