lambda.invoke call through Node AWS SDK is timing out at 5 minutes while Lambda is completing under 4 minutes - node.js

I have a Lambda function that is set to timeout at 5 minutes, and occasionally when I invoke the function using the node SDK, I am getting a timeout error after 5 minutes.
However, when I have checked the CloudWatch logs after getting the client-side error, and am seeing the function successfully complete after ~3:30 minutes of execution time.
I have also noticed that the client side will log the timeout 2 minutes before the CloudWatch logs read that the function completed (ie, client side error timestamps at 4:08 and the CloudWatch successful end timestamps at 4:10), but I am assuming that is a lag between log writes, not run time. Thought I would mention it just in case that is something worth thinking more about.
From what I can tell, when the function runs under two minutes, it is consistently getting a positive response.
Code looks like this:
let aws_config = new AWS.Config({
region: 'us-east-1',
credentials: credentials,
httpOptions: {
proxy: process.env.HTTP_PROXY,
timeout: 300000
}
})
var lambda = new AWS.Lambda(aws_config);
lambda.invoke(params, (err, data: Result) => {
...
}
Error looks like this:
{ TimeoutError: Connection timed out after 300000ms
at ClientRequest.<anonymous> (.../node_modules/aws-sdk/lib/http/node.js:83:34)
at Object.onceWrapper (events.js:273:13)
at ClientRequest.emit (events.js:182:13)
at ClientRequest.EventEmitter.emit (domain.js:442:20)
at Socket.emitRequestTimeout (_http_client.js:661:40)
at Object.onceWrapper (events.js:273:13)
at Socket.emit (events.js:182:13)
at Socket.EventEmitter.emit (domain.js:442:20)
at Socket._onTimeout (net.js:449:8)
at ontimeout (timers.js:425:11)
message: 'Connection timed out after 300000ms',
code: 'TimeoutError',
time: 2018-10-05T20:08:35.719Z,
region: 'us-east-1',
hostname: 'lambda.us-east-1.amazonaws.com',
retryable: true } 'TimeoutError: Connection timed out after 300000ms\n at ClientRequest.<anonymous> (.../node_modules/aws-sdk/lib/http/node.js:83:34)\n at Object.onceWrapper (events.js:273:13)\n at ClientRequest.emit (events.js:182:13)\n at ClientRequest.EventEmitter.emit (domain.js:442:20)\n at Socket.emitRequestTimeout (_http_client.js:661:40)\n at Object.onceWrapper (events.js:273:13)\n at Socket.emit (events.js:182:13)\n at Socket.EventEmitter.emit (domain.js:442:20)\n at Socket._onTimeout (net.js:449:8)\n at ontimeout (timers.js:425:11)'

Have you looked into
context.callbackWaitsForEmptyEventLoop = false;
Here'e a related SO posting:
Why does AWS Lambda function always time out?

Related

What causes this Node/Postgres FATAL Error

Any help would be appreciated...
Not sure why I get this error every now and then. I've search off and on for a solution but keep coming up empty.
I run my Node App as a service on Centos7. It can run for a few days ... or even a month before getting this error.
I use a Pooled connection to Postgres and am inserting data into several different databases every few seconds.
events.js:377
throw er; // Unhandled 'error' event
^
error: terminating connection due to administrator command
at Parser.parseErrorMessage (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg-protocol/dist/parser.js:287:98)
at Parser.handlePacket (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg-protocol/dist/parser.js:126:29)
at Parser.parse (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg-protocol/dist/parser.js:39:38)
at Socket.<anonymous> (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg-protocol/dist/index.js:11:42)
at Socket.emit (events.js:400:28)
at addChunk (internal/streams/readable.js:290:12)
at readableAddChunk (internal/streams/readable.js:265:9)
at Socket.Readable.push (internal/streams/readable.js:204:10)
at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
Emitted 'error' event on Client instance at:
at Client._handleErrorEvent (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg/lib/client.js:319:10)
at Client._handleErrorMessage (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg/lib/client.js:330:12)
at Connection.emit (events.js:400:28)
at /home/beenth12/public_html/ws/wxbox/node/node_modules/pg/lib/connection.js:114:12
at Parser.parse (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg-protocol/dist/parser.js:40:17)
at Socket.<anonymous> (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg-protocol/dist/index.js:11:42)
[... lines matching original stack trace ...]
at Socket.Readable.push (internal/streams/readable.js:204:10) {
length: 116,
severity: 'FATAL',
code: '57P01',
detail: undefined,
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'postgres.c',
line: '3193',
routine: 'ProcessInterrupts'
}
The documentation will inform you that SQLSTATE 57P01 is admin_shutdown. So somebody shutdown the database, and it is not surprising that your database connection got terminated.

mysql2 connection.connect() hangs in typescript jest test on macOS

I am trying to create a mysql2 connection to a local mysql-Database in a jest-Test on macOS using typescript. And the call to connect() hangs.
Environment:
MacOS: High Sierra (10.13.6)
MySQL: 8.0.18
Typescript: 3.7.2
mysql2: 2.0.1
nodejs: v10.16.3
I am using the promise-Wrapper. The code looks like this:
import { createConnection, ConnectionOptions, Connection } from 'mysql2/promise'
...
async openConnection(options: ConnectionOptions): Promise<Connection> {
console.log('About to create mysql connection using options ', options);
const connection = await createConnection(options);
console.log('Connection created => about to call connect()');
await connection.connect();
console.log('Connection created => connect() returned'); // this line is never reached!
....
return connection;
}
I already added the debug-Option and this is the output:
Dieters-MBP-6:db-typescript rehdie$ node_modules/.bin/jest src/tests/mysql/mysql-connection.spec.ts --testNamePattern="Should open connection"
console.log src/lib/mysql/driver/mysql-driver.ts:21
About to create mysql connection using options { user: 'mysql-test',
password: '1q2w3e4r',
host: 'localhost',
port: 3306,
database: 'mysql_test_db',
charset: 'latin1_swedish_ci',
connectTimeout: 5000,
debug: true }
console.log node_modules/mysql2/lib/connection.js:425
Add command: ClientHandshake
console.log node_modules/mysql2/lib/connection.js:381
raw: 0a382e302e31380018000000277919626469296e00ffffff0200ffc715000000000000000000006d6335316a3c3d454220053a0063616368696e675f736861325f70617373776f726400
console.error console.js:274
Trace:
at Connection.handlePacket (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:387:17)
at PacketParser.Connection.packetParser.p [as onPacket] (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:70:12)
at PacketParser.executeStart (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/packet_parser.js:75:16)
at Socket.Connection.stream.on.data (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:77:25)
at Socket.emit (events.js:198:13)
at addChunk (_stream_readable.js:288:12)
at readableAddChunk (_stream_readable.js:269:11)
at Socket.Readable.push (_stream_readable.js:224:10)
at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
console.log node_modules/mysql2/lib/connection.js:395
0 undefined ==> ClientHandshake#unknown name(0,,78)
console.log node_modules/mysql2/lib/commands/client_handshake.js:96
Server hello packet: capability flags:3355443199=(long password, found rows, long flag, connect with db, no schema, compress, odbc, local files, ignore space, protocol 41, interactive, ssl, ignore sigpipe, transactions, reserved, secure connection, multi statements, multi results, ps multi results, plugin auth, connect attrs, plugin auth lenenc client data, can handle expired passwords, session track, deprecate eof, ssl verify server cert, remember options)
console.log node_modules/mysql2/lib/commands/client_handshake.js:41
Sending handshake packet: flags:11203535=(long password, found rows, long flag, connect with db, odbc, local files, ignore space, protocol 41, ignore sigpipe, transactions, reserved, secure connection, multi results, plugin auth, plugin auth lenenc client data, session track)
console.log node_modules/mysql2/lib/connection.js:253
0 24 <== ClientHandshake#unknown name(1,,104)
console.log node_modules/mysql2/lib/connection.js:257
0 24 <== 64000001cff3aa00000000000800000000000000000000000000000000000000000000006d7973716c2d746573740014a13dd11599725225bcae39464cc3700a5fa085fc6d7973716c5f746573745f6462006d7973716c5f6e61746976655f70617373776f726400
console.log node_modules/mysql2/lib/connection.js:381
raw: fe63616368696e675f736861325f70617373776f726400277919626469296e6d6335316a3c3d454220053a00
console.error console.js:274
Trace:
at Connection.handlePacket (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:387:17)
at PacketParser.Connection.packetParser.p [as onPacket] (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:70:12)
at PacketParser.executeStart (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/packet_parser.js:75:16)
at Socket.Connection.stream.on.data (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:77:25)
at Socket.emit (events.js:198:13)
at addChunk (_stream_readable.js:288:12)
at readableAddChunk (_stream_readable.js:269:11)
at Socket.Readable.push (_stream_readable.js:224:10)
at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
console.log node_modules/mysql2/lib/connection.js:395
0 24 ==> ClientHandshake#unknown name(2,,48)
console.log node_modules/mysql2/lib/connection.js:253
0 24 <== ClientHandshake#unknown name(3,,36)
console.log node_modules/mysql2/lib/connection.js:257
0 24 <== 20000003374930d53dd94bc2dbe6b54e4b83cd628658a1984656eb6ee4f31d7df596ce36
console.log node_modules/mysql2/lib/connection.js:381
raw: 0103
console.error console.js:274
Trace:
at Connection.handlePacket (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:387:17)
at PacketParser.Connection.packetParser.p [as onPacket] (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:70:12)
at PacketParser.executeStart (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/packet_parser.js:75:16)
at Socket.Connection.stream.on.data (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:77:25)
at Socket.emit (events.js:198:13)
at addChunk (_stream_readable.js:288:12)
at readableAddChunk (_stream_readable.js:269:11)
at Socket.Readable.push (_stream_readable.js:224:10)
at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
console.log node_modules/mysql2/lib/connection.js:395
0 24 ==> ClientHandshake#unknown name(4,,6)
console.log node_modules/mysql2/lib/connection.js:381
raw: 000000024000000010010e0d6d7973716c5f746573745f6462
console.error console.js:274
Trace:
at Connection.handlePacket (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:387:17)
at PacketParser.Connection.packetParser.p [as onPacket] (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:70:12)
at PacketParser.executeStart (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/packet_parser.js:75:16)
at Socket.Connection.stream.on.data (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:77:25)
at Socket.emit (events.js:198:13)
at addChunk (_stream_readable.js:288:12)
at readableAddChunk (_stream_readable.js:269:11)
at Socket.Readable.push (_stream_readable.js:224:10)
at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
console.log node_modules/mysql2/lib/connection.js:395
0 24 ==> ClientHandshake#unknown name(5,maybeOK,29)
console.log src/lib/mysql/driver/mysql-driver.ts:24
Connection created => about to call connect()
RUNS src/tests/mysql/mysql-connection.spec.ts
When I perform a show process list in mysql, I see a new process with Command 'Sleep' (state is empty and info is null).
Any ideas?
You need to ensure that node is using the system time and not fake timers. Add the below line immediately after your imports in the test file
jest.useRealTimers();
Or better still you can set timers to real in your jest config
"timers":"real"

Intermittent redis connection error: 'UNCERTAIN STATE'

Time and time again, we get this error:
{ Error: Redis connection to some-redis-server:6379 failed - read ETIMEDOUT
at TCP.onStreamRead (internal/stream_base_commons.js:171:27) errno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'read' }
{ AbortError: Redis connection lost and command aborted. It might have been processed.
at RedisClient.flush_and_error (/usr/src/app/node_modules/redis/index.js:362:23)
at RedisClient.connection_gone (/usr/src/app/node_modules/redis/index.js:664:14)
at Socket.<anonymous> (/usr/src/app/node_modules/redis/index.js:289:14)
at Object.onceWrapper (events.js:281:20)
at Socket.emit (events.js:193:13)
at TCP._handle.close (net.js:614:12)
code: 'UNCERTAIN_STATE',
command: 'GET'
args: [ 'some-key' ] }
We are using https://www.npmjs.com/package/redis to access redis.
Some details of the setup:
Currently 25GB of data stored in Redis
Writes = a rate of approximately 2MB/sec.
Max number of concurrent clients connected: 400
I don't know where to really start looking, so any help would be appreciated.

Node.js, ws-tcp-relay with nats: connection error

Please help solve the problem:
When I start the nats server listening 4222 port:
./gnatsd --addr localhost --port 4222, then
./ws-tcp-relay localhost:4222 -p 4223 and try to connect through websockets using websocket-nats like this (in service.js for example):
'use strict';
const nats = require('websocket-nats').connect('ws://localhost:4223');
I get this error:
node server.js
events.js:188
throw err;
^
Error: Unhandled "error" event. (Could not connect to server: Error: getaddrinfo ENOTFOUND localhost:4223 localhost:4223:4223)
at Client.emit (events.js:186:19)
at Socket.<anonymous> (/home/dev/projects/node-pub/node_modules/websocket-nats/lib/nats.js:404:14)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at emitErrorNT (internal/streams/destroy.js:64:8)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
But, if I try to connect to nats directly using tcp client port 4222, the connection establishes succesfull:
const nats = require('nats').connect('nats://localhost:4222');

Create Task in Asana Node.js

Using the below in Node trying to create a task within Workspace. I have omitted the AccessToken & WorkspaceID numbers. Suspect I'm doing something really dumb :)
var asana = require('asana');
var client = asana.Client.create().useAccessToken('0/XXXXXXXX');
var newTask = { name: "Your Mission" };
client.tasks.createInWorkspace(1111111111, newTask).then(function(response) {
tasks = response.data;
console.log(tasks);
});
At runtime I get the following output...
Unhandled rejection Error: Invalid Request
at InvalidRequest.AsanaError (/Users/D/Asana_Crons/node_modules/asana/lib/errors/error.js:4:11)
at new InvalidRequest (/Users/D/Asana_Crons/node_modules/asana/lib/errors/invalid_request.js:5:14)
at Request._callback (/Users/D/Asana_Crons/node_modules/asana/lib/dispatcher.js:152:23)
at Request.self.callback (/Users/D/Asana_Crons/node_modules/request/request.js:186:22)
at emitTwo (events.js:125:13)
at Request.emit (events.js:213:7)
at Request.<anonymous> (/Users/D/Asana_Crons/node_modules/request/request.js:1163:10)
at emitOne (events.js:115:13)
at Request.emit (events.js:210:7)
at IncomingMessage.<anonymous> (/Users/D/Asana_Crons/node_modules/request/request.js:1085:12)
at Object.onceWrapper (events.js:314:30)
at emitNone (events.js:110:20)
at IncomingMessage.emit (events.js:207:7)
at endReadableNT (_stream_readable.js:1047:12)
at _combinedTickCallback (internal/process/next_tick.js:102:11)
at process._tickCallback (internal/process/next_tick.js:161:9)
The only issue I can see with your code is that the workspace id needs to be a string, not an integer. This is documented in the tasks section of /gen folder in the Asana Node client library.

Resources