NodeJs: v0.12.4
Couchbase: 2.0.8
Service deployed with PM2
Instance of the bucket is created once per service rather than once per call based on recommendation from couchbase support as instantiating and connecting bucket is expensive.
During the load everything seems to be in order with near 0 failure rate.
After couple of days of service being barely if at all in use client fails to connected to the bucket with the following error:
{"message":"Client-Side timeout exceeded for operation. Inspect network conditions or increase the timeout","code":23}
Recycling the node.js process using 'pm2 restart' resolves the issue.
Any ideas/suggestions short of re-creating instance of the bucket and re-connecting to the bucket?
Related
This is the error I'm getting after executing serverless deploy --verbose
Uploading service api.zip file to S3 (20.81 MB)
Recoverable error occurred (Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.), sleeping for ~4 seconds. Try 1 of 4
I tried with set AWS_CLIENT_TIMEOUT=300000 but it does not change a thing, unfortunately.
Any idea on how to go about this? This happened after installing the AWS-SDK package and creating a function that's supposed to be triggered by an S3 upload but I don't think that's the problem, but rather the big file size and my slow internet connection.
Thanks in advance.
I'm new to using AWS EBS and ECS, so please bear with me if I ask questions that might be obvious for others. To the issue:
I've got a single-container Node/Express application that runs on EBS. The local docker container works as expected. On EBS, I can access one endpoint of the API and get the expected output. For the second endpoint, which runs longer (around 10-15 seconds) I get no response and run after 60 seconds into a time out: "504 Gateway Time-out".
I wonder how I would approach debugging this as I can't connect to the container directly? Currently there isn't any debugging functionality in the code included either as I'm not sure what the best node approach for a EBS container is - any recommendations are highly appreciated.
Thank you in advance!
You can see the EC2 instances running on EBS in your AWS, and you can choose to give them IP addresses in your EBS options. That will let you SSH directly into them if you need to.
Otherwise check the keepAliveTimeout field in your server (the value returned by app.listen() of you're using express).
I got a decent number of 504s when my Node server timeout was less than my load balancer timeout.
Your application takes longer than expected (> 60 seconds) to respond, so either nginx or the Load Balancer terminates your request.
See my answer here
I am working on AWS lambda functions (NodeJS) that connects to a MongoDB server running on EC2 Instance.
Lambda function is place in a VPC-1 and MongoDB server (EC2 Instance) is in VPC-2.
We have setup VPC peering between VPC-1 and VPC-2
The lambda function is intermittently throwing timeout error. It works 50% of the time and 50% of the time, it's throwing timeout error.
Note: The MongoDB is running on an EC2 Instance is specially setup for the development of this project. It does not get any additional traffic.
Also, another component of this project developed in NodeJS again running from another EC2 instance can communicate with the MongoDB server without any timeout issues.
Could someone help me in understanding the possible cause of the timeout issues?
Thanks in advance.
Hope below article might solve your problem:
To fix: Increase the timeout setting/memory on the configuration page of your Lambda function
For nodejs async related issues, please refer below link:
AWS Lambda: Task timed out
Lambda timeouts can best be described as
The amount of time that Lambda allows a function to run before stopping it. The default is 3 seconds. The maximum allowed value is 900 seconds.
Within the console you can increase this timeout to a greater number.
When you click on the Lambda function there will be a monitoring tab. From here you should be able to see execution time of Lambda functions. You might find that its always close to the bar.
I'd recommend increasing the timeout a bit higher than you anticipate it needs then reviewing these metrics. Once you have a baseline adjust this timeout value again
I have a very simple piece of code written in node.js which runs on Kubernetes and AWS. The app just does POST/GET request to create and get data from other services. service1-->service2->service3
Service1 get post request and call service2, service2 calls postgres DB (using sequlize) and create a new row and then call service3, service3 get data from the DB and returns the response to service2, service2 returns the response to service1.
Most of the times it works, but once in 4-5 attempts + concurrency, it dropped and I got a timeout. the problem is that the service1 receives the response back (according to the logs and network traces) but it seems that the connection was dropped somewhere between the services and I got a timeout (ESOCKETTIMEDOUT).
I've tried to use to replace request.js with node-fetch
I've tried to use NewRelic/Elastic APM
I've tried to use node -prof and analyze it with node --prof-process with no conclusions.
Is it possible Kubernetes drops my connection?
Hard to tell without debugging but since some connections are getting dropped when you add more load + concurrency it's likely that you need more replicas on your Kubernetes deployments and possibly adjusts the Resources on your container pod specs.
If this turns out to be the case you can also configure an HPA (Horizontal Pod Autoscaler) to handle your load.
I have setup 2 instance under aws load balancer. I have deployed node.js web services + mongodb in both instance. load balancer works fine with web services.
But, Problem is I have one timer service (node.js service only). the behavior of this timer is updating my mongodb based on some calculation.
My problem is, I must need to run this timer service (timer.js) at only one aws instance (out of 2) at same time. and expected that if one aws instance goes down then timer service at other instance will come up.
i know elb not providing this kind of facility.Can any one please help me to make it done ?
Condition : At a time only one timer service must be run with amazon load balancer.
Thanks.
You would have to implement this yourself using a locking algorithm using a shared data store that supports atomic operations
Alternatively, consider starting a "timer" server in an Auto Scale Group of Min:1, Max: 1 so Amazon keeps it running. This instance can be a t2.micro which is very cheap. It can either run the job itself, or just make an http request to your load balancer to run the job at the desired internal. If you so that, only one of your servers will run each job
Wouldn't it make more sense to handle this like any other "service" that needs to keep running?
upstart service
running node.js server using upstart causes 'terminated with status 127' on 'ubuntu 10.04'
This guy had a bad path in his file but his upstart script looks okay
monit
Node.js (sudo) and monit