I have a VM with a RabbitMQ instance running, and a separate VM with a Node process that will consume things off of a queue.
After some time of inactivity, my Node process will just quit and display an error. I have forever keeping the Node process alive but here is what the error looks like:
data: server.js:18171 - error: Script restart attempt #33
data: server.js:18171 - [rabbitmq] Waiting for messages in savvy_shipping_to_logic. To exit press CTRL+C
data: server.js:18171 - Error: read ETIMEDOUT
data: server.js:18171 - at errnoException (net.js:901:11)
data: server.js:18171 - at TCP.onread (net.js:556:19)
data: server.js:18171 - error: Forever detected script exited with code: 8
data: server.js:18171 - error: Script restart attempt #34
data: server.js:18171 - [rabbitmq] Waiting for messages in savvy_shipping_to_logic. To exit press CTRL+C
data: server.js:18171 - Error: read ETIMEDOUT
data: server.js:18171 - at errnoException (net.js:901:11)
data: server.js:18171 - at TCP.onread (net.js:556:19)
data: server.js:18171 - error: Forever detected script exited with code: 8
data: server.js:18171 - error: Script restart attempt #35
both VMs are Ubuntu 14.04 running on Azure. I should note that I did not experience the same issues locally on my Macbook Pro running El Cap. I feel like I'm missing some flag?
I'm connecting to amqp://my.example.com
you may need to set a heartbeat in your connection setting to keep the connection alive. try adding a heartbeat of 30 seconds, for example, and see if that helps.
you may also want to increase the connection timeout.
Related
I'm running a forever script sending data from raspberry pi 3 to azure-iot-hub, using the following,
root#raspberrypi3:~# forever start /home/pi/azure/iam/ble_azure.js
After working for about 1.5 days, I stopped getting messages at auzre-iot-hub, when I checked forever list I got the following:
root#raspberrypi3:~# forever list
info: Forever processes running
data: uid command script forever pid id logfile uptime
data: [0] NWgI /usr/bin/nodejs /home/pi/azure/iam/ble_azure.js 8990 3784 /root/.forever/NWgI.log 0:21:17:38.742
When I checked the log file I get this error message:
/home/pi/azure/iam/node_modules/applicationinsights/AutoCollection/Exceptions.js:27
throw error;
^
NotConnectedError: mqtt.js returned client disconnecting error
at translateError (/home/pi/azure/iam/node_modules/azure-iot-device-mqtt/lib/mqtt-translate-error.js:25:11)
at MqttTwinReceiver._handleError (/home/pi/azure/iam/node_modules/azure-iot-device-mqtt/lib/mqtt-twin-receiver.js:201:42)
at /home/pi/azure/iam/node_modules/azure-iot-device-mqtt/lib/mqtt-twin-receiver.js:64:18
at MqttClient._checkDisconnecting (/home/pi/azure/iam/node_modules/mqtt/lib/client.js:314:7)
at MqttClient.subscribe (/home/pi/azure/iam/node_modules/mqtt/lib/client.js:423:12)
at /home/pi/azure/iam/node_modules/azure-iot-device-mqtt/lib/mqtt-twin-receiver.js:62:22
at _combinedTickCallback (internal/process/next_tick.js:73:7)
at process._tickCallback (internal/process/next_tick.js:104:9)
error: Forever detected script exited with code: 7
error: Script restart attempt #34
state has changed poweredOn
started scanning
[IoT hub Client] Connect error: mqtt.js returned premature close error
Through the log file, mqtt error keeps happening multiple times and forever handled it successfully, what I can't understand why after 1.5 days, I get this error:
**error: Forever detected script exited with code: 7
error: Script restart attempt #34**
Also why I keep getting such mqtt error, why it keeps disconnecting?
**NotConnectedError: mqtt.js returned client disconnecting error**
Forever --version
v0.15.3
root#raspberrypi3:~# uname -a
Linux raspberrypi3 4.9.35-v7+ #1014 SMP Fri Jun 30 14:47:43 BST 2017 armv7l GNU/Linux
Thanks
If you're using Client.fromConnectionString to instantiate the client object the SDK disconnects and reconnects every 45 minutes to renew the shared access signature token. (it doesn't happen with AMQP that uses a different authentication mechanism). It might be that when re-establishing the connection the client hits this "premature close" error that we've been tracking in this issue.
There are 2 things that could help limit potential errors linked to disconnecting/reconnecting:
Use X509 certificates instead of connection strings to authenticate.
Create a client object using Client.fromSharedAccessSignature and build a long-lived signature that does not require disconnecting and reconnecting as often.
Last but not least, the next release of the SDK (1.2.0) will include a retry/reconnect logic that is way more robust than what was there before. I'll update the issue to point to it when it is released.
I have deployed a Bitnami Parse stack to a Google Cloud VM. This stack is basically Apache running as a proxy, Then a MEAN stack handling parse server and dashboard etc.
Everything appears to be working fine when accessing the dashboard and sending individual CURL requests to test it's all available.
However, there appears to be a fatal flaw in that. When Parse returns one of it's errors e.g.
{"code":101,"error":"Object not found."}
Express sees it as an uncaught exception and dies. Now I realise that it's intentional behaviour (There's enough information out there as to why you don't want to persist after an uncaught exception)
The Bitnami stack uses Forever to run the Parse Server, but as you can see from the logs I will include below, you still run into issues with refused connections.
2016-11-25T11:41:16+00:00 DEBUG (7): {"code":101,"error":"Object not found."}
2016-11-25T11:41:16+00:00 DEBUG (7): Failed to connect to X.X.X.X: Connection refused
2016-11-25T11:41:17+00:00 DEBUG (7): Failed to connect to X.X.X.X: Connection refused
2016-11-25T11:41:17+00:00 DEBUG (7): Failed to connect to X.X.X.X: Connection refused
2016-11-25T11:41:17+00:00 DEBUG (7): Failed to connect to X.X.X.X: Connection refused
Meanwhile in the Forever logs:
Error generating response. ParseError { code: 101, message: 'Object not found.' } code=101, message=Object not found.
[object Object]
/opt/bitnami/apps/parse/htdocs/node_modules/parse-server/lib/ParseServer.js:425
throw err;
^
Error: EACCES: permission denied, open '/opt/bitnami/apps/parse/htdocs/logs/parse-server.info.2016-11-25'
at Error (native)
error: Forever detected script exited with code: 7
error: Script restart attempt #1
parse-server running on port 1337
parse-dashboard running on port 4040
So you can see that immediately following a Parse error being returned, The service is unavailable until Forever can restart the Parse Server.
This issue appears that it would affect all users as well, so if one user receives an exception, all users lose connection for a brief period.
So the question is, how should this be properly handled? Is Parse + Express not the way to go about it? Or is there some method of configuring Node or Express to return the API error to the requester, without Express dying (And without having to fall into the trap of using uncaughtException to keep Node alive)
Any help would be much appreciated, thank you
The logs from Forever said it's EACCES error, that means ParseServer couldn't access the log file and not able to write the info log in it.
Error: EACCES: permission denied, open '/opt/bitnami/apps/parse/htdocs/logs/parse-server.info.2016-11-25'
at Error (native)
error: Forever detected script exited with code: 7
error: Script restart attempt #1
parse-server running on port 1337
parse-dashboard running on port 4040
I think you have to change the /opt/bitnami/apps/parse/htdocs/logs with proper permission so that ParseServer can write log files in it.
I have nodejs process that listens for an connection. I want to let foreverjs restart it on any connection error this way:
amqpConnection.on('error', function (err) {
console.error(err);
process.exit(1);
});
But this don't work. After error I see my process in this state:
~> forever list
info: Forever processes running
data: uid command script logfile
data: [0] OpQF /usr/bin/node my-script.js my-script-log.log STOPPED
Log file has connection error and two attempts to restart script.
error: Script restart attempt #2
{ [Error: connect ECONNREFUSED]
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect' }
error: Forever detected script exited with code: 1
Why forever stops trying to restart my process?
I should have read warnings that forever prints!
Script that not stay running more than --minUptime (1000ms by default) considered "spinning". One should set --spinSleepTime to make spinning scripts to be restarted. When this parameter is omitted spinning script won't be restarted.
I have reproduced your problem and found that forever successfully restart your process if some delay (1sec) take place before process.exit.
It's look that forever don't restart process when it crash too fast.
setTimeout(function() {
process.exit(1);
}, 1000);
I have a node server running on AWS and I am using forever to keep it running in the background.
I have been very happy with forever so far; it's super easy to set up and use. However, even though my app is very simple, sometimes it crashes with this error:
Express server listening on port 3001
events.js:72
throw er; // Unhandled 'error' event
^
Error: connect ECONNREFUSED
at errnoException (net.js:901:11)
at Object.afterConnect [as oncomplete] (net.js:892:19)
error: Forever detected script exited with code: 8
error: Forever restarting script for 4 time
When I look back at the logs, I can see that this has happened 4 more times, and each time forever manages to restart the server. However, when it happens for the 5th time, for some reason forever fails to restart the server and my app becomes unresponsive.
Express server listening on port 3001
events.js:72
throw er; // Unhandled 'error' event
^
Error: connect ECONNREFUSED
at errnoException (net.js:901:11)
at Object.afterConnect [as oncomplete] (net.js:892:19)
error: Forever detected script exited with code: 8
/usr/lib/node_modules/forever/node_modules/forever-monitor/node_modules/broadway/node_modules/eventemitter2/lib/ev$
throw arguments[1]; // Unhandled 'error' event
^
Error: Cannot stop process that is not running.
at /usr/lib/node_modules/forever/node_modules/forever-monitor/lib/forever-monitor/monitor.js:332:26
at process._tickCallback (node.js:415:13)
Has anyone else experienced similar problems?
You have to use the option -MAX like this:
forever -MAX 10000000000 Mypath/ToMyFile.js
Also put a -minUptime for the minimum time the server can stay up, if it crash before that laps of time then it won't go up again so set it to 1.
forever --minUptime (number-in-milisecond-here) -MAX (number-of-restart-here) path/file.js
Just try this:
forever --minUptime 1 -MAX 1000000000 YourPath/ToYourFile.js
More documentation there
https://www.npmjs.com/package/forever
To allow a max of 100 restarts when the server crashes
forever start -m 100 index.js
I'm getting socket hang up errors on Nodejitsu, what does it mean?
error: Error running command deploy
error: socket hang up
error: Error: socket hang up
error: at createHangUpError (http.js:1107:15)
error: at Socket.onend (http.js:1188:27)
error: at TCP.onread (net.js:369:26)
info: Nodejitsu not ok
Two things that you might be doing that cause this:
Make sure you're using port > 1024 (no sudo) Make sure your app is <
50mb disk space
If this doesn't work please either email support at nodejitsu or visit #nodejitsu at irc.freenode.org