I'm trying to increase the open file descriptors limit for a transient scope (enable_e0d20b64-6e48-4e38-a718-6ae53745eddb.scope), but I'm not able to.
Here's the hierarchy of the slice and scope units:
├─azure.slice
│ ├─waagent.service
│ │ ├─18305 /usr/bin/python -u /usr/sbin/waagent -daemon
│ │ ├─27946 python -u bin/WALinuxAgent-2.9.0.4-py2.7.egg -run-exthandlers
│ │ ├─28127 /bin/bash /var/lib/waagent/Microsoft.ManagedServices.ApplicationHealthLinux-1.0.5/bin/applicationhealth-shim enable
│ │ ├─28129 tee -ia /var/log/azure/applicationhealth-extension/handler.log
│ │ └─28144 /var/lib/waagent/Microsoft.ManagedServices.ApplicationHealthLinux-1.0.5/bin/applicationhealth-extension enable
│ └─azure-vmextensions.slice
│ └─azure-vmextensions-Microsoft.Azure.Monitoring.DependencyAgent.DependencyAgentLinux_9.10.13.19190.slice
│ └─enable_e0d20b64-6e48-4e38-a718-6ae53745eddb.scope
│ ├─18431 /bin/sh /opt/microsoft/dependency-agent/bin/microsoft-dependency-agent-manager
│ └─18515 /opt/microsoft/dependency-agent/bin/microsoft-dependency-agent
I have tried adding an override at 'azure.slice', but it doesn't look slices support the LimitNOFILE directive.
$ systemctl cat azure.slice
# /usr/lib/systemd/system/azure.slice
[Unit]
Description=Slice for Azure VM Agent and Extensions
DefaultDependencies=no
Before=slices.target
# /etc/systemd/system/azure.slice.d/override.conf
[Slice]
LimitNOFILE=4096
Error
Jan 04 17:14:48 mq01.####.onmicrosoft.com systemd[1]: [/etc/systemd/system/azure.slice.d/override.conf:2] Unknown lvalue 'LimitNOFILE' in section 'Slice'
systemctl cat azure.slice
# /usr/lib/systemd/system/azure.slice
[Unit]
Description=Slice for Azure VM Agent and Extensions
DefaultDependencies=no
Before=slices.target
# /etc/systemd/system/azure.slice.d/override.conf
[Service]
LimitNOFILE=4096
Error
Jan 04 17:23:03 mq01.####.onmicrosoft.com systemd[1]: [/etc/systemd/system/azure.slice.d/override.conf:1] Unknown section 'Service'. Ignoring.
systemd.resource-control doesn't show anything that would be applicable to what I need.
Open file limits are a per-process rlimit, not a cgroup option, so they cannot be set at slice level.
Unlike transient services, a transient scope is usually created with existing processes (i.e. ones that were spawned directly by some other service or app), so you should instead raise LimitNOFILE= on the parent – which I'm assuming is waagent.service – and the new limit will be inherited by all of its child processes, regardless of their later placement in the cgroup tree.
Related
I'm running a NodeJS application on ubuntu LTS 20.4 managed by PM2. Application is running fine but when I check the logs I see lots of EADDRINUSE address already in use message.
I started the server using the command sudo pm2 start index.js
Error: listen EADDRINUSE: address already in use :::8000
at Server.setupListenHandle [as _listen2] (node:net:1432:16)
at listenInCluster (node:net:1480:12)
at Server.listen (node:net:1568:7)
at file:///home/ubuntu/wapi/index.js:105:10
at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
code: 'EADDRINUSE',
errno: -98,
syscall: 'listen',
address: '::',
port: 8000
}
cleanup
Stack trace is pointing to line number 105 of the file below.
https://github.com/billbarsch/myzap/blob/myzap2.0/index.js
What I don't understand is why PM2 is trying to start the server almost every second (because this message appears in the log every second) when the service is already running?
And sudo pm2 ls is listing 2 processes
│ id │ name │ namespace │ version │ mode │ pid │ uptime │ ↺ │ status │ cpu │ mem │ user │ watching │
│ 0 │ index │ default │ 1.0.0 │ fork │ 1673211 │ 103s │ 130 │ online │ 0% │ 111.8mb │ root │ disabled │
│ 1 │ index │ default │ 1.0.0 │ fork │ 1673848 │ 2s │ 450… │ online │ 66.7% │ 120.3mb │ root │ disabled │
Really appreciate some help.
Thanks
It appears that you already have another process of pm2 which is running the same application. That is why you are seeing EADDRINUSE.
And the reason you are getting the same log every second is that pm2 tends to restart the application when it errors out.
You can stop all the processes using
pm2 stop all
And then try to re-run your process.
Your error tell that another process already use the specified port.
That can be every process on your server, not only a node process running under PM2.
To determine what process already use the port, you can issue the netstat command:
netstat -ano -p -t | grep 8000
This will print out ALL process connected to this port, server as client. To identify server process look for LISTEN.
If not logged as privileged user, use sudo:
sudo netstat -ano -p -t | grep 8000
Is there a way to figure out what process triggered pinentry prompt?
In other words, imagine the prompt pops up, and you have no idea why (what process, what action triggered it). How would you figure it out?
Another question is more general - what is signalling mechanism behind such dialogs (dbus? gpg unix socket? something else?).
P.S.
Unfortunately process tree does not help:
├─systemd,138622 --user
│ ├─(sd-pam),138623
│ ├─dbus-daemon,138647 --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
│ ├─dconf-service,139188
│ │ ├─{gdbus},139190
│ │ └─{gmain},139189
│ ├─gpg-agent,139317 --supervised
│ │ ├─pinentry,139327 --display :0
│ │ │ ├─{QDBusConnection},139349
│ │ │ └─{QXcbEventQueue},139330
I have a NestJS application running in docker with PM2 and it's extremely slow although it consumes very little resources. The reason is definitely not the traffic as there is nearly no traffic. When looking into PM2 Monitoring I see that the HTTP Latency is extremely high.
When running the same application locally I can't see any of this issues.
This is a snapshot of one of the clusters in PM2.
│ Heap Size 106.32 MiB │
│ Heap Usage 86.07 % │
│ Used Heap Size 91.51 MiB │
│ Active requests 0 │
│ Active handles 16 │
│ Event Loop Latency 0.61 ms │
│ Event Loop Latency p95 1.59 ms │
│ HTTP Mean Latency 2 ms │
│ HTTP P95 Latency 9752 ms │
│ HTTP 0 req/min │
Any ideas what I can change in the configuration or how I can investigate into this issue? I haven't found anything on this topic anywhere.
You can delete that app and restart, pm2 delete app and pm2 start index.js. This form will clear that latency. This has happened with my application
I am using CentOS 7 server, node version 10.23.0, child-process version 6.14.9 and I should watch the status of the given services.
For that, I'm using Childprocess.exec function in format systemctl status servicename. This command is working properly when service is active, but giving an error when service is inactive. In command line, despite status of the service, all is working good.
I've tried to use systemctl is-active servicename but there is also error. I don't know what is the reason. Error message is
{Error: Command failed: systemctl status crond
at ChildProcess.exithandler (child_process.js:294:12)
at ChildProcess.emit (events.js:198:13)
at maybeClose (internal/child_process.js:982:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)
killed: false,
code: 3,
signal: null,
cmd: 'systemctl status crond' }
NOTE: I should use child-process.
Systemctl's exit codes are documented in man systemctl as:
EXIT STATUS
On success, 0 is returned, a non-zero failure code otherwise.
systemctl uses the return codes defined by LSB, as defined in LSB 3.0.0[2].
Table 3. LSB return codes
┌──────┬───────────────────────────┬──────────────────────────┐
│Value │ Description in LSB │ Use in systemd │
├──────┼───────────────────────────┼──────────────────────────┤
│0 │ "program is running or │ unit is active │
│ │ service is OK" │ │
├──────┼───────────────────────────┼──────────────────────────┤
│1 │ "program is dead and │ unit not failed (used by │
│ │ /var/run pid file exists" │ is-failed) │
├──────┼───────────────────────────┼──────────────────────────┤
│2 │ "program is dead and │ unused │
│ │ /var/lock lock file │ │
│ │ exists" │ │
├──────┼───────────────────────────┼──────────────────────────┤
│3 │ "program is not running" │ unit is not active │
├──────┼───────────────────────────┼──────────────────────────┤
│4 │ "program or service │ no such unit │
│ │ status is unknown" │ │
In your output you have code: 3, so it's telling you what you already know, that the service is not active, but since it is exiting with a non-zero code exec() thinks it is an error.
When you say it runs fine on the command line, it's actually operating the exact same way, but you wouldn't notice the exit code was 3 unless you checked the variable $? afterwards.
You can parse the error in your callback against systemctl's documented exit codes to determine if it was an actual error or not given your use case.
I have install pm2 globally
sudo pm2 install -g
pm2 start server.js
pm2 status (gives this output)
┌──────────┬────┬──────┬───────┬────────┬─────────┬────────┬─────┬───────────┬─────────┬──────────┐
│ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ user │ watching │
├──────────┼────┼──────┼───────┼────────┼─────────┼────────┼─────┼───────────┼─────────┼──────────┤
│ server │ 0 │ fork │ 10094 │ online │ 0 │ 85s │ 0% │ 44.7 MB │ ubuntu │ disabled │
└──────────┴────┴──────┴───────┴────────┴─────────┴────────┴─────┴───────────┴─────────┴──────────┘
When pm2 log 0
we get following error
1|server | 2018-01-23 14:35 +00:00: Tue, 23 Jan 2018 14:35:03 GMT zap2it:server Server now running on localhost:4040
1|server | 2018-01-23 14:35 +00:00: Tue, 23 Jan 2018 14:35:03 GMT zap2it:server spawning worker #53
1|server | 2018-01-23 14:35 +00:00: EPERM, Operation not permitted on call initgroups
1|server | 2018-01-23 14:35 +00:00: ubuntu is not accessible
What permission I need for pm2 to run or where I can look for errors?
Can I install and run pm2 using root?
you must execute pm2 update like:
sudo npm install -g pm2
pm2 update
pm2 start server.js
PM2 logs can be found at <HOME>/.pm2/logs/ and you should be able to install and run it using root, although this is not recommended (as stated in the comments by savior123).
I've just run into the same issue and error messages as you a while ago - although not running PM2 with sudo - and solved it by updating PM2 version (from 2.9.2 to 2.9.3), as commented by Unitech
I solved this issue on windows by going to the services and stop the pm2 service from running and at my terminal, I did the pm2 start app.js and everything ran fine