couchdb.service: Failed with result 'start-limit-hit' - couchdb

After I installed couchdb, I could get the welcome information
$ curl localhost:5984
{"couchdb":"Welcome","version":"2.1.2","features":["scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
But I can't check the status by systemctl
$ systemctl status couchdb.service
● couchdb.service
Loaded: not-found (Reason: No such file or directory)
Active: failed (Result: start-limit-hit) since 一 2018-12-03 14:52:14 CST; 6min ago
Main PID: 30946 (code=killed, signal=USR2)
12月 03 14:52:14 gpuhuawei systemd[1]: couchdb.service: Unit entered failed state.
12月 03 14:52:14 gpuhuawei systemd[1]: couchdb.service: Failed with result 'signal'.
12月 03 14:52:14 gpuhuawei systemd[1]: couchdb.service: Service hold-off time over, scheduling restart.
12月 03 14:52:14 gpuhuawei systemd[1]: Stopped Apache CouchDB.
12月 03 14:52:14 gpuhuawei systemd[1]: couchdb.service: Start request repeated too quickly.
12月 03 14:52:14 gpuhuawei systemd[1]: Failed to start Apache CouchDB.
12月 03 14:52:14 gpuhuawei systemd[1]: couchdb.service: Unit entered failed state.
12月 03 14:52:14 gpuhuawei systemd[1]: couchdb.service: Failed with result 'start-limit-hit'.
12月 03 14:53:53 gpuhuawei systemd[1]: Stopped Apache CouchDB.
12月 03 14:53:53 gpuhuawei systemd[1]: Stopped Apache CouchDB.
When I run couchdb by command line, I got
$ couchdb
{"init terminating in do_boot",{{badmatch,{error,{bad_return,{{couch_app,start,[normal,["/etc/couchdb/default.ini","/etc/couchdb/local.ini"]]},{'EXIT',{{badmatch,{error,{error,eacces}}},[{couch_server_sup,start_server,1,[{file,"couch_server_sup.erl"},{line,56}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,273}]}]}}}}}},[{couch,start,0,[{file,"couch.erl"},{line,18}]},{init,start_it,1,[]},{init,start_em,1,[]}]}}
[1] 2288 user-defined signal 2 couchdb
My work enviroment
$ uname -a
Linux gpuhuawei 4.15.0-34-generic #37~16.04.1-Ubuntu SMP Tue Aug 28 10:44:06 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

This is a bit late, but the "start-limit-hit" message is a red herring. I have seen something very similar with a Moodle installation using MySQL and it's actually saying that you (or the service start process) have tried to restart the database too many times or too soon after a failed attempt to start. Basically, this start-limit-hit message is saying "stop trying to do the same thing and expecting different results".
The actual issue will be further up in the syslog. Unhelpfully, service status does not return enough lines of the error messages to actually see what is wrong. Try a service start, and go and look in the actual syslog, and you will see the series of start attempts and a line just above each one hopefully will tell you the actual issue. In my case here, you can see that the problem is the mount point containing the database is missing - thanks, Azure. For one attempt of service start, it tries to start 5 times in quick succession, failing each time because the data dir was not mounted, and on the sixth it fails with the start-limit-hit.

Always back up your data/ and etc/ directories prior to upgrading CouchDB.
We recommend that you overwrite your etc/default.ini file with the version provided by the new release. New defaults sometimes contain mandatory changes to enable default functionality. Always places your customization in etc/local.ini or any etc/local.d/*.ini file.
(I was followed this and it worked)
https://docs.couchdb.org/en/3.0.0/install/upgrading.html

Related

Elastisearch Enabling Remote Connection - Crashes AFTER Change*

I just installed filebeat, logstash, kibana and elasticsearch all running smoothly just to trial this product out for additional monthly reports/monitoring and noticed every time I try to change the "/etc/elasticsearch/elasticsearch.yml" config file for remote web access it'll basically crash the service every time I make the change.
Just want to say I'm new to the forum and this product, and my end goal for this question is to figure out how to allow remote connections to access elastisearch as I guinea pig and test without crashing elasticsearch.
For reference here is the error code when I run the 'sudo systemctl status elasticsearch' query:
Dec 30 07:27:37 ubuntu systemd[1]: Starting Elasticsearch...
Dec 30 07:27:52 ubuntu systemd-entrypoint[4067]: ERROR: [1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
Dec 30 07:27:52 ubuntu systemd-entrypoint[4067]: bootstrap check failure [1] of [1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.se>
Dec 30 07:27:52 ubuntu systemd-entrypoint[4067]: ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elasticsearch/elasticsearch.log
Dec 30 07:27:53 ubuntu systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/CONFIG
Dec 30 07:27:53 ubuntu systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Dec 30 07:27:53 ubuntu systemd[1]: Failed to start Elasticsearch.
Any help on this is greatly appreciated!

Why is puppet server was running fine but failing after some time fails?

When I logged in to my VM (Ubuntu 18.04) for second time, its showing error:
# systemctl status puppetserver.service
puppetserver.service - puppetserver Service Loaded: loaded
(/lib/systemd/system/puppetserver.service; enabled; vendor preset: enabled)
**Active**: failed (Result: exit-code) since Wed 2019-10-02 11:42:52 UTC; 2min 31s ago
Process: 23034
ExecStart=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver
start (code=exited, status=1/FAILURE)
Oct 02 11:42:52 puppet-master systemd[1]: puppetserver.service:
Control process exited, code=exited status=1 Oct 02 11:42:52
puppet-master systemd[1]: puppetserver.service: Failed with result
'exit-code'. Oct 02 11:42:52 puppet-master systemd[1]: Failed to start
puppetserver Service. Oct 02 11:42:52 puppet-master systemd[1]:
puppetserver.service: Service hold-off time over, scheduling restart.
Oct 02 11:42:52 puppet-master systemd[1]: puppetserver.service:
Scheduled restart job, restart counter is at 5. Oct 02 11:42:52
puppet-master systemd[1]: Stopped puppetserver Service. Oct 02
11:42:52 puppet-master systemd[1]: puppetserver.service: Start request
repeated too quickly. Oct 02 11:42:52 puppet-master systemd[1]:
puppetserver.service: Failed with result 'exit-code'. Oct 02 11:42:52
puppet-master systemd[1]: Failed to start puppetserver Service.
Is there a way to identify the issue?
To troubleshoot more, you can run:
journalctl -xe -u puppetserver
And also check the /var/log/puppetlabs/puppetserver/puppetserver.log file for more info, but based on the error message:
Start request repeated too quickly
It seems that you didn't have patience to restart (as it can take few minutes).
The best to restart I recommend to you:
systemctl restart puppet*
systemctl restart puppet pxp-agent
In this way, the services manage the dependency between all puppet* services.
Note: Server Fault, part of Stack Exchange (includes Stack Overflow) provides support for managing information technology systems in a business environment.

Can't restart webmin [status 2]

I've updated webmin, but now, it refuse to restart :
● webmin.service - LSB: web-based administration interface for Unix systems
Loaded: loaded (/etc/init.d/webmin; generated; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2018-07-29 09:30:29 CEST; 12s ago
Docs: man:systemd-sysv-generator(8)
Process: 1485 ExecStart=/etc/init.d/webmin start (code=exited, status=2)
Jul 29 09:30:26 vps513135 systemd[1]: Starting LSB: web-based administration interface for Unix systems...
Jul 29 09:30:27 vps513135 perl[1486]: pam_unix(webmin:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=root
Jul 29 09:30:29 vps513135 systemd[1]: webmin.service: Control process exited, code=exited status=2
Jul 29 09:30:29 vps513135 systemd[1]: Failed to start LSB: web-based administration interface for Unix systems.
Jul 29 09:30:29 vps513135 systemd[1]: webmin.service: Unit entered failed state.
Jul 29 09:30:29 vps513135 systemd[1]: webmin.service: Failed with result 'exit-code'.
Can someone explain me what does pam_unix(webmin:auth): authentication failure mean ?
some more infos :
root#vps513135:~# uname -a
Linux vps513135 4.9.0-7-amd64 #1 SMP Debian 4.9.110-1 (2018-07-05) x86_64 GNU/Linux
Thank you :)
SOLUTION
I tried to start like this
root#vps513135:~# /etc/webmin/start
Starting Webmin server in /usr/share/webmin
Failed to open SSL key /home/sowdowdow/domains/sow.sowdowdow.fr/ssl.key at /usr/share/webmin/miniserv.pl line 4414.
The output is a bit more clear, and finally found a solution here.
Comment out the lines related to the borked server in /etc/webmin/miniserv.conf.
#ipcert_sow.sowdowdow.fr,*.sow.sowdowdow.fr=/home/sowdowdow/domains/sow.sowdowdow.fr/ssl.cert
#ipkey_sow.sowdowdow.fr,*.sow.sowdowdow.fr=/home/sowdowdow/domains/sow.sowdowdow.fr/ssl.key
Webmin doesn't support systemctl command. Instead of that please use the following command to start Webmin service.
/etc/rc.d/init.d/webmin stop
systemctl start webmin
I tried this command and I am able to start Webmin service on my server.

haproxy error on restarting

HAProxy error on restarting. This is the error i have:
# systemctl status haproxy
● haproxy.service - SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments.
Loaded: loaded (/etc/rc.d/init.d/haproxy; bad; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2017-11-03 03:34:04 EDT; 16s ago
Docs: man:systemd-sysv-generator(8)
Process: 6170 ExecStart=/etc/rc.d/init.d/haproxy start (code=exited, status=1/FAILURE)
Nov 03 03:34:04 server systemd[1]: Starting SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments....
Nov 03 03:34:04 server haproxy[6170]: /etc/rc.d/init.d/haproxy: line 26: [: =: unary operator expected
Nov 03 03:34:04 server haproxy[6170]: Starting haproxy: [WARNING] 306/033404 (6178) : config : frontend 'GLOBAL' has no 'bind' directive. Please declare it... intended.
Nov 03 03:34:04 server haproxy[6170]: [ALERT] 306/033404 (6178) : Starting frontend http_front: cannot bind socket [0.0.0.0:80]
Nov 03 03:34:04 server haproxy[6170]: [FAILED]
Nov 03 03:34:04 server systemd[1]: haproxy.service: control process exited, code=exited status=1
Nov 03 03:34:04 server systemd[1]: Failed to start SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments..
Nov 03 03:34:04 server systemd[1]: Unit haproxy.service entered failed state.
Nov 03 03:34:04 server systemd[1]: haproxy.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
My configuration file:
haproxy.cfg
Since you are using systemd, you should use systemd unit file instead of the init.d script.
I don't know how you installed haproxy, you can find haproxy.service in haproxy source directory(contrib/systemd), copy it to systemd folder and use it.
cp contrib/systemd_haproxy.service /lib/systemd/system/
systemctl daemon-reload
systemctl enable haproxy
systemctl start haproxy
systemctl status haproxy

Arangodb stops and won't restart after dev-xvdb times out

I have arangodb 3.1.16 installed on an AWS C4 Instance. I have a Foxx Service trying to run in production. It is getting an average of 10 packets of 200 octets per second, and returning a flow of 20 packets of 200 octets per second.
Each time I start running my process, the foxx service runs with consistent performance for an hour and then suddenly stops. I do not have access to my foxx api anymore : all requests get connection timeout errors, and do not print on the foxx logs. I do not have access to the web interface anymore : the page just doesn’t load.
After a minute or so, the foxx logs show me an error message : 'ArangoError 18: lock timeout’
After an other minute the logs show me requests that are usually fast but took a very long time (WARNING {queries} slow query: took: 1770.862498)
Using "journalctl -xe", I learned that after a foreign IP tried to connect, I got = "Job dev-xvdb.device/start timed out"
I managed to restart arango using :
ps -eaf |grep arangod
sudo kill #
sudo apt-get --reinstall install arangodb3=3.1.16
How can I solve this recurring issue ?
"journalctl -xe" gives me :
Apr 04 15:03:10 my-ip systemd[1]: arangodb3.service: Failed with result 'exit-code’.
-- Subject: Unit arangodb3.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit arangodb3.service has begun starting up.
Apr 04 15:03:10 my-ip arangodb3[11481]: * Starting arango database server arangod
Apr 04 15:03:10 my-ip arangodb3[11481]: * database version check failed, maybe you need to run 'upgrade'?
Apr 04 15:03:10 my-ip systemd[1]: arangodb3.service: Control process exited, code=exited status=1
Apr 04 15:03:10 my-ip systemd[1]: Failed to start LSB: arangodb.
-- Subject: Unit arangodb3.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit arangodb3.service has failed.
--
-- The result is failed.
Apr 04 15:03:10 my-ip systemd[1]: arangodb3.service: Unit entered failed state.
Apr 04 15:03:10 my-ip systemd[1]: arangodb3.service: Failed with result 'exit-code'.
Apr 04 15:03:10 my-ip sudo[11346]: pam_unix(sudo:session): session closed for user root
Apr 04 15:03:17 my-ip sshd[11502]: Did not receive identification string from UNKNOWN IP 1
Apr 04 15:03:21 my-ip sshd[11503]: Connection closed by UNKNOWN IP 2 port 54736 [preauth]
Apr 04 15:03:21 my-ip sshd[11507]: Did not receive identification string from UNKNOWN IP 2
Apr 04 15:03:21 my-ip sshd[11506]: fatal: Unable to negotiate with UNKNOWN IP 2 port 54730: no matching host key type found. Their offer: ssh-dss [preauth]
Apr 04 15:03:21 my-ip sshd[11504]: Connection closed by UNKNOWN IP 2 port 54732 [preauth]
Apr 04 15:03:22 my-ip sshd[11505]: Connection closed by UNKNOWN IP 2 port 54734 [preauth]
Apr 04 15:03:40 my-ip systemd[1]: dev-xvdb.device: Job dev-xvdb.device/start timed out.
Apr 04 15:03:40 my-ip systemd[1]: Timed out waiting for device dev-xvdb.device.
-- Subject: Unit dev-xvdb.device has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit dev-xvdb.device has failed.
--
-- The result is timeout.
Apr 04 15:03:40 my-ip systemd[1]: Dependency failed for File System Check on /dev/xvdb.
-- Subject: Unit systemd-fsck#dev-xvdb.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit systemd-fsck#dev-xvdb.service has failed.
--
-- The result is dependency.
Apr 04 15:03:40 my-ip systemd[1]: Dependency failed for /mnt.
-- Subject: Unit mnt.mount has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mnt.mount has failed.
--
-- The result is dependency.
Apr 04 15:03:40 my-ip systemd[1]: mnt.mount: Job mnt.mount/start failed with result 'dependency'.
Apr 04 15:03:40 my-ip systemd[1]: systemd-fsck#dev-xvdb.service: Job systemd-fsck#dev-xvdb.service/start failed with result 'dependency'.
Apr 04 15:03:40 my-ip systemd[1]: dev-xvdb.device: Job dev-xvdb.device/start failed with result 'timeout'.
I tried :
sudo curl --dump - -X GET http://127.0.0.1:8529/_api/version && echo
It gives me :
HTTP/1.1 401 Unauthorized
Www-Authenticate: Bearer token_type="JWT", realm="ArangoDB"
Server: ArangoDB
Connection: Keep-Alive
Content-Type: text/plain; charset=utf-8
Content-Length: 0
I tried :
ps auxw | fgrep arangod
It gives me :
root 10439 0.0 0.1 82772 8664 ? Ss 10:09 0:00 /usr/sbin/arangod --uid arangodb --gid arangodb --pid-file /var/run/arangodb/arangod.pid --temp.path /var/tmp/arangod --log.foreground-tty false --supervisor
arangodb 10440 5.7 94.5 12901776 7242340 ? Sl 10:09 16:36 /usr/sbin/arangod --uid arangodb --gid arangodb --pid-file /var/run/arangodb/arangod.pid --temp.path /var/tmp/arangod --log.foreground-tty false --supervisor
ubuntu 11339 0.0 0.0 12916 1000 pts/0 R+ 14:59 0:00 grep -F --color=auto arangod
arangod restart gives me :
2017-04-04T15:01:16Z [11344] INFO ArangoDB 3.1.16 [linux] 64bit, using VPack 0.1.30, ICU 54.1, V8 5.0.71.39, OpenSSL 1.0.2g 1 Mar 2016
2017-04-04T15:01:16Z [11344] INFO using SSL options: SSL_OP_CIPHER_SERVER_PREFERENCE, SSL_OP_TLS_ROLLBACK_BUG
2017-04-04T15:01:16Z [11344] FATAL could not open shutdown file '/var/log/arangodb3/restart/SHUTDOWN': internal error
'service arangodb3 restart’ gives me (after a short wait time) :
Job for arangodb3.service failed because the control process exited with error code. See "systemctl status arangodb3.service" and "journalctl -xe" for details.
'systemctl status arangodb3.service' gives me :
arangodb3.service - LSB: arangodb
Loaded: loaded (/etc/init.d/arangodb3; bad; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2017-04-04 15:03:10 UTC; 34s ago
Docs: man:systemd-sysv-generator(8)
Process: 11352 ExecStop=/etc/init.d/arangodb3 stop (code=exited, status=0/SUCCESS)
Process: 11481 ExecStart=/etc/init.d/arangodb3 start (code=exited, status=1/FAILURE)
Tasks: 83
Memory: 6.5G
CPU: 73ms
CGroup: /system.slice/arangodb3.service
├─10439 /usr/sbin/arangod --uid arangodb --gid arangodb --pid-file /var/run/arangodb/arangod.pid --temp.path /var/tmp/arangod --log.foreground-tty false --supervisor
└─10440 /usr/sbin/arangod --uid arangodb --gid arangodb --pid-file /var/run/arangodb/arangod.pid --temp.path /var/tmp/arangod --log.foreground-tty false --supervisor
Apr 04 15:03:10 my-ip systemd[1]: Starting LSB: arangodb...
Apr 04 15:03:10 my-ip arangodb3[11481]: * Starting arango database server arangod
Apr 04 15:03:10 my-ip arangodb3[11481]: * database version check failed, maybe you need to run 'upgrade'?
Apr 04 15:03:10 my-ip systemd[1]: arangodb3.service: Control process exited, code=exited status=1
Apr 04 15:03:10 my-ip systemd[1]: Failed to start LSB: arangodb.
Apr 04 15:03:10 my-ip systemd[1]: arangodb3.service: Unit entered failed state.
From your log output it seems that the mounted disk volume goes away.
If the storage goes away under any kind of Database there is no reasonable way to continue working.
Thus the effects you see is that the ArangoDB isn't able to work with its data anymore - from its perspective its simply not there anymore.
One effect observed by others is that I/O credits on AWS dry up, which could also be the reason for what you see above.
https://aws.amazon.com/blogs/aws/new-burst-balance-metric-for-ec2s-general-purpose-ssd-gp2-volumes/
If I got that correctly, you can get more credits if you choose a bigger volume size. If that doesn't help, you either need to lower your test scenario, or choose a different hosting approach that doesn't have limitations on I/O operations.

Resources