Below is my filebeat.yml file , I am unable to start service getting error
filebeat.prospectors:
- input_type: log
** paths:**
** - /opt/apache-tomcat-7.0.82/logs/*.log**
document_type: apache-access
fields_under_root: true
output.logstash:
** hosts: '${host}'**
host is environment variable i did export host="10.2.3.1:5044"
Apr 10 06:59:35 node1 filebeat[401]: Exiting: error initializing publisher: missing field accessing 'output.logstash.hosts' (source:'/etc/filebeat/filebeat.yml')
Apr 10 06:59:35 node1 systemd[1]: filebeat.service: main process exited, code=exited, status=1/FAILURE
Apr 10 06:59:35 node1 systemd[1]: Unit filebeat.service entered failed state.
Apr 10 06:59:35 node1 systemd[1]: filebeat.service failed.
Apr 10 06:59:35 node1 systemd[1]: filebeat.service holdoff time over, scheduling restart.
Apr 10 06:59:35 node1 systemd[1]: start request repeated too quickly for filebeat.service
Apr 10 06:59:35 node1 systemd[1]: Failed to start filebeat.
Apr 10 06:59:35 node1 systemd[1]: Unit filebeat.service entered failed state.
Apr 10 06:59:35 node1 systemd[1]: filebeat.service failed.``
I found solution, That's the issue is Systemd does not pass environment variables. I started filebeat as ./filebeat -e -c filebeat.yml it worked for me
By looking at this I see multiple syntax differences to your configuration. There are some examples at the bottom of the page even for hosts.
https://www.elastic.co/guide/en/beats/filebeat/1.2/using-environ-vars.html
Hope it helps.
Related
Docker service running on Centos 7 failed to start, I have some docker images which I want to save at any cost. I have searched a couple of online docs and they all say to delete /var/lib/docker/ dir which I don't want to because all the images and containers stuff is there. Can someone please save me how to get docker back up and running with losing any data.
Log:
[root#BuyPandGDev01 /]# systemctl status docker.service -l
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Sun 2018-04-22 00:05:23 UTC; 19min ago
Docs: http://docs.docker.com
Process: 1539 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES (code=exited, status=1/FAILURE)
Main PID: 1539 (code=exited, status=1/FAILURE)
Apr 22 00:05:22 BuyPandGDev01 systemd[1]: Starting Docker Application Container Engine...
Apr 22 00:05:22 BuyPandGDev01 dockerd-current[1539]: time="2018-04-22T00:05:22.068920976Z" level=info msg="libcontainerd: new containerd process, pid: 1550"
Apr 22 00:05:23 BuyPandGDev01 dockerd-current[1539]: time="2018-04-22T00:05:23.101036303Z" level=warning msg="devmapper: Usage of loopback devices is strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or use `man docker` to refer to dm.thinpooldev section."
Apr 22 00:05:23 BuyPandGDev01 dockerd-current[1539]: time="2018-04-22T00:05:23.155223108Z" level=error msg="[graphdriver] prior storage driver \"devicemapper\" failed: devmapper: Base Device UUID and Filesystem verification failed: devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed"
Apr 22 00:05:23 BuyPandGDev01 dockerd-current[1539]: time="2018-04-22T00:05:23.155708413Z" level=fatal msg="Error starting daemon: error initializing graphdriver: devmapper: Base Device UUID and Filesystem verification failed: devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed"
Apr 22 00:05:23 BuyPandGDev01 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Apr 22 00:05:23 BuyPandGDev01 systemd[1]: Failed to start Docker Application Container Engine.
Apr 22 00:05:23 BuyPandGDev01 systemd[1]: Unit docker.service entered failed state.
Apr 22 00:05:23 BuyPandGDev01 systemd[1]: docker.service failed.
journalctl -xe:
[root#BuyPandGDev01 /]# journalctl -xe
-- Unit docker-storage-setup.service has begun starting up.
Apr 22 00:25:58 BuyPandGDev01 container-storage-setup[2111]: INFO: Volume group backing root filesystem could not be determined
Apr 22 00:25:58 BuyPandGDev01 container-storage-setup[2111]: ERROR: No valid volume group found. Exiting.
Apr 22 00:25:58 BuyPandGDev01 systemd[1]: docker-storage-setup.service: main process exited, code=exited, status=1/FAILURE
Apr 22 00:25:58 BuyPandGDev01 systemd[1]: Failed to start Docker Storage Setup.
-- Subject: Unit docker-storage-setup.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker-storage-setup.service has failed.
--
-- The result is failed.
Apr 22 00:25:58 BuyPandGDev01 systemd[1]: Unit docker-storage-setup.service entered failed state.
Apr 22 00:25:58 BuyPandGDev01 systemd[1]: docker-storage-setup.service failed.
Apr 22 00:25:58 BuyPandGDev01 systemd[1]: Starting Docker Application Container Engine...
-- Subject: Unit docker.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.service has begun starting up.
Apr 22 00:25:58 BuyPandGDev01 dockerd-current[2140]: time="2018-04-22T00:25:58.731142431Z" level=info msg="libcontainerd: new containe
Apr 22 00:25:59 BuyPandGDev01 dockerd-current[2140]: time="2018-04-22T00:25:59.767061431Z" level=warning msg="devmapper: Usage of loop
Apr 22 00:25:59 BuyPandGDev01 kernel: device-mapper: table: 253:1: thin: Couldn't open thin internal device
Apr 22 00:25:59 BuyPandGDev01 kernel: device-mapper: ioctl: error adding target to table
Apr 22 00:25:59 BuyPandGDev01 dockerd-current[2140]: time="2018-04-22T00:25:59.835261589Z" level=error msg="[graphdriver] prior storag
Apr 22 00:25:59 BuyPandGDev01 dockerd-current[2140]: time="2018-04-22T00:25:59.835697590Z" level=fatal msg="Error starting daemon: err
Apr 22 00:25:59 BuyPandGDev01 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Apr 22 00:25:59 BuyPandGDev01 systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.service has failed.
--
-- The result is failed.
Apr 22 00:25:59 BuyPandGDev01 systemd[1]: Unit docker.service entered failed state.
Apr 22 00:25:59 BuyPandGDev01 systemd[1]: docker.service failed.
Apr 22 00:25:59 BuyPandGDev01 polkitd[703]: Unregistered Authentication Agent for unix-process:2105:147803 (system bus name :1.43, obj
lines 2751-2788/2788 (END)
Any response would be helpful and appreciated.
Thx,
kumar
This error occurred for me when I was upgrading docker. Solution that worked for me was to remove legacy docker files /var/lib/docker/ and restart the docker service. Here is the solution.
# Remove docker files
$ rm -rf /var/lib/docker/
# Restart docker via service or via systemctl
$ service docker restart
$ service docker status
$ systemctl start docker.service
$ systemctl status docker.service
I had this error also starting the docker service:
kernel: device-mapper: table: 253:1: thin: Couldn't open thin internal device
I fixed it by creating a soft link from /var/lib/docker to another location on the machine which had more disk space.
cd /var/lib/
mv docker docker.old
ln -s /path/to/big/disk/docker/ docker
Restart the service:
systemctl restart docker
I've added a service (Seafile, in this case) that I want to have running at all times to systemd with a service file. It works great, but every time the unattended updates run, the service gets shutdown properly - but never restarted.
Here's what the service file looks like:
[Unit]
Description=Seafile
Requires=mysql.service
After=mysql.service
[Install]
WantedBy=multi-user.target
[Service]
Type=forking
User=root
Group=root
PermissionsStartOnly=true
ExecStart=/srv/start-seafile
TimeoutSec=600
Restart=on-failure
/srv/start-seafile looks like this:
#!/bin/bash
cd /srv/seafile/XXXX/seafile-server-latest && nohup ./seafile.sh start
Like I said - this works perfectly - systemd can enable / disable / start / stop the service, realize if it's started / running - so I must be doing something right.
# systemctl start seafile
# systemctl status seafile
* seafile.service - Seafile
Loaded: loaded (/etc/systemd/system/seafile.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2018-01-28 17:53:51 CET; 8s ago
Process: 6569 ExecStart=/srv/start-seafile (code=exited, status=0/SUCCESS)
CGroup: /system.slice/seafile.service
|-6598 /srv/seafile/XXXX/seafile-server-5.1.4/seafile/bin/seafile-controller -c /srv/seafile/XXXX/ccnet -d /volume1/Seafile -F /srv/seafile/XXXX/conf
|-6600 ccnet-server -F /srv/seafile/XXXX/conf -c /srv/seafile/XXXX/ccnet -f /srv/seafile/XXXX/logs/ccnet.log -d -P /srv/seafile/XXXX/pids/ccnet.pid
`-6602 seaf-server -F /srv/seafile/XXXX/conf -c /srv/seafile/XXXX/ccnet -d /volume1/Seafile -l /srv/seafile/XXXX/logs/seafile.log -P /srv/seafile/XXXX/pids/seaf-server.pid
Jan 28 17:53:48 XXXX systemd[1]: Starting Seafile...
Jan 28 17:53:48 XXXX start-seafile[6569]: [01/28/18 17:53:48] ../common/session.c(132): using config file /srv/seafile/XXXX/conf/ccnet.conf
Jan 28 17:53:48 XXXX start-seafile[6569]: Starting seafile server, please wait ...
Jan 28 17:53:51 XXXX start-seafile[6569]: Seafile server started
Jan 28 17:53:51 XXXX start-seafile[6569]: Done.
Jan 28 17:53:51 XXXX systemd[1]: Started Seafile.
However, every time the unattended updates come, this happens:
Jan 23 06:38:08 XXXX systemd[1]: Starting Daily apt activities...
Jan 23 06:40:13 XXXX systemd[1]: Reloading.
Jan 23 06:40:13 XXXX systemd[1]: Stopping Seahub...
Jan 23 06:40:14 XXXX systemd[1]: Stopped Seahub.
Jan 23 06:40:14 XXXX systemd[1]: Stopping Seafile...
Jan 23 06:40:15 XXXX systemd[1]: Stopped Seafile.
Jan 23 06:40:15 XXXX systemd[1]: Stopping MySQL Community Server...
Jan 23 06:40:16 XXXX systemd[1]: Stopped MySQL Community Server.
Jan 23 06:40:17 XXXX systemd[1]: Reloading.
Jan 23 06:40:17 XXXX systemd[1]: Stopped MySQL Community Server.
...
Jan 23 06:40:43 XXXX systemd[1]: Reloading.
Jan 23 06:40:44 XXXX systemd[1]: Stopped MySQL Community Server.
Jan 23 06:40:44 XXXX systemd[1]: Reloading.
Jan 23 06:40:44 XXXX systemd[1]: Reloading.
Jan 23 06:40:44 XXXX systemd[1]: Starting MySQL Community Server...
Jan 23 06:40:45 XXXX systemd[1]: Started MySQL Community Server.
Jan 23 06:40:47 XXXX systemd[1]: Reloading.
Jan 23 06:40:48 XXXX systemd[1]: Stopping MySQL Community Server...
Jan 23 06:40:49 XXXX systemd[1]: Stopped MySQL Community Server.
Jan 23 06:40:49 XXXX systemd[1]: Starting MySQL Community Server...
Jan 23 06:40:50 XXXX systemd[1]: Started MySQL Community Server.
(... continues with unrelated services )
So it realizes that Seafile needs to be stopped before MySQL, and does so, but no longer starts it after it restarts MySQL.
Does anyone have any experience as to what could be causing this? ie. under which circumstances systemd services will be stopped during an update, but not restarted?
Use Restart=always if you want it to run at all times. When mysql service update happens, this service does a clean stop and therefore systemd doesn't restart it. You have Restart=on-failure set, which only restarts if the stop has a return code other than 0.
Restart = always
RestartSec = 10
RestartSec
Configures the time to sleep before restarting a service (as configured with Restart=). Takes a unit-less value in seconds, or a time span value such as "5min 20s". Defaults to 100ms.
We have an application that is running on RHEL6/32 bit and RHEL6/64 bit. This application uses postgresql 8.4 from the beginning. Now, we want to provide support for this application on RHEL7/64 bit. RHEL7 comes with default postgresql 9.2 in its yum list and this is getting installed and its related services are running properly as well. But after installing postgresql 8.4 on RHEL7, it seems like the services are never running. Please find below the logs:
[root#linpubn218 postgres]# service postgresql status
postgresql.service - SYSV: PostgreSQL database server.
Loaded: loaded (/etc/rc.d/init.d/postgresql)
Active: failed (Result: resources) since Mon 2016-07-25 12:40:28 IST; 2h 0min ago
Docs: man:systemd-sysv-generator(8)
Jul 25 12:40:26 linpubn218.gl.avaya.com systemd[1]: Starting SYSV: PostgreSQL database server....
Jul 25 12:40:28 linpubn218.gl.avaya.com postgresql[26957]: Starting postgresql service: [ OK ]
Jul 25 12:40:28 linpubn218.gl.avaya.com systemd[1]: PID file /var/run/postmaster-8.4.pid not readable (yet?) after start.
Jul 25 12:40:28 linpubn218.gl.avaya.com systemd[1]: Failed to start SYSV: PostgreSQL database server..
Jul 25 12:40:28 linpubn218.gl.avaya.com systemd[1]: Unit postgresql.service entered failed state.
Jul 25 12:40:28 linpubn218.gl.avaya.com systemd[1]: postgresql.service failed.
Jul 25 14:33:45 linpubn218.gl.avaya.com systemd[1]: Unit postgresql.service cannot be reloaded because it is inactive.
Jul 25 14:33:45 linpubn218.gl.avaya.com systemd[1]: Unit postgresql.service cannot be reloaded because it is inactive.
After looking at the logs in journalctl -xe
[root#linpubn218 postgres]# journalctl -xe
Jul 25 14:39:21 linpubn218.gl.avaya.com yum[29260]: Installed: postgresql84-libs-8.4.17-1PGDG.rhel6.x86_64
Jul 25 14:39:45 linpubn218.gl.avaya.com yum[29275]: Installed: postgresql84-8.4.17-1PGDG.rhel6.x86_64
Jul 25 14:40:01 linpubn218.gl.avaya.com useradd[29316]: failed adding user 'postgres', exit code: 9
Jul 25 14:40:02 linpubn218.gl.avaya.com CROND[29320]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jul 25 14:40:02 linpubn218.gl.avaya.com systemd[1]: Reloading.
Jul 25 14:40:03 linpubn218.gl.avaya.com systemd[1]: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as config
Jul 25 14:40:03 linpubn218.gl.avaya.com yum[29309]: Installed: postgresql84-server-8.4.17-1PGDG.rhel6.x86_64
Jul 25 14:42:05 linpubn218.gl.avaya.com polkitd[819]: Registered Authentication Agent for unix-process:29459:43987285 (system bus name :1.292 [/usr/bin/pkttyagent --not
Jul 25 14:42:05 linpubn218.gl.avaya.com systemd[1]: Starting SYSV: PostgreSQL database server....
Jul 25 14:42:06 linpubn218.gl.avaya.com runuser[29473]: pam_unix(runuser-l:session): session closed for user postgres
Jul 25 14:42:08 linpubn218.gl.avaya.com postgresql[29464]: Starting postgresql service: [ OK ]
Jul 25 14:42:08 linpubn218.gl.avaya.com systemd[1]: PID file /var/run/postmaster-8.4.pid not readable (yet?) after start.
Jul 25 14:42:08 linpubn218.gl.avaya.com systemd[1]: Failed to start SYSV: PostgreSQL database server..
Can postgresql 8.4 be installed on RHEL7, which is a systemd based OS? If yes, then what should I do to remove the above error?
I noticed that in /etc/init.d/postgresql-8.4 there is a declared variable:
pidfile="/var/run/postmaster-${PGMAJORVERSION}.${PGPORT}.pid"
But in systemctl, PIDfile is not the same:
# systemctl show postgresql-8.4.service -p PIDFile
PIDFile=/var/run/postmaster-8.4.pid
So, to fix the problem edit /etc/init.d/postgresql-8.4 and replace
pidfile="/var/run/postmaster-${PGMAJORVERSION}.${PGPORT}.pid"
with
pidfile="/var/run/postmaster-${PGMAJORVERSION}.pid"
then reload systemctl:
# systemctl daemon-reload
#/etc/init.d/postgresql-8.4 start
Starting postgresql-8.4 (via systemctl): [ OK ]
Generally permissions caused this type of error
su - postgres
After that:
chmod 700 -R <data_directory>
And you should check SELinux as well.
I installed packstack on my fresh installation of Fedora 21 with all updates. When I run
packstack --allinone I received this error:
ERROR : Error appeared during Puppet run: 192.168. 1.*_keystone.pp Error:
Could not start Service[keystone]: Execution of '/sbin/service openstack-keystone
start'` returned 1: Redirecting to /bin/systemctl start openstack-keystone.service
You will find full trace in log /var/tmp/packstack/20141223-022613-whLvTs/manifests
/192.168.1.*_keystone.pp.log
And this is the log:
Notice: /Stage[main]/Cinder::Keystone::Auth/Keystone_user_role[cinder#services]:
Dependency Service[keystone] has failures: true
Warning: /Stage[main]/Cinder::Keystone::Auth/Keystone_user_role[cinder#services]:
Skipping because of failed dependencies
Notice: Finished catalog run in 13.02 seconds
With systemctl status openstack-keystone.service get this:
openstack-keystone.service - OpenStack Identity Service (code-named Keystone)
Loaded: loaded (/usr/lib/systemd/system/openstack-keystone.service; disabled)
Active: failed (Result: start-limit) since Tue 2014-12-23 19:47:36 EET; 1min 59s ago
Process: 22526 ExecStart=/usr/bin/keystone-all (code=exited, status=1/FAILURE)
Main PID: 22526 (code=exited, status=1/FAILURE)
Dec 23 19:47:35 localhost.localdomain systemd[1]: Failed to start OpenStack...
Dec 23 19:47:35 localhost.localdomain systemd[1]: Unit openstack-keystone.s...
Dec 23 19:47:35 localhost.localdomain systemd[1]: openstack-keystone.servic...
Dec 23 19:47:36 localhost.localdomain systemd[1]: start request repeated to...
Dec 23 19:47:36 localhost.localdomain systemd[1]: Failed to start OpenStack...
Dec 23 19:47:36 localhost.localdomain systemd[1]: Unit openstack-keystone.s...
Dec 23 19:47:36 localhost.localdomain systemd[1]: openstack-keystone.servic...
This can happen due SELinux avc denial because of a missing policy.
You can try to put SELinux to permissive mode:
# setenforce 0
A similar bug
When I try to start the apache this happens:
Job for apache2.service failed. See ‘systemctl status apache2.service’ and ‘journalctl -xn’ for details.
system.ctl status apache2.service -l returns me this:
Mar 24 23:41:57 glauber-pc.site start_apache2[3249]: httpd2-prefork: Syntax error on line 179 of /etc/apache2/httpd.conf: Syntax error on line 102 of /etc/apache2/default-server.conf: Syntax error on line 1 of /etc/apache2/conf.d/mod_evasive.conf: Cannot load /usr/lib64/apache2/mod_evasive20.so into server: /usr/lib64/apache2/mod_evasive20.so: cannot open shared object file: No such file or directory
Mar 24 23:41:57 glauber-pc.site systemd[1]: apache2.service: main process exited, code=exited, status=1/FAILURE
Mar 24 23:41:57 glauber-pc.site systemd[1]: Failed to start The Apache Webserver.
Mar 24 23:41:57 glauber-pc.site systemd[1]: Unit apache2.service entered failed state.
Everything seems right in those lines, any clue what it can be?
Change the first line in
/etc/apache2/conf.d/mod_evasive.conf
to
LoadModule evasive20_module /usr/lib64/apache2/mod_evasive24.so
(...24.so instead of ...20.so)