Change docker log messages location - linux

I met a problem with docker logging and after reading a lot of sources didn't find solution: is there a way to discard messages of docker daemon in /var/log/messages and select another location?

Ok, I know that this question is quite old but I don't think it has been answered well and no correct answer has been stated.
First of all the reason why it saves messages to that particular place starts in rsyslog configuration (/etc/rsyslog.conf) with the line:
$ModLoad imjournal # provides access to the systemd journal
So, because docker saves information to systemd journal it ends at /var/log/messages.
To be able to save it to other places, you have to create a rule like the following at /etc/rsyslog.d/docker.conf.
$FileCreateMode 0644
template(name="DockerLogFileName" type="list") {
constant(value="/var/log/docker/")
property(name="syslogtag" securepath="replace" \
regex.expression="docker/\\(.*\\)\\[" regex.submatch="1")
constant(value="/docker.log")
}
if $programname == 'dockerd' then \
/var/log/docker/combined.log
if $programname == 'dockerd' then \
if $syslogtag contains 'docker/' then \
?DockerLogFileName
else
/var/log/docker/no_tag/docker.log
$FileCreateMode 0600
I found the information for this configuration here:
https://www.simulmedia.com/blog/2016/02/19/centralized-docker-logging-with-rsyslog/

Configure rsyslog to isolate the Docker logs into their own file. To do this create /etc/rsyslog.d/10-docker.conf and copy the following content into the file.
# Docker logging
daemon.* {
/var/mylog
stop
}
In summary this will write all logs for the daemon category to /var/mylog then stop processing that log entry so it isn’t written to the systems default syslog file.

According to the Docker documentation, you can specify a different driver either as a command-line argument for the docker daemon or (preferably) in the daemon.json config file. Several drivers are available, e.g. for Syslog, HTTP-based logging, ...
Update
Here's an example configuration section for Syslog (from the documentation):
{
"log-driver": "syslog",
"log-opts": {
"syslog": "udp://1.2.3.4:1111"
}
}

Related

Docker - unable to run script

What I'm doing
I am using AWS batch to run a docker container for a large compute job. I have configured the ECR/ECS successfully to the best of my knowledge but am having issues running the required commands for reasons that are beyond my level of understanding with docker ( newbie )
What I need to do is pass the below commands into my application and start my application to perform some heavy computing tasks; all commands listed below must be present.
The Issue(s)
The issue arises when I send the submit job to AWS batch; this service pulls the image from the ACR ( amazon container repository ) and spins up a compute environment. The issue comes from when I try to run the command I pass in, below I will go throgh it.
"command": [
"mkdir -p logging",
"chmod 777 logging/",
"docker run -t -i -e my-application", # container name
"-e APIKEY",
"-e BASEURI",
"-e APIUSER",
"-v WORKSPACE /logging:/src/log",
"DOCKERIMAGE",
"python my_app.py",
"-t APP_USER",
"-e APP_ENVIRONMENT",
"-u APP_USERNAME",
"-p APP_PASSWORD",
"-i IN_PATH",
"-o OUT_PATH",
"-b tmp/"
]
The command above generates the following error(s)
container_linux.go:370: starting container process caused: exec: "mkdir -p log": executable file not found in $PATH
I tried to pass in the command to echo the env var $PATH but was unsuccesfull getting a response and resulted in a similar error.
I have ran successfully "ls" and was able to see the directory contents of my application inside.
I am not however able to run any of these commands that I have included in the command [] section. I have tried just running python and such in hopes of getting a more detailed error but was unsuccessful.
Logic in plain English
Create a path called logging if it doesnt exist
set the permissions for logging
run the docker container and pass in the environment variables while doing so
Tell docker to run the python file my_app.py and pass in the expected runtime args
Execute and perform the required logic deligated in the python3 application
Questions
Why can I not create a directory here called "logging" where am I ?
Am I running these properly as defined by AWS batch? or docker
What am I missing or where am I going wrong?
AWS Batch high level doc
AWS Batch link specific to what i'm doing
Assuming that you're following the syntax described in the Container
Properties
section of the AWS docs, you have several problems with the syntax of
your command directive.
First
The command directive can only run a single command. You can't mash together a bunch of commands as you're trying to do in your example. If you need to run multiple commands you would need to embed them as an argument to a shell. For example, something like:
command: ["/bin/sh", "-c", "mkdir -p logging; chmod 777 logging; ..."]
Second
You must properly tokenize your
command lines -- that is, when you type mkdir -p logging at the
command prompt, the shell splits this into three parts (or "tokens"): ['mkdir', '-p', 'logging']. You need to do the same thing when building up the
list of arguments to command.
This is invalid:
command: ["mkdir -p logging"]
That would looking for a command named mkdir -p logging, and of course no such command exists. That would properly be written as:
command: ["mkdir", "-p", "logging"]
Third
I'm not very familiar with the AWS batch environment, but it's unlikely you can run a docker command inside a docker` container as you're trying to do. It's unclear why you're doing this, though: why not just configure your AWS batch job with the appropriate image, environment variables, etc?
Take a look at some of these example job definitions.

How to change the logging options for JSON log

When we run up a container on a Compute Engine using COS, it writes its logs to JSON files. We are finding an error:
"level=error msg="Failed to log msg \"\" for logger json-file: write /var/lib/docker/containers/[image]-json.log: no space left on device".
I was looking to change the logging settings for Docker and found this article on changing the logging driver settings:
https://docs.docker.com/config/containers/logging/json-file/
My puzzle is I don't know how to set the parameters through the console or gcloud in order to set log-opts.
It seems that /var/lib/docker is on the / filesystem, and if this filesystem is running out of inodes, you will receive that message when you’ll try to run up a container and it tries to write its logs to JSON files. You can check this by running
df -i /var/lib/docker
You can configure your logging drivers to change the default values in ‘/etc/docker/daemon.json’
This is a configuration example of the daemon.json file
cat /etc/docker/daemon.json
{
"live-restore": true,
"storage-driver": "overlay2"
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "3",
"labels": "production_status",
"env": "os,customer"
}
}
Don’t forget to restart the docker daemon after changed the file.:
systemctl restart docker.service
You can check the following documentation for further information about how to configure logging drivers.
Please let me know the results.

No logs appear on Cloudwatch log group for elastic beanstalk environment

I have an elastic beanstalk environment, which is running a docker container that has a node js API. On the AWS Console, if I select my environment, then go to Configuration/Software I have the following:
Log groups: /aws/elasticbeanstalk/my-environment
Log streaming: Enabled
Retention: 3 days
Lifecycle: Keep after termination.
However, if I click on that log group on the Cloudwatch console, I have a Last Event Time of some weeks ago (which I believe corresponds to when the environment was created) and have no content on the logs.
Since this is a dockerized application, Logs for the server itself should be at /aws/elasticbeanstalk/my-environment/var/log/eb-docker/containers/eb-current-app/stdouterr.log.
If I instead get the Logs directly from the instances by going once again to my EB environment, clicking "Logs" and then "Request last 100 Lines" the logging is happening correctly. I just can't see a thing when using CloudWatch.
Any help is gladly appreciated
I was able to get around this problem.
So CloudWatch makes a hash based on the first line of your log file and the log stream key, and the problem is that my first line on the stdouterr.log file was actually an empty line!
After couple of days playing around and getting help from the good AWS support team, I first connected via SSH to my EC2 instance associated to the EB environment and you need to add the following line to the /etc/awslogs/config/beanstalklogs.conf file, right after the "file=/var/log/eb-docker/containers/eb-current-app/stdouterr.log" line:
file_fingerprint_lines=1-20
With these, you tell the AWS service that it should calculate the hash using lines 1 through 20 on the log file. You could change 20 for larger or smaller numbers depending on your logging content; however I don't know if there is an upper limit for the value.
After doing so, you need to restart the AWS Logs Service on the instance.
For this you would execute:
sudo service awslogs stop
sudo service awslogs start
or simpler:
sudo service awslogs restart
After these steps I started using my environment and the logging was now being properly streamed to the CloudWatch console!
However this would not work if a new deployment is made, if the EC2 instance gets replaced or the auto scalable group spawns another.
To have a fix for this, it is possible to add log config via the .ebextensions directory, at the root of your application before deploying.
I added a file called logs.config to the newly created .ebextensions directory and placed the following content:
files:
"/etc/awslogs/config/beanstalklogs.conf":
mode: "000644"
user: root
group: root
content: |
[/var/log/eb-docker/containers/eb-current-app/stdouterr.log]
log_group_name=/aws/elasticbeanstalk/EB-ENV-NAME/var/log/eb-docker/containers/eb-current-app/stdouterr.log
log_stream_name={instance_id}
file=/var/log/eb-docker/containers/eb-current-app/*stdouterr.log
file_fingerprint_lines=1-20
commands:
01_remove_eb_stream_config:
command: 'rm /etc/awslogs/config/beanstalklogs.conf.bak'
02_restart_log_agent:
command: 'service awslogs restart'
Changing of course EB-ENV-NAME by my environment name on EB.
Hope it can help someone else!
For 64 bit Amazon Linux 2 the setup is slightly different.
For the delivery of log the AWS CloudWatch Agent is installed in /opt/aws/amazon-cloudwatch-agent and the Elastic Beanstalk configuration is in /opt/aws/amazon-cloudwatch-agent/etc/beanstalk.json. It is set to log the output of the container assuming there's a file called stdouterr.log, here's a snippet of the config:
{
"file_path": "/var/log/eb-docker/containers/eb-current-app/stdouterr.log",
"log_group_name": "/aws/elasticbeanstalk/EB-ENV-NAME/var/log/eb-docker/containers/eb-current-app/stdouterr.log",
"log_stream_name": "{instance_id}"
}
However when I look for the file_path it doesn't exist, instead I have a file path that encodes the current docker container ID /var/log/eb-docker/containers/eb-current-app/eb-e4e26c0bc464-stdouterr.log.
This logfile is created by a script /opt/elasticbeanstalk/config/private/eb-docker-log-start that is started by the eb-docker-log service, the default contents of this file are:
EB_CONFIG_DOCKER_CURRENT_APP=`cat /opt/elasticbeanstalk/deployment/.aws_beanstalk.current-container-id | cut -c 1-12`
mkdir -p /var/log/eb-docker/containers/eb-current-app/
docker logs -f $EB_CONFIG_DOCKER_CURRENT_APP >> /var/log/eb-docker/containers/eb-current-app/eb-$EB_CONFIG_DOCKER_CURRENT_APP-stdouterr.log 2>&1
To temporarily fix the logging you can manually run (replacing the docker ID) and then logs will start to appear in CloudWatch:
ln -sf /var/log/eb-docker/containers/eb-current-app/eb-e4e26c0bc464-stdouterr.log /var/log/eb-docker/containers/eb-current-app/stdouterr.log
To make this permanant I added an .ebextension to fix the eb-docker-log service so it re-makes this link so create a file in your source code in .ebextensions called fix-cloudwatch-logging.config and set it's contents to:
files:
"/opt/elasticbeanstalk/config/private/eb-docker-log-start" :
mode: "000755"
owner: root
group: root
content: |
EB_CONFIG_DOCKER_CURRENT_APP=`cat /opt/elasticbeanstalk/deployment/.aws_beanstalk.current-container-id | cut -c 1-12`
mkdir -p /var/log/eb-docker/containers/eb-current-app/
ln -sf /var/log/eb-docker/containers/eb-current-app/eb-$EB_CONFIG_DOCKER_CURRENT_APP-stdouterr.log /var/log/eb-docker/containers/eb-current-app/stdouterr.log
docker logs -f $EB_CONFIG_DOCKER_CURRENT_APP >> /var/log/eb-docker/containers/eb-current-app/eb-$EB_CONFIG_DOCKER_CURRENT_APP-stdouterr.log 2>&1
commands:
fix_logging:
command: systemctl restart eb-docker-log.service
cwd: /home/ec2-user
test: "[ ! -L /var/log/eb-docker/containers/eb-current-app/stdouterr.log ] && systemctl is-active --quiet eb-docker-log"

How to add Dell Equillogic to Nagios

Note: All firmware and models are compatible, that is why nothing is posted about it.
I've been working on this now for a few hours (reading manuals and such) so I'm not just coming here right out of the blue. I am working on a PRE-EXISTING Nagios server where there are several other existing plugins and checks running and working. Now I want to add another server there to check so I made the following modifications:
First and foremost, I added a file to /usr/local/nagios/libexec named: check_equallogic.sh. The permissions are 755, the same as all others. I have chowned to nagios:nagios and in the listing it shows the Owner as Nagios.
I then added a command to the commands.cfg file in \usr\local\nagios\etc\objects that shows the following:
# 'check_equallogic' command definition
define command{
command_name check_equallogic
command_line $USER1$/check_equallogic -H $HOSTADDRESS$ -C $ARG1$ -t $ARG2$ $ARG3$
}
Following this, I created a file named equallogic.cfg in the objects directory and it contains (more or less):
define host{
use linux-server ; Inherit default values from a template
host_name 172.16.50.11 ; The name we're giving to this device
alias EqualLogic ; A longer name associated with the device
address 172.16.50.11 ; IP address of the device
contact_groups admins
}
Check Equallogic Information
define service{
use generic-service
host_name 172.16.50.11
service_description General Information
check_command check_equallogic!public!info
}
After ensuring that permissions are okay for all files, I restart the nagios service, no errors. When I go into the WebGUI, I get the following errors AFTER the check runs:
(Return code of 127 is out of bounds - plugin may be missing)
Extra, probably unrelated problem
Furthermore, when I log into the EquilLogic server, under Audit logs I get the following error:
Level: AUDIT
Time: 26/05/2014 3:59:13 PM
Member: ps4100-1
Subsystem: agent
Event ID: 22.7.1
SNMP packet validation failed, request received from 172.16.10.11
An snmpwalk receives a timeout, whereas others succeed. I will work on importing the MIBs tomorrow. The reason why I am mentioning it is because I want to make sure that it is only a MIB issue for the SNMP. If it is, then ignore this area.
I am entirely unsure of what to do here.
This doesn't look like a MiBs issue at all.
If snmpwalk fails, your device is not configured properly for snmp or the credentials in your possession are wrong.
Furthermore, on a general note, it is bad practice to create commands definitions for untested plugins. First you need to make sure that your plugin works from the command line, then you add it to Nagios' config.
Since I don't see this essential step in what you wrote, i will assume you didn't test the plugin.
If the plugin does not work and you need help with that please open a new question.

Fuse Fabric: How to delete a configuration PID from a profile?

I began modifying a profile and made some mistakes along the way.
Because of this I have PIDs in the profile which I'd like to delete entirely.
These can be seen in the fabric:profile-display default output shown at the bottom of this post.
They are:
http:
patch.repositories=http:
org.ops4j.pax.url.mvn.repositories=http:
I can't find the correct command to delete this. I've tried:
config:delete org.ops4j.pax.url.mvn.repositories=http:
which successfully completes. But the default profile still lists this pid.
I've also tried:
fabric:profile-edit --delete -p org.ops4j.pax.url.mvn.repositories=http: default
which fails with:
Error executing command: String index out of range: -1
This indicates a property path /property must be specified.
Appending simply / doesn't work either.
One more problem is that I have a pid with a seemingly empty name, as indicated by this line:
PID: (nothing follows this output prefix).
Output of fabric:profile-display default:
Profile id: default
Version : 1.0
Parents :
Associated Containers :
Container settings
----------------------------
Repositories :
mvn:org.fusesource.fabric/fuse-fabric/7.0.1.fuse-084/xml/features
Features :
fabric-agent
karaf
fabric-jaas
fabric-core
Agent Properties :
patch.repositories = http://repo.fusesource.com/nexus/content/repositories/releases,
http://repo.fusesource.com/nexus/content/groups/ea
org.ops4j.pax.url.mvn.repositories = http://repo1.maven.org/maven2,
http://repo.fusesource.com/nexus/content/repositories/releases,
http://repo.fusesource.com/nexus/content/groups/ea,
http://repository.springsource.com/maven/bundles/release,
http://repository.springsource.com/maven/bundles/external,
http://scala-tools.org/repo-releases
org.ops4j.pax.url.mvn.defaultRepositories = file:${karaf.home}/${karaf.default.repository}#snapshots,
file:${karaf.home}/local-repo#snapshots
Configuration details
----------------------------
PID:
PID: org.ops4j.pax.url.mvn
org.ops4j.pax.url.mvn.useFallbackRepositories false
org.ops4j.pax.url.mvn.disableAether true
org.ops4j.pax.url.mvn.repositories ${profile:org.fusesource.fabric.agent/org.ops4j.pax.url.mvn.repositories}
org.ops4j.pax.url.mvn.defaultRepositories ${profile:org.fusesource.fabric.agent/org.ops4j.pax.url.mvn.defaultRepositories}
PID: patch.repositories=http:
PID: org.ops4j.pax.url.mvn.repositories=http:
PID: http:
PID: org.fusesource.fabric.zookeeper
zookeeper.url ${zk:root/ip}:2181
I'd be extremely grateful if someone could point the correct command(s).
I had a look at the command-line code for fabric:profile-edit with --delete and unfortunately this function seems to be desgined for deleting key/value pairs from the PID, rather than deleting the PID itself.
(Here's the code for ProfileEdit.java on github)
So basically you can use that command to "empty out" the PIDs, but not to remove them.
fabric:profile-edit --delete --pid mypid/mykey=myvalue myprofile
Knowing that this doesn't help you much, I asked my colleague who sits next to me (and is much smarter than me) and he recommended the following:
Enable fuse management console with container-add-profile root fmc
Opem fmc in a browser (mine is on localhost at port 8181), go to the Profiles page, choose your profile from the list
Go to the Config Files tab, find the PID you want to nuke and click the cross (X).
Et voila, the pid should be gone. Interested to know if this works for you, including on the "blank" profile...
The following works in Fuse 6.2:
1) for property files (which become PID objects)
# create
profile-edit --resource foobar.properties default
# delete
profile-edit --delete --pid foobar default
2) for arbitrary files
# create
profile-edit --resource foobar.xml default
#delete
only via hawtio web console, see screenshot:

Resources