when attempting to run pulsar-client consume command locally, I'm getting the error:
Failed to load an authentication plugin
This is the command I ran:
./pulsar-client consume persistent://data-log/ingest/data-log-kafka -s damo
I have a client.conf configured, and the full "stack trace" from pulsar is listed below. I can't see what I am missing
Failed to load an authentication plugin
Usage: pulsar-client [options] [command] [command options]
Options:
--auth-params
Authentication parameters, e.g., "key1:val1,key2:val2".
Default: token:eyJh##############################
--auth-plugin
Authentication plugin class name.
Default: org.apache.pulsar.client.impl.auth.AuthenticationToken
-h, --help
Show this help.
Default: false
--url
Broker URL to which to connect.
Default: pulsar+ssl://aaa-bbb-ccc-ms-tls.acme.com:6651
Commands:
consume Consume messages from a specified topic
Usage: consume [options] TopicName
Options:
--hex
Display binary messages in hex.
Default: false
-n, --num-messages
Number of messages to consume, 0 means to consume forever.
Default: 1
-r, --rate
Rate (in msg/sec) at which to consume, value 0 means to consume
messages as fast as possible.
Default: 0.0
* -s, --subscription-name
Subscription name.
-t, --subscription-type
Subscription type: Exclusive, Shared, Failover.
Default: Exclusive
Possible Values: [Exclusive, Shared, Failover]
Any tips appreciated! Thanks
Problem was I was on old version of pulsar, and works fine on 2.5.0. d'oh!!!!
Related
I have an elastic beanstalk environment, which is running a docker container that has a node js API. On the AWS Console, if I select my environment, then go to Configuration/Software I have the following:
Log groups: /aws/elasticbeanstalk/my-environment
Log streaming: Enabled
Retention: 3 days
Lifecycle: Keep after termination.
However, if I click on that log group on the Cloudwatch console, I have a Last Event Time of some weeks ago (which I believe corresponds to when the environment was created) and have no content on the logs.
Since this is a dockerized application, Logs for the server itself should be at /aws/elasticbeanstalk/my-environment/var/log/eb-docker/containers/eb-current-app/stdouterr.log.
If I instead get the Logs directly from the instances by going once again to my EB environment, clicking "Logs" and then "Request last 100 Lines" the logging is happening correctly. I just can't see a thing when using CloudWatch.
Any help is gladly appreciated
I was able to get around this problem.
So CloudWatch makes a hash based on the first line of your log file and the log stream key, and the problem is that my first line on the stdouterr.log file was actually an empty line!
After couple of days playing around and getting help from the good AWS support team, I first connected via SSH to my EC2 instance associated to the EB environment and you need to add the following line to the /etc/awslogs/config/beanstalklogs.conf file, right after the "file=/var/log/eb-docker/containers/eb-current-app/stdouterr.log" line:
file_fingerprint_lines=1-20
With these, you tell the AWS service that it should calculate the hash using lines 1 through 20 on the log file. You could change 20 for larger or smaller numbers depending on your logging content; however I don't know if there is an upper limit for the value.
After doing so, you need to restart the AWS Logs Service on the instance.
For this you would execute:
sudo service awslogs stop
sudo service awslogs start
or simpler:
sudo service awslogs restart
After these steps I started using my environment and the logging was now being properly streamed to the CloudWatch console!
However this would not work if a new deployment is made, if the EC2 instance gets replaced or the auto scalable group spawns another.
To have a fix for this, it is possible to add log config via the .ebextensions directory, at the root of your application before deploying.
I added a file called logs.config to the newly created .ebextensions directory and placed the following content:
files:
"/etc/awslogs/config/beanstalklogs.conf":
mode: "000644"
user: root
group: root
content: |
[/var/log/eb-docker/containers/eb-current-app/stdouterr.log]
log_group_name=/aws/elasticbeanstalk/EB-ENV-NAME/var/log/eb-docker/containers/eb-current-app/stdouterr.log
log_stream_name={instance_id}
file=/var/log/eb-docker/containers/eb-current-app/*stdouterr.log
file_fingerprint_lines=1-20
commands:
01_remove_eb_stream_config:
command: 'rm /etc/awslogs/config/beanstalklogs.conf.bak'
02_restart_log_agent:
command: 'service awslogs restart'
Changing of course EB-ENV-NAME by my environment name on EB.
Hope it can help someone else!
For 64 bit Amazon Linux 2 the setup is slightly different.
For the delivery of log the AWS CloudWatch Agent is installed in /opt/aws/amazon-cloudwatch-agent and the Elastic Beanstalk configuration is in /opt/aws/amazon-cloudwatch-agent/etc/beanstalk.json. It is set to log the output of the container assuming there's a file called stdouterr.log, here's a snippet of the config:
{
"file_path": "/var/log/eb-docker/containers/eb-current-app/stdouterr.log",
"log_group_name": "/aws/elasticbeanstalk/EB-ENV-NAME/var/log/eb-docker/containers/eb-current-app/stdouterr.log",
"log_stream_name": "{instance_id}"
}
However when I look for the file_path it doesn't exist, instead I have a file path that encodes the current docker container ID /var/log/eb-docker/containers/eb-current-app/eb-e4e26c0bc464-stdouterr.log.
This logfile is created by a script /opt/elasticbeanstalk/config/private/eb-docker-log-start that is started by the eb-docker-log service, the default contents of this file are:
EB_CONFIG_DOCKER_CURRENT_APP=`cat /opt/elasticbeanstalk/deployment/.aws_beanstalk.current-container-id | cut -c 1-12`
mkdir -p /var/log/eb-docker/containers/eb-current-app/
docker logs -f $EB_CONFIG_DOCKER_CURRENT_APP >> /var/log/eb-docker/containers/eb-current-app/eb-$EB_CONFIG_DOCKER_CURRENT_APP-stdouterr.log 2>&1
To temporarily fix the logging you can manually run (replacing the docker ID) and then logs will start to appear in CloudWatch:
ln -sf /var/log/eb-docker/containers/eb-current-app/eb-e4e26c0bc464-stdouterr.log /var/log/eb-docker/containers/eb-current-app/stdouterr.log
To make this permanant I added an .ebextension to fix the eb-docker-log service so it re-makes this link so create a file in your source code in .ebextensions called fix-cloudwatch-logging.config and set it's contents to:
files:
"/opt/elasticbeanstalk/config/private/eb-docker-log-start" :
mode: "000755"
owner: root
group: root
content: |
EB_CONFIG_DOCKER_CURRENT_APP=`cat /opt/elasticbeanstalk/deployment/.aws_beanstalk.current-container-id | cut -c 1-12`
mkdir -p /var/log/eb-docker/containers/eb-current-app/
ln -sf /var/log/eb-docker/containers/eb-current-app/eb-$EB_CONFIG_DOCKER_CURRENT_APP-stdouterr.log /var/log/eb-docker/containers/eb-current-app/stdouterr.log
docker logs -f $EB_CONFIG_DOCKER_CURRENT_APP >> /var/log/eb-docker/containers/eb-current-app/eb-$EB_CONFIG_DOCKER_CURRENT_APP-stdouterr.log 2>&1
commands:
fix_logging:
command: systemctl restart eb-docker-log.service
cwd: /home/ec2-user
test: "[ ! -L /var/log/eb-docker/containers/eb-current-app/stdouterr.log ] && systemctl is-active --quiet eb-docker-log"
I met a problem with docker logging and after reading a lot of sources didn't find solution: is there a way to discard messages of docker daemon in /var/log/messages and select another location?
Ok, I know that this question is quite old but I don't think it has been answered well and no correct answer has been stated.
First of all the reason why it saves messages to that particular place starts in rsyslog configuration (/etc/rsyslog.conf) with the line:
$ModLoad imjournal # provides access to the systemd journal
So, because docker saves information to systemd journal it ends at /var/log/messages.
To be able to save it to other places, you have to create a rule like the following at /etc/rsyslog.d/docker.conf.
$FileCreateMode 0644
template(name="DockerLogFileName" type="list") {
constant(value="/var/log/docker/")
property(name="syslogtag" securepath="replace" \
regex.expression="docker/\\(.*\\)\\[" regex.submatch="1")
constant(value="/docker.log")
}
if $programname == 'dockerd' then \
/var/log/docker/combined.log
if $programname == 'dockerd' then \
if $syslogtag contains 'docker/' then \
?DockerLogFileName
else
/var/log/docker/no_tag/docker.log
$FileCreateMode 0600
I found the information for this configuration here:
https://www.simulmedia.com/blog/2016/02/19/centralized-docker-logging-with-rsyslog/
Configure rsyslog to isolate the Docker logs into their own file. To do this create /etc/rsyslog.d/10-docker.conf and copy the following content into the file.
# Docker logging
daemon.* {
/var/mylog
stop
}
In summary this will write all logs for the daemon category to /var/mylog then stop processing that log entry so it isn’t written to the systems default syslog file.
According to the Docker documentation, you can specify a different driver either as a command-line argument for the docker daemon or (preferably) in the daemon.json config file. Several drivers are available, e.g. for Syslog, HTTP-based logging, ...
Update
Here's an example configuration section for Syslog (from the documentation):
{
"log-driver": "syslog",
"log-opts": {
"syslog": "udp://1.2.3.4:1111"
}
}
When I am running this command for weave network, it is showing this error.
[root#ts ~]# kubectl apply -f https://git.io/weave-kube
error validating "https://git.io/weave-kube": error validating data: [unexpected type: object, unexpected type: object, unexpected type: object, unexpected type: object]; if you choose to ignore these errors, turn validation off with --validate=false
How to resolve this?
#verma_neeraj,
Does this still and consistently happen to you?
Which Kubernetes version are you using?
What happens if you run curl https://git.io/weave-kube?
I can confirm the YAML file available at https://git.io/weave-kube successfully configures the Weave Net daemonset under Kubernetes versions 1.5.+, but I have not tried other versions.
Anticipating an issue related to Kubernetes versions, note that there is work being done to support multiple Kubernetes versions, see these two GitHub issues.
This should be available in the coming weeks.
(Disclosure: I work for Weaveworks)
Amazon offers these readymade files for sending Tomcat/Apache/nginx logs to Cloudwatch Logs, which work great.
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html
However for my purposes they only send nginx logs, which isn't really sufficient and unfortunately they also provide zero documentation on the file format. What I'm trying to achieve is to send node.js logs from my Docker application to Cloudwatch (since autoscaling makes instances come and go).
So having files like /var/log/eb-docker/containers/eb-current-app/add839a3b599-stdouterr.log to appear in Cloudwatch.
So, what I have tried so far is adapt the webrequests config from the link above:
##############################################################################
## Sends docker logs to CloudWatch Logs
##############################################################################
Mappings:
CWLogs:
ApplicationLogGroup:
LogFile: "/var/log/eb-docker/containers/eb-current-app/*-stdouterr.log"
TimestampFormat: "%Y-%m-%d %H:%M:%S"
Outputs:
ApplicationLogGroup:
Description: "The name of the Cloudwatch Logs Log Group created for this environments web server access logs. You can specify this by setting the value for the environment variable: WebRequestCWLogGroup. Please note: if you update this value, then you will need to go and clear out the old cloudwatch logs group and delete it through Cloudwatch Logs."
Value: { "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0ApplicationLogGroup"}
Resources :
AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0ApplicationLogGroup: ## Must have prefix: AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0
Type: "AWS::Logs::LogGroup"
DependsOn: AWSEBBeanstalkMetadata
DeletionPolicy: Retain ## this is required
Properties:
LogGroupName:
"Fn::GetOptionSetting":
Namespace: "aws:elasticbeanstalk:application:environment"
OptionName: ApplicationLogGroup
DefaultValue: {"Fn::Join":["-", [{ "Ref":"AWSEBEnvironmentName" }, "stdouterr"]]}
RetentionInDays: 14
## Register the files/log groups for monitoring
AWSEBAutoScalingGroup:
Metadata:
"AWS::CloudFormation::Init":
CWLogsAgentConfigSetup:
files:
## any .conf file put into /tmp/cwlogs/conf.d will be added to the cwlogs config (see cwl-agent.config)
"/tmp/cwlogs/conf.d/stdouterr.conf":
content : |
[stdouterr]
file = `{"Fn::FindInMap":["CWLogs", "ApplicationLogGroup", "LogFile"]}`
log_group_name = `{ "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0ApplicationLogGroup" }`
log_stream_name = {instance_id}
datetime_format = `{"Fn::FindInMap":["CWLogs", "ApplicationLogGroup", "TimestampFormat"]}`
mode : "000400"
owner : root
group : root
Unfortunately this doesn't seem to work. :/
Also, does anyone have any idea if logs appear at all if fe. the timestamp format is wrong? Specially important since by default exceptions don't really have timestamps, so the actual errors would just disappear.
My application log lines currently look like this:
2016-07-05 09:11:31 ::1 - GET / 200 (5.107 ms)
You can use this link to setup cloudwatch agents on your beanstalk instances (if you haven't already) - http://serebrov.github.io/html/2015-05-20-cloudwatch-setup.html.
Next - try to send the files in /var/lib/docker/containers//.json to collect your docker logs. It's where the containers stdout and stderr is written to.
Note: All firmware and models are compatible, that is why nothing is posted about it.
I've been working on this now for a few hours (reading manuals and such) so I'm not just coming here right out of the blue. I am working on a PRE-EXISTING Nagios server where there are several other existing plugins and checks running and working. Now I want to add another server there to check so I made the following modifications:
First and foremost, I added a file to /usr/local/nagios/libexec named: check_equallogic.sh. The permissions are 755, the same as all others. I have chowned to nagios:nagios and in the listing it shows the Owner as Nagios.
I then added a command to the commands.cfg file in \usr\local\nagios\etc\objects that shows the following:
# 'check_equallogic' command definition
define command{
command_name check_equallogic
command_line $USER1$/check_equallogic -H $HOSTADDRESS$ -C $ARG1$ -t $ARG2$ $ARG3$
}
Following this, I created a file named equallogic.cfg in the objects directory and it contains (more or less):
define host{
use linux-server ; Inherit default values from a template
host_name 172.16.50.11 ; The name we're giving to this device
alias EqualLogic ; A longer name associated with the device
address 172.16.50.11 ; IP address of the device
contact_groups admins
}
Check Equallogic Information
define service{
use generic-service
host_name 172.16.50.11
service_description General Information
check_command check_equallogic!public!info
}
After ensuring that permissions are okay for all files, I restart the nagios service, no errors. When I go into the WebGUI, I get the following errors AFTER the check runs:
(Return code of 127 is out of bounds - plugin may be missing)
Extra, probably unrelated problem
Furthermore, when I log into the EquilLogic server, under Audit logs I get the following error:
Level: AUDIT
Time: 26/05/2014 3:59:13 PM
Member: ps4100-1
Subsystem: agent
Event ID: 22.7.1
SNMP packet validation failed, request received from 172.16.10.11
An snmpwalk receives a timeout, whereas others succeed. I will work on importing the MIBs tomorrow. The reason why I am mentioning it is because I want to make sure that it is only a MIB issue for the SNMP. If it is, then ignore this area.
I am entirely unsure of what to do here.
This doesn't look like a MiBs issue at all.
If snmpwalk fails, your device is not configured properly for snmp or the credentials in your possession are wrong.
Furthermore, on a general note, it is bad practice to create commands definitions for untested plugins. First you need to make sure that your plugin works from the command line, then you add it to Nagios' config.
Since I don't see this essential step in what you wrote, i will assume you didn't test the plugin.
If the plugin does not work and you need help with that please open a new question.