I have a Node.js app running in Elastic Beanstalk and logging using console.log, console.error etc...then I have CloudWatch logs turned on. When I go to the Insights and do a query it shows up but somehow it is logging line by line instead of the entire error.
In the example screenshot I want the entire log from a single console.log to go to a single log record...so one to one, instead of splitting by new lines...is there a way to do this without removing all line breaks during console.log? Say a configuration option or something?
The output of the application is sent to standard out (stdout) and standard error (stderr). The AWS Elastic Beanstalk environment leverages Linux rsyslog to capture stdout and stderr to write the information into log files.
This is done through standard rsyslog configuration found here: /etc/rsyslog.d/web.conf
if $programname == 'web' then {
*.=warning;*.=err;*.=crit;*.=alert;*.=emerg /var/log/web.stderr.log
*.=info;*.=notice /var/log/web.stdout.log
}
It is rsyslog that interprets the stack trace from stdout as multiple entries and writes multiple lines in AWS CloudWatch Logs.
I wrote a small article on GitHub that describe the solution for a Java environment but you can do something similar for Node.JS.
Related
I'm having trouble getting Pino logs to show up in Datadog APM traces, even though, it would appear that the log injection is working fine.
So I have dd-trace all running fine, and traces and spans appearing perfectly in APM. I then hook up Pino, I have all env vars set correctly and when my Pino log outputs I can see the trace_id and span_id in the log... but under Logs in APM I see nothing.
My Pino log looks like this:
{
"level":30,
"time":1658480164226,
"pid":20400,
"hostname":"local",
"dd":{
"trace_id":"1314152611599688171",
"span_id":"6560268894829180062",
"service":"datadog-sandbox",
"version":"development",
"env":"development"
},
"foo":"bar",
"msg":"How am I doing?"
}
As you can see, the trace_id and span_id have been injected in to the log. But when I look at this trace and span in APM I see no logs connected at all:
Am I missing some configuration here? I'm happy to supply any other code if that helps.
Thanks
dd-trace is just injecting the tracing context into the logs in order to correlate between traces and logs. It does not send the logs anywhere on its own.
Logs sending can be achieved by Datadog Agent or directly via HTTP. I was able to make it work with Agentless logging
I'm trying to use cloud run for receiving millions of log messages through HTTPS (streaming) and sending them to cloud logging.
But I found there is some loss of data, the number of messages in cloud logging is less than cloud run receiving.
This is the sample code that I tried,
# unzip data
data = gzip.decompress(request.data)
# split by lines
logs = data.decode('UTF-8').split('\n')
# output the logs
log_cnt = 0
for log in logs:
try:
# output to jsonPayload
print(json.dumps(json.loads(log_str))
log_cnt += 1
except Exception as e:
logging.error(F"messsage: {str(e)}")
and If I compare the log_cnt and number of logs in cloud logging, the log_cnt is more. So some print is not finishing the delivering message.
I tried using logging API instead of print(), but the number of logs is too many for sending using logging API (12,000 calls limit for a minute), so it causes the latency very bad, and could not handle requests stably.
I dought the moving number of instances might cause it, so I test when the active instance is not changed, but still, 3-5% of messages are missing.
Is there something I can do for sending all of the messages to cloud logging without any loss?
(updated)
the line of data looks like this, (around 1kb)
{"key1": "ABCDEFGHIJKLMN","key2": "ABCDEFGHIJKLMN","key3": "ABCDEFGHIJKLMN","key4": "ABCDEFGHIJKLMN","key5": "ABCDEFGHIJKLMN","key6": "ABCDEFGHIJKLMN","key7": "ABCDEFGHIJKLMN","key8": "ABCDEFGHIJKLMN","key9": "ABCDEFGHIJKLMN","key10": "ABCDEFGHIJKLMN","key11": "ABCDEFGHIJKLMN","key12": "ABCDEFGHIJKLMN","key13": "ABCDEFGHIJKLMN","key14": "ABCDEFGHIJKLMN","key15": "ABCDEFGHIJKLMN","key16": "ABCDEFGHIJKLMN","key17": "ABCDEFGHIJKLMN","key18": "ABCDEFGHIJKLMN","key19": "ABCDEFGHIJKLMN","key20": "ABCDEFGHIJKLMN","key21": "ABCDEFGHIJKLMN","key22": "ABCDEFGHIJKLMN","key23": "ABCDEFGHIJKLMN","key24": "ABCDEFGHIJKLMN","key26": "ABCDEFGHIJKLMN","key27": "ABCDEFGHIJKLMN","key28": "ABCDEFGHIJKLMN","key29": "ABCDEFGHIJKLMN","key30": "ABCDEFGHIJKLMN","key31": "ABCDEFGHIJKLMN","key32": "ABCDEFGHIJKLMN","key33": "ABCDEFGHIJKLMN","key34": "ABCDEFGHIJKLMN","key35": "ABCDEFGHIJKLMN"}
I can recommend you to use the entries.write API that allow you to write a bulk of entries in the same time. Have a look on my test in the API explorer
It could solve the JSON format and the multiple write in the same time. Have a try with this API and let me know if it's better for you. If not, I will remove this answer!
I decided to post this question because, I have ran out of debugging ideas, just ideas are golden since I know it can be difficult to help debugging a virtual instance through here (debugging code is hard enough jaja). Anyway, I have created a virtual machine in Compute engine , I created a logs file that I populate, for example, with this command in a python script, let's call it logging.py:
import logging
logging.basicConfig(filename= 'app.log' , level = logging.INFO , format = ' %(asctime)s - %(name) - %(levelname)s - %(message)s')
logging.info('Some message ' + str(type(variable)))
everytime I use python3 logging.py , the app.log is effectively populated. ( Logging.py and app.log are in the same directory the /home/username/ folder )
I want stackdriver to show this log in the logging viewer everytime it's written, so , I installed the stackdriver agent as follows, in the virtual machine command line:
$ curl -sSO https://dl.google.com/cloudagents/install-logging-agent.sh
$ sudo bash install-logging-agent.sh
No errors that I see are delivered here, in fact, you can see here the messages obtained
Messags on the stackdriver viewer:
After this, I proceed to create a .conf file that I create in /etc/google-fluentd/config.d/app.conf
with this parameters
<source>
type tail
format none
path /home/username/app.log
pos_file /var/lib/google-fluentd/pos/app.pos
read_from_head true
tag whatever-tag
</source>
After that is created, I launch sudo service google-fluentd restart.
Aftert I execute, python3 logging.py , no logs are added to stack drivers logging viewer.
So, where might Have I gone wrong?
Things I have tried/checked:
-Have more than 13 gygabytes of RAM available
-If I run logger "some message" on the command line, I effectively add a log with "some message" to the log viewer
-If I run
ps ax | grep fluentd
I obtain :
3033 ? Sl 0:09 /opt/google-fluentd/embedded/bin/ruby /usr/sbin/google-fluentd --log /var/log/google-fluentd/google-fluentd.log --no-supervisor
3309 pts/0 S+ 0:00 grep --color=auto fluentd
-Both my user, and the service account I use, have logger admin permission in IAM roles.
-This is the documentation I have based myself on:
https://cloud.google.com/logging/docs/agent/troubleshooting?hl=es-419
https://cloud.google.com/logging/docs/reference/v2/rest/v2/entries/list?hl=es-419
https://cloud.google.com/logging/docs/agent/configuration?hl=es-419
https://medium.com/google-cloud/how-to-log-your-application-on-google-compute-engine-6600d81e70e3
https://cloud.google.com/logging/docs/agent/installation
-If I run sudo service google-fluentd status , the agent appears active.
-My instance hass access, to all the apis. It's an n1-standard-4 (4 vCPUs, 15 GB of memory) using ubuntu linux 18:04
So, what else can I check to debug this? I'm out of ideas here , hope I'm not being an idiot here :(
Based on my understanding, I think that you looking for the following fluentd resource types:
generic_node
“A generic node identifies a machine or other computational resource for which no more specific resource type is applicable. The label values must uniquely identify the node.”
generic_task
“A generic task identifies an application process for which no more specific resource is applicable, such as a process scheduled by a custom orchestration system. The label values must uniquely identify the task.”
The source of my information has been found here
This document explain how to send logs from your application in different ways:
Cloud Logging API
Cloud Logging Agent
Generic fluentd
As you mentioned having installed fluentd, let me provide more focused documentation about Cloud Logging Agent. I also found some python Client Library documentation that you may be interested.
Finally, I found a nginx/apache use-case guide that you may use as reference.
For some reason, if I change the directory to which both the .conf file points, and the directory where the logg is to /var/logs/ , being the final path as /var/logs/app.logs, it does work correctly. Possibly there is a configuration issue, causing the logging agent to only capture logs in specific predetermined folders, or a permissions issue that stops it from working if the log is in the username directory.
I found this solution, however, by chance(random testing basically.
). Did not find anything in the main articles that are supposed to teach me how to configure the logging agent, that could point me in the right direction, being those articles this ones,
https://cloud.google.com/logging/docs/agent/troubleshooting?hl=es-419 https://cloud.google.com/logging/docs/reference/v2/rest/v2/entries/list?hl=es-419 https://cloud.google.com/logging/docs/agent/configuration?hl=es-419 https://medium.com/google-cloud/how-to-log-your-application-on-google-compute-engine-6600d81e70e3 https://cloud.google.com/logging/docs/agent/installation
If I needed it to work in my username directory, it's not clear just by checking this articles how to do it,what configuration file I would need to change or where to start, so I recommend to google to improve that aspect of the docs.
This documentation you have sent https://docs.fluentd.org/quickstart is pretty interesting, maybe I can find the explanation there, thank you for your help.
What is the recommended way to implement logging levels for aws lambda functions in nodejs. I was going through many third party libraries e.g winston, winston cloudwatch, logplease, but it seems like we can also achieve using the native console. e.g
console.log(), console.error(), console.warn(), console.info()
Any recommendations?
The relevant code is here:
https://github.com/aws/aws-lambda-nodejs-runtime-interface-client/blob/a850fd5adad5f32251350ce23ca2c8934b2fa542/src/utils/LogPatch.ts#L69-L89
So, you can use 7 console methods to get 6 CloudWatch log levels:
FATAL: console.fatal()
ERROR: console.error()
WARN: console.warn()
INFO: console.info() or console.log()
DEBUG: console.debug()
TRACE: console.trace()
console.trace() doesn't produce the same trace it produces in plain Node 14
console.fatal() is missing in plain Node 14, it's added by the AWS Lambda Runtime
This was tested with the Node 14.x runtime on Aug 25 2022. YMMV.
Since the Lambda console output goes directly into CloudWatch Logs, you really don't need to use something like Winston CloudWatch if that is your preferred log destination. If you wanted to send the logs somewhere else like Loggly then you might want to use something like Winston Loggly.
However, even if you just want to send all console output to CloudWatch Logs, I would still recommend using a basic Winston configuration, so that you could quickly and easily enable debug logging, for example through an environment variable, and then turn off debug logging once you are ready to use the Lambda function in production.
I have run in to a weird bug and don't know how to proceed/debug. I have an app that's written in Nodejs and uses Winston for logging. Everything was working fine until I brought up a new production server yesterday and retired the old one.
My prod server has 4 Nodejs processes running. On the new production server, Winston logs the very first log message per .js file, period. It stops logging after that, and changing the log level doesn't work. My app has got about 6 .js files, and in case of any error on any of those files, the very first error message gets logged but any subsequent errors/warning/info are not logged.
The funny thing is Winston was working just fine on the old prod server and the dev server still works fine.
I am on Winston 0.6.2 on both dev and prod. As far as I know, all the sw packages are the same between dev and prod.
How can I debug this issue?
After some research, I came across this issue => https://github.com/flatiron/winston/issues/227
Looks like, the new way of handling streams in the latest version of node has broken file transport in winston. I am going back to node v0.8.22 for the time being as a work around.
What transports are you using for logging? Does the console transport work? Perhaps the new production server has a network issue that prevents it logging to a remote service such as CouchDB or Loggly.
If you add a simple console.log('...') line next to your winston log lines do those get fired. This will confirm or deny that your winston log lines are getting called on the production server
winston.info('winston log test')
console.log('console log test')
You can can expose the logger instance and have a URL to trigger the required log level.
I had the same need so I came up with a dynamic log level setter for Winston https://github.com/yannvr/Winston-dynamic-loglevel