I'm wondering if there is any way of hijacking the standard "Task timed out after 1.00 seconds" log.
Bit of context : I'm streaming lambda function logs into AWS Elasticsearch / Kibana, and one of the things I'm logging is whether or not the function successfully executed (good to know). I've set up a test stream to ES, and I've been able to define a pattern to map what I'm logging to fields in ES.
From the function, I console log something like:
"\"FAIL\"\"Something messed up\"\"0.100 seconds\""
and with the mapping, I get a log structure like:
Status - Message -------------------- Execution Time
FAIL ---- Something messed up --- 0.100 seconds
... Which is lovely. However if a log comes in like:
"Task timed out after 1.00 seconds"
then the mapping will obviously not apply. If it's picked up by ES it will likely dump the whole string into "Status", which is not ideal.
I thought perhaps I could query context.getRemainingMillis() and if it goes maybe within 10 ms of the max execution time (which you can't get from the context object??) then fire the custom log and ignore the default output. This however feels like a hack.
Does anyone have much experience with logging from AWS Lambda into ES? The key to creating these custom logs with status etc is so that we can monitor the activity of the lambda functions (many), and the default log formats don't allow us to classify the result of the function.
**** EDIT ****
The solution I went with was to modify the lambda function generated by AWS for sending log lines to Elasticsearch. It would be nice if I could interface with AWS's lambda logger to set the log format, however for now this will do!
I'll share a couple key points about this:
The work done for parsing the line and setting the custom fields is done in transform() before the call to buildSource().
The message itself (full log line) is found in logEvent.message.
You don't just reassign the message in the desired format (in fact leaving it be is probably best since the raw line is sent to ES). The key here is to set the custom fields in logEvent.extractedFields. So once I've ripped apart the log line, I set logEvent.extractedFields.status = "FAIL", logEvent.extractedFields.message = "Whoops.", etc etc.
Related
I have an almost-default installation of Auditbeat on several of my hosts, that are also auditing changes of /etc, that forward log data to a Logstash instance elsewhere. I want to generate a message based on these logs, as by default Auditbeat does not fill the message field with value (they moved it to event.original and anyway it's disabled, and I want to be as close to production as possible with my configs), so that Kibana displays "failed to find message" when I try viewing logs from auditbeat-*. So I went to parsing and adding fields to events with Logstash.
I have encountered an interesting issue: If I query something that belongs to any custom tree under root in JSON but event, Logstash filters work, but should I decide to query [event][type], the result is always false. The problem is, if I just stuff "%{[event][type]}" into my message, the value is in there! I have tried if ([event][type] == "info") {...}, if ([type] == "info") and also tried if ([event][action] == "change") to no avail, while when I do output a debug message value with "%{[event][type]} %{[event][action]}" both values are present and equal to whatever I'm comparing to. Note that [event][module] filter actually works, so this behavior with [event][type] really baffles me.
So, how to filter based on [event][type] in Logstash, provided they are present in incoming data?
The answer was pretty simple. Both event.type and event.action are arrays and not strings, so comparing an array to a string returned false. The proper way of filtering through these is using "in", like this:
if "info" in [event][type] {...}
I am preparing some logging using python, but whenever I run my code it generates the logs but shows twice on stackdriver console (one as info and one as error). Anyone have idea on how to deal with this problem.
my code:
import logging
from google.cloud import logging as gcp_logging
log_client = gcp_logging.Client()
log_client.setup_logging()
# here executing some bigquery operations
logging.info("Query result loaded into temporary table: {}".format(temporary_table))
# here executing some bigquery operations
logging.error("Query executed with empty result set.")
When I run above code it show above logs twice on stackdriver.
Info:2019-10-17T11:54:02.504Z cf-mycloudfunctionname Query result loaded into temporary table: mytable
Error:2019-10-17T11:54:02.505Z cf-mycloudfunctionname Query result loaded into temporary table: mytable
What I can see is that both (error & info) has been recognized as a plane text, so it sent the same message as info for stderr and stdout, this is why you are getting two same messages.
What you need to do is to correct phrase the this two logs into an structured JSON, this to be recognized by stackdriver as one entity with the correct payload to be displayed.
Additional, you can configure the stackdriver agent to send the logs as you need to be sent, take a look to this document.
Also It will depend from where you are trying to retrieve this logs GCE, GKE, BQ. In some cases its preferible to change the structure of fluentd directly instead of the stackdriver agent.
I burned a couple of hours on a problem today and thought I would share.
I tried to start up a previously-working Azure Stream Analytics job and was greeted by a quick failure:
Failed to start Streaming Job 'shayward10ProcessLogs'.
I looked at the JSON log and found nothing helpful whatsoever. The only description of the problem was:
Stream Analytics job has validation errors: The given key was not present in the dictionary.
Given the error and some changes to our database, I tried the following to no effect:
Deleting and Recreating all Inputs
Deleting and Recreating all Outputs
Running tests against the data (coming from Event Hub) and the output looked good
My query looked as followed:
SELECT
dateTimeUtc,
context.tenantId AS tenantId,
context.userId AS userId,
context.deviceId AS deviceId,
changeType,
dataType,
changeStatus,
failureReason,
ipAddress,
UDF.JsonToString(details) AS details
INTO
[MyOutput]
FROM
[MyInput]
WHERE
logType = 'MyLogType';
Nothing made sense so I started deconstructing my query. I took it down to a single field and it succeeded. I went field by field, trying to figure out which field (if any) was the cause.
See my answer below.
The answer was simple (yet frustrating). When I got to the final field, that's where the failure was:
UDF.JsonToString(details) AS details
This was the only field that used a user-defined function. After futsing around, I noticed that the Function Editor showed the title of the function as:
udf.JsonToString
It was a casing issue. I had UDF in UPPERCASE and Azure Stream Analytics expected it in lowercase. I changed my final field to:
udf.JsonToString(details) AS details
It worked.
The strange thing is, it was previously working. Microsoft may have made a change to Azure Stream Analytics to make it case-sensitive in a place where it seemingly wasn't before.
It makes sense, though. JavaScript is case-sensitive. Every JavaScript object is basically a dictionary of members. Consider the error:
Stream Analytics job has validation errors: The given key was not present in the dictionary.
The "udf" object had a dictionary member with my function in it. The UDF object would be undefined. Undefined doesn't have my function as a member.
I hope my 2-hour head-banging session helps someone else.
We are trying to log some lengthy message using AppInsights trackEvent() message. But it is not logging into AppInsights and not giving any error.
Please help me in logging lengthy string.
Please let us know the max limit for the trackEvent()
if you want to log messages then you should be using the trackTrace methods of the AI SDK, not trackEvent. trackTrace is intended for long messages and has a huge limit: (32k!) See https://github.com/Microsoft/ApplicationInsights-dotnet/blob/develop/Schema/PublicSchema/MessageData.bond#L13
trackEvent is intended for named "events" like "opened file" or "clicked retry" or "canceled frobulating", where you might want to make charts, and track usage of a thing over time.
you can attach custom properties (string key, string value) and custom metrics (string key, double value) to anything. and if you set the operationId field on things in the sdk, anything with the same operationId can be easily found together via queries or visualized in the Azure Portal or in Visual Studio:
There are indeed limitation regarding the length. For example, the limit of the Name property of an event is 512 characters. See https://github.com/Microsoft/ApplicationInsights-dotnet/blob/master/src/Core/Managed/Shared/Extensibility/Implementation/Property.cs#L23
You can split it on substrings and put in Properties collection, each collection value length is 8 * 1024. I got this as a tip when I asked for it. See https://social.msdn.microsoft.com/Forums/en-US/84bd5ade-0b21-47cc-9b39-c6c7a292d87e/dependencytelemetry-sql-command-gets-truncated?forum=ApplicationInsights. Never tried it myself though
I am new in BPEL writing. I have realized the simple process below:
receive1
|
|
invoke1
|
|
receive2
|
|
invoke2
The problem is that the process correctly runs till to the "receive2" but when I invoke, via soapUI, the operation associated to the "receive2" nothing happens. I have read other posts about BPEL but nothing matching with this question. Below the real activities (I omitted the Assign ones) involved.
<bpel:receive name="receiveInput" partnerLink="client"
portType="tns:HealthMobility"
operation="initiate" variable="input"
createInstance="yes"/>
<bpel:invoke name="getTreatmentOptions"
partnerLink="treatmentProviderPL" operation="getTreatmentOptions"
inputVariable="getTreatmentOptionsReq" outputVariable="getTreatmentOptionsResp">
</bpel:invoke>
<bpel:receive name="bookMobility" partnerLink="client" operation="bookMobility"
variable="bookMobilityReq" portType="tns:HealthMobility"/>
<bpel:invoke name="getTripOptions" partnerLink="mobilityMultiProvidersPL"
operation="getTripOptions" inputVariable="getTripOptionsReq"
outputVariable="getTripOptionsResp"></bpel:invoke>
I have tried to make debugging simply by deleting the receive and initializing statically the input variable required by the getTriOptions invoke. In this case all works fine so it means, necessarly, that the process continue to wait on the receive also if I invoke bookMobility via SOAPUI. My question is: Why? I'm missing something?
Thanks
You need to define a correlation set for the second receive. Each message that is sent to the operation that is connected to the first receive activity will create a new process instance. This means you may have multiple instance running in parallel. When these instances have reached the second receive, they are waiting for the second message, but in your example, there are no means to distinguish, which message is targeted to which process instance. I assume that your BPEL engine also logged that it could not route the message to a target instance.
To solve this problem, you need to find an identifier in the payload of a message and initialize a correlation set with this value. Then, when using the same correlation set with the second receive, all messages that contain the same identifier will be routed to this particular process instance. For further information about correlation sets I recommend reading the BPEL primer, section 4.2.4.