I'm working on the Azure Function (durable function) that implements an HTTP trigger. All it does is waiting for an HTTP call from the backend that shares a link to a blob storage object (image) so it can be processed by a function. I need to implement a reliable logging solution using SEQ, that's being used for other projects in our company (mostly .NET).
Using some official documentation from here all I'm receiving in the SEQ console is a stream of unstructured events and it's hard to gain where and when the processing starts, how much time did it take, etc. It makes it impossible to troubleshoot.
With .NET projects we were using Serilog that allows you to write so-called enrichers and filters, so you can structurize the logs and the information that is really needed, including the call performance (e.g. elapsed time). I don't see anything even close to that available for Python 3. Can anyone suggest where do I start? What's the best approach to capture the events I'm looking for?
Thanks.
Ok guys, here's the answer:
You need to install the lib called seqlog via requirements.txt for the purpose
In the Python script where you plan to use logger import respective namespace, i.e. import logging
Define SEQ configuration in a JSON file (something like this):
In the init.py load SEQ config:
with open('./seq.config.json', 'r') as f:
seq_config = json.load(f)
f.close()
Use logging object to stream the logs to SEQ:
logging.info("[" + obj.status + "] >> Data has been processed!")
Enjoy the logs posted to the SEQ console
p.s. if you're debugging locally, set http://localhost+port in the seq.config.json instead of the remote console address
Hope this info will help someone.
Related
Anyone have Python CDK code to enable Amazon Kinesis Data Firehose delivery stream Logging in WAF? Any language CDK code is fine for my reference as I didn't find any proper syntax or examples to enable in official python CDK/api documentation nor in any blog.
From the existing documentation (as of CDK version 1.101 and by extension Cloudformation) there seems to be no way of doing this out of the box.
But there is API call which can be utilized with boto3 for example: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/wafv2.html#WAFV2.Client.put_logging_configuration
What you need to have in order to invoke the call:
ResourceArn of the web ACL
List of Kinesis Data Firehose ARN(s) which should receive the logs
This means that you can try using Custom Resource and implement this behavior. Given you have created Firehose and web ACL in the stack previously, use this to create Custom Resource:
https://docs.aws.amazon.com/cdk/api/latest/python/aws_cdk.custom_resources/README.html
crd_provider = custom_resources.Provider(
self,
"Custom resource provider",
on_event_handler=on_event_handler,
log_retention=aws_logs.RetentionDays.ONE_DAY
)
custom_resource = core.CustomResource(
self,
"WAF logging configurator",
service_token=crd_provider.service_token,
properties={
"ResourceArn": waf_rule.attr_arn,
"FirehoseARN": firehose.attr_arn
}
)
on_event_handler in this case is a lambda function which you need to implement.
It should be possible to simplify this further by using AwsSdkCall:
on_event_handler = AwsSdkCall(
action='PutLoggingConfiguration',
service='waf',
parameters={
'ResourceArn': waf_rule.attr_arn,
'LogDestinationConfigs': [
firehose.attr_arn,
]
)
This way you don't need to write your own lambda. But your use case might change and you might want to add some extra functionality to this logging configurator, so I'm showing both approaches.
Disclaimer: I haven't tested this exact code, rather this is an excerpt of similar code written by me to solve similar problem of circumventing the gap in Cloudformation coverage.
I don't have python CDK example, but I had it working in Typescript using CfnDeliverySteam and CfnLoggingConfiguration. I would imagine you can find matching class in python CDK.
I want to live log some logs in the project I am working, and after searching I found the Google Cloud Live Logging Api.
After searching around and reading different guides, I created a project and enabled the logging API.
Then I tried testing it with the below code.
# Imports the Google Cloud client library
import google.cloud.logging
from google.cloud import logging
from google.oauth2 import service_account
# Imports Python standard library logging
import logging
# Instantiates a client
credentials = service_account.Credentials.from_service_account_file('my_path_to_service_account_json')
client = google.cloud.logging.Client(project='name_of_my_project', credentials=credentials)
# Connects the logger to the root logging handler; by default this captures
# all logs at INFO level and higher
client.setup_logging()
# The data to log
text = 'Hello, world!'
# Emits the data using the standard logging module
logging.warning(text)
It executed successfully and I went in the Logs Viewer in google cloud to check the logs.
Nothing was there, although it showed requests in the API Overview.
Do you have any idea on what is going wrong? Is it something in the code snippet above or I missed something in the Google cloud configurations part?
Thanks in advance.
I finally managed to solve this problem.
The problem wasn't the code, but the fact that I didn't choose "Global" at "Logs Viewer".
When I chose "Global", I saw all my logs.
I've begun using AWSXRay in order to get more insight into why performance is not ideal in my lambda function. This lambda function runs a gql service meaning it has lots of outbound requests to other lambda functions as well as dynamodb for caching.
I've added tracing to all aws-sdk client calls by utilizing the following in my handler. It mutates the imported AWS module so that all subsequent usage of AWS clients successful includes aws-xray tracing, regardless of what module imports it. Awesome!
import AWS from 'aws-sdk';
import AWSXRay from 'aws-xray-sdk';
AWSXRay.captureAWS(AWS);
Heres an example of the output:
The Problem
The problem is that none of the "Traces" have any annotation regarding the parameters of the requests. Both the annotation and metadata of each trace is empty:
The Hope
The hope is that there is a way to configure the AWSXRay CaptureAWS modifications so that they include the arguments of each aws-client request in the annotations or metadata.
The Question
Is it possible to request that AWSXRay.captureAWS(AWS); includes the parameters passed to the aws sdk client invocations in either the annotations or the metadata of the traces it produces?
The resources section contains high level arguments for some clients e.g. DynamoDB table name. Not all arguments are captured by default. This is because they may contain information that the users do not wish to track in their trace and may also be verbose.
For now opt-in is not available in X-Ray SDK for arbitrary API parameters. As a workaround for now, I would suggest that you wrap your sdk calls in a local subsegment and record the parameters you want to capture as annotations or Metadata for that subsegment. Let me know if you need any help in locating docs that allow you to create your own subsegments.
I have Clarion 9 app that I want to be able to communicate with HTTP servers. I come from PHP background. I have 0 idea on what to do.
What I wish to be able to do:
Parse JSON data and convert QUEUE data to JSON [Done]
Have a global variable like 'baseURL' that points to e.g. http://localhost.com [Done]
Call functions such apiConnection.get('/users') would return me the contents of the page. [I'm stuck here]
apiConnection.post('/users', myQueueData) would POST myQueueData contents.
I tried using winhttp.dll by reading it from LibMaker but it didn't read it. Instead, I'm now using wininet.dll which LibMaker successfully created a .lib file for it.
I'm currently using the PROTOTYPE procedures from this code on GitHub https://gist.github.com/ddur/34033ed1392cdce1253c
What I did was include them like:
SimpleApi.clw
PROGRAM
INCLUDE('winInet.equ')
ApiLog QUEUE, PRE(log)
LogTitle STRING(10)
LogMessage STRING(50)
END
MAP
INCLUDE('winInetMap.clw')
END
INCLUDE('equates.clw'),ONCE
INCLUDE('DreamyConnection.inc'),ONCE
ApiConnection DreamyConnection
CODE
IF DreamyConnection.initiateConnection('http://localhost')
ELSE
log:LogTitle = 'Info'
log:LogMessage = 'Failed'
ADD(apiLog)
END
But the buffer that winInet's that uses always returns 0.
I have created a GitHub repository https://github.com/spacemudd/clarion-api with all the code to look at.
I'm really lost in this because I can't find proper documentation of Clarion.
I do not want a paid solution.
It kind of depends which version of Clarion you have.
Starting around v9 they added ClaRunExt which provides this kind of functionality via .NET Interop.
From the help:
Use HTTP or HTTPS to download web pages, or any other type of file. You can also post form data to web servers. Very easy way to send HTTP web requests (and receive responses) to Web Servers, REST Web Services, or standard Web Services, with the most commonly used HTTP verbs; POST, GET, PUT, and DELETE.
Otherwise, search the LibSrc\ directory for "http" and you will get an idea of what is already there. abapi.inc for example, appears to provide a wrapper around wininet.lib.
I want log every http request being made by a particular node app, and all of its modules. Wrapping requests in a function could work for all non-module code, the disadvantage would obviously be it doesn't include module code, and be cumbersome to do.
This is for apps already in production, only other option I thought of was tcpdump.
Setting NODE_DEBUG=http will make node log out detailed HTTP request information to the console.
Examples:
NODE_DEBUG=http,http2 node index.js
NODE_DEBUG=http,http2 npm start
For more information see:
NODE_DEBUG documentation
This blog post: Debugging tools and practices in node.js.
As of 2022-11-23, this is the list of the available NODE_DEBUG attributes (based on the blog post above and checking the nodejs source code):
child_process
cluster
esm
fs
http
http2
inspect
module
net
policy
repl
source_map
stream
stream_socket
timer
tls
tracing
worker
You can pass multiple modules as a comma-separated list: NODE_DEBUG=http,http2,tls
How to find the module IDs in nodejs source code:
Unfortunately, there isn't a listing of the available debug module IDs in the nodejs documentation.
To find the IDs in the source: search/grep the nodejs .js files for .debuglog( usage:
# Grep inside the nodejs project
$ grep -r -E --color "\.debuglog\('" nodejs/lib
This will return results like:
let debug = require('internal/util/debuglog').debuglog('esm', (fn) => {
let debug = require('internal/util/debuglog').debuglog('http2', (fn) => {
The string passed to .debuglog(...) (e.g. 'esm' and 'http2') is the module id that can be passed to NODE_DEBUG.
The easiest / least intrusive way is with a web proxy. Either an off the shelf one or one you write yourself in node. The machines the apps live on would have to get configured* to send all outbound traffic through the proxy and then the proxy can log the traffic. Details on implementation will vary based on which proxy/ approach you pick.
*Arguably, there are ways to do this such that the machines don't even know they're being proxied, but I've found in practice that's really hard to get right, especially with https traffic
In 2022, you can use grackle_tracking library. It helps you track all traffic, errors and analytics into your database or just to the console https://www.getgrackle.com/analytics_and_tracking
Check HTTP request logger called Morgan here