I'm building an ELK stack (for the first time) to track end-user REST API usage for a CloudFront distribution (in front of an S3 origin). Users pass a refresh token as part of their request and I was hoping to use this token to identify which users were making which request. Unfortunately, it looks like CloudFront access logs are missing some header information (particularly Authorization/Accept in my use case). This leaves me with three questions:
Is there a way to tell CloudFront to log additional items? It appears the answer is no.
As an alternative strategy, I tried modifying the request object with lambda#edge (in Viewer Request) to move the header information into the query string (so that it would get logged) but any manipulation in lambda#edge does not seem to be reflected in the log. (though it is reflected in the Origin Request function). Should this be possible?
If doing what I want is impossible, I think the alternative approach is forgo CloudFront logs completely and just fire an http request to logstash with every user request, but I feel like this could be easy to overload.
Thanks
After a few days of research and reaching out to Amazon, I was finally able to answer my own questions:
CloudFront logs can't be customized, they are what they are.
See 1.
It turns out that customization is the wrong approach. What I really need to do is aggregate two separate logs that have the information I need into a single logstash entry. It turns out that the Viewer Response lambda#edge function contains a requestId property (actually event.Records[0].cf.config.requestId) which matches the CloudFront log x-edge-request-id column. So while I haven't finished implementing it yet, these two columns can be used in the logstash config for aggregation. I just need to make sure I set up a Viewer Response event that logs out a consistent format that I can then part with logstash. I'm using the logstash-input-cloudwatch_logs to retrieve teh cloudwatch logs.
Related
We are developing a chatbot to handle internal and external processes for a local authority. We are trying to display contact information for a particular service from our api endpoint. The HTTP request is successful and delivers, in part, exactly what we want but there's still some unnecessary noise we can't exclude.
We specifically just want the text out of the response ("Response").
Logically, it was thought all we need to do is drill down into ${dialog.api_response.content.Response} but that fails the HTTP request and ${x.content} returns successful but includes Tags, response and the fields within 1.
Is there something simple we've missed using composer to access what we're after or do we need to change the way our endpoint is responding 2? Unfortunately the MS documentation for FrwrkComp is lacking to say the very least.
n.b. The response is currently set up as a (syntactically) SSML response, this is just a test case using an existing resource.
Response in the Emulator
Snippet from FwrkComp
Turns out it was the first thing I tried just syntactically correct. For the case of the code given it was as simple as:
${dialog.api_response.content[0].Response}
Is it possible to use cache based on a key in the request payload?
Eg. let's say we got a json or xml request payload where one of the elements is CustomerId.
Would it then be possible to cache based on CustomerId?
Thanks
I hope I understood your query properly and am not too late. I think you want to cache only when 'CustomerId' is present in the input OR it contains a certain value.
You can refer to the samples given in the foll link
https://azure.microsoft.com/en-us/blog/policy-expressions-in-azure-api-management/
It will help you to write policy expressions to check the presence or value of a particular field. Then you can cache or ignore based on that.
On a side note, Custom Caching is also something cool to check
https://learn.microsoft.com/en-us/azure/api-management/api-management-sample-cache-by-key
I recently began working with JavaScript and am looking at various get and post requests one can send to a server.
For get, as far as I know, all of the information of the query is contained in the URL that the user triggers. On the server side this has to be dissected to retrieve the necessary parameters.
I was just wondering how larger and more detailed requests are handled with this get method? For instance what if I had millions and millions of parameters that make up my whole request? Would they all be jumbled into the URL? Is there a limit as to the number of unique URLs one can have? I read this post:
How do URL shorteners guarantee unique URLs when they don't expire?
I would really like some more input.
Thank You!
I am having trouble invalidating CloudFront cached content with query string data.
For example, say I am trying to invalidate the following:
http://d114hh0cykwyb0.cloudfront.net/imagesizer.php?h=80&src=/images/foo.jpg
I use the AWS SDK to send the invalidation path, which is:
/imagesizer.php?h=80&src=/images/foo.jpg
I'll then go into the AWS Console->CloudFront->Distribution Settings->Invalidations
I see my new invalidation request. If I see the 'Details' on it, the Object Paths says:
/imagesizer.php%3Fh%3D80%26src%3D/images/foo.jpg
So the path has been encoded. However, after the invalidation has been Completed,
http://d114hh0cykwyb0.cloudfront.net/imagesizer.php?h=80&src=/images/foo.jpg
remains the same.
I also tried using the 'Create Invalidation' from within the AWS Console using the path:
/imagesizer.php?h=80&src=/images/foo.jpg
but I get an XML parse error (because the URL is not encoded).
Has anyone dealt with this kind of issue before? Were you able to find a solution?
Thanks!
a very easy way to use invalidation is the use of third party tools. Personally i use CloudBerry - > http://www.cloudberrylab.com/
I am able to find the GET parameters that are made as part of a request but I am not able to retrieve the POST parameters for a request. Can you guys tell me what should be my search parameters for the same?
Does IIS actually log this?
Thanks in advance.
IIS does not log POST parameters. POST is commonly used for large data sets and file uploads which would take up a ton of space on your disk and could cause your server to run out of space easily.
You can setup some manual logging with something like log4net and log POST parameters. File growth will still be a problem but log4net can be configured to limit growth and roll-over at a certain size. You can then index your log4net logs using splunk