I am able to find the GET parameters that are made as part of a request but I am not able to retrieve the POST parameters for a request. Can you guys tell me what should be my search parameters for the same?
Does IIS actually log this?
Thanks in advance.
IIS does not log POST parameters. POST is commonly used for large data sets and file uploads which would take up a ton of space on your disk and could cause your server to run out of space easily.
You can setup some manual logging with something like log4net and log POST parameters. File growth will still be a problem but log4net can be configured to limit growth and roll-over at a certain size. You can then index your log4net logs using splunk
Related
I have a node.js application and I am using Application Insights to collect telemetry on our users. We are using the applicationinsights npm package.
Our users' privacy is very important to us and we want to collect as little data about them as we need. To this end, we do not want to collect location data (country, state/province, and client-ip). However, I can't see how we can avoid sending that data to azure. Is it possible to avoid sending it?
I'm guessing that the location data is coming directly from the http request. So, it might be that I need to change something in the npm package to remove the location headers from the request, but this does not appear to be exposed to the application.
Any ideas on how to fix this?
As Matt mentioned, you can change the data being sent to App Insights- this is certainly necessary in cases where the default logging contains information you don't want sent to our servers for any reason. The only thing I would adjust from his suggestion is the it is recommended to use TelemetryInitializers instead of TelemetryProcessors to do this modification. Any part of the data model should be able to be adjusted or removed from an initializer. This is also particularly useful if there is anything in the request data that you would consider PII. You can see the non-custom data model here: https://learn.microsoft.com/en-us/azure/azure-monitor/app/export-data-model
All location data in App Insights is based on IP address. The IP address is sent to App Insights and from there is is processed using the GeoLite2 database. Once the conversion happens, we discard the IP address so that we are never keeping IP addresses permanently. That's why when you query your logs the IP address field will always contain 0.0.0.0 for all records.
I'm trying to log the total size of a request sent out from an ASP.NET Core website hosted in Azure. The goal is to be able to attribute some sense of what the data out cost is for specific functionality within the application. (Specifically we are looking at some functionality that uses Azure Blob Storage and allows downloads of the blobs but I don't think that's relevant to the question.)
I think the solution is some simple middleware that logs out the sizes to a storage mechanism (less concerned with that part) but not sure what to put inside the middleware.
Does anyone know how you would "construct" the total size of the response being sent out of the web app.?
I would presume it will include HttpContext.Response.Body.Length but I'm pretty confident that doesn't include the headers. Also not sure if that's the compressed size of the response body or not.
Thanks
I'm building an ELK stack (for the first time) to track end-user REST API usage for a CloudFront distribution (in front of an S3 origin). Users pass a refresh token as part of their request and I was hoping to use this token to identify which users were making which request. Unfortunately, it looks like CloudFront access logs are missing some header information (particularly Authorization/Accept in my use case). This leaves me with three questions:
Is there a way to tell CloudFront to log additional items? It appears the answer is no.
As an alternative strategy, I tried modifying the request object with lambda#edge (in Viewer Request) to move the header information into the query string (so that it would get logged) but any manipulation in lambda#edge does not seem to be reflected in the log. (though it is reflected in the Origin Request function). Should this be possible?
If doing what I want is impossible, I think the alternative approach is forgo CloudFront logs completely and just fire an http request to logstash with every user request, but I feel like this could be easy to overload.
Thanks
After a few days of research and reaching out to Amazon, I was finally able to answer my own questions:
CloudFront logs can't be customized, they are what they are.
See 1.
It turns out that customization is the wrong approach. What I really need to do is aggregate two separate logs that have the information I need into a single logstash entry. It turns out that the Viewer Response lambda#edge function contains a requestId property (actually event.Records[0].cf.config.requestId) which matches the CloudFront log x-edge-request-id column. So while I haven't finished implementing it yet, these two columns can be used in the logstash config for aggregation. I just need to make sure I set up a Viewer Response event that logs out a consistent format that I can then part with logstash. I'm using the logstash-input-cloudwatch_logs to retrieve teh cloudwatch logs.
I'm working with a production system that has a moderate amount of load. The amount of trace events and AI sends up is way too detailed, and makes it difficult to wade through logs later.
Each request to the server has information such as:
Message='Selected formatter='JsonMediaTypeFormatter', content-type='application/json; charset=utf-8'', Operation=DefaultContentNegotiator.Negotiate
and
Message='Action returned 'RZ.API.Support.Controllers.OperationActionResult`1[System.Collections.Generic.List`1[RZ.Entity.System.ClientMessage]]'', Operation=ReflectedHttpActionDescriptor.ExecuteAsync
There are maybe 30 entries for each request!
I just need the request type:
12/16/2015, 9:17:29 AM - REQUEST
GET /api/v1/user/messages
And the result code - as well as any custom stuff I do along the way.
So basically I want to trim most the traces except the request and the result (and any errors etc).
I have my eye on this bad boy in the AI config:
<Add Type="Microsoft.ApplicationInsights.Web.RequestTrackingTelemetryModule, Microsoft.AI.Web"/>
... but I cannot for the life of me see any doco on how to ask it to reduce the amount of stuff that is sent!
Any help is much appreciated.
Jordan.
P.S. All the extra logging has put us over the 15m a month plan, we had to upgrade!
RequestTrackingTelemetryModule does not do anything like you described. It adds requests, exceptions and dependencies collection. And in you example you are saying you see verbose WebApi traces being forwarded to ApplicationInsights. I assume you actually use Application Insights logging adapter.
Here you can read how WebApi traces can be forwarded to AI Version 1: http://apmtips.com/blog/2014/11/13/collect-asp-dot-net-mvc-web-api-traces-with-application-insights/
Here you can read how WebApi traces can be forwarded to AI Version 2:
http://apmtips.com/blog/2016/01/05/webapi-tracing-powered-by-ai-in-vs2015-update1/
Source code of logging adapters: https://github.com/Microsoft/ApplicationInsights-dotnet-logging
Documentation: https://azure.microsoft.com/en-us/documentation/articles/app-insights-search-diagnostic-logs/#trace
So you have multiple options:
Do not use logging adapters
Change verbosity of WebApi tracing (read http://www.asp.net/web-api/overview/testing-and-debugging/tracing-in-aspnet-web-api). I would prefer this one since you probably want to collect failures.
Remove WebApi tracing (as you did)
To answer my own question.
In my WebApiConfig file, I had:
config.EnableSystemDiagnosticsTracing();
Removing this line drastically cut down the clutter to what I was trying to achieve.
As of version 2.0 of the Application Insights SDKs, you can also limit the data sent by enabling sampling:
https://azure.microsoft.com/en-us/documentation/articles/app-insights-sampling/
if you add
<MaxTelemetryItemsPerSecond>5</MaxTelemetryItemsPerSecond>
to your ApplicationInsights.config, the sdk can limit how much goes out. The article above has a LOT more settings/configuration you can use to get other specific behavior, but the one above is the simplest.
As far as I know there are no configuration options available for the RequestTrackingTelemetryModule. You could just turn it off (by uninstalling the respective NuGet package or commenting the xml) and / or install different / additional telemetry modules.
See app-insights-configuration-with-applicationinsights-config for a list of modules and configuration options.
I've got an Azure app up and running, but various requests generate a 500 error. There are no other details that come back from the server to let me know exactly what the problem is. No stack trace, no error message. The only thing I get back from the server are the http headers indicating I've got an error.
I've done a little looking around but can't seem to find a way to retrieve the error details that I'm looking for. I've seen some articles that suggest that I enable logging, but I'm not sure 1) how to do that, 2) where those log files would go and 3) how to access said log files. I've seen posts that say to add a whole bunch of code to my application to enable logging, but all I'm looking for is an error message and a stack trace from a 500 error. Do I really have to add a bunch of code to my app to see that information? If not, how can I get at it?
Thanks!
Chris
The best long-term solution is to enable Azure Diagnostics, which I think is what you're referring to. If you want a quick-and-dirty solution, you can log errors out to a file and then RDP into the role instances to view them. This is very similar to what you would do on a server in your own datacenter.
You can create the logs however you like. I've used log4net and RollingFileAppenders with some success. Setting the logfile path to something like "\logs\mylog.txt" will place the logs in the E: drive of the VM. Note you'll still need code somewhere in your app to capture the error and write it to the log - typically the global error handler in Global.asax is a good place for that.
You'll also have to enable RDP access to your role instances. There are many articles detailing how to do that. Here's one.
This is not a generally recommended approach because the logs may disappears when the role recycles or is recreated. It's also a pain in the butt to log to keep an eye on all those different servers.
One other warning - it's possible that the 500 error is due to some failure in your web.config. If that is the case, all the the application-level error logging in the world isn't going to help you. So be sure that your web.config is valid, and also check the Windows Event Logs while you're RDP'd into the server.
500 internal server error is most generally caused by some problem on the server when it was not able to understand incoming requests or there was some problem in configuration. So, try to run the app locally and see if there is some problem. You can record errors in a database in catches/application_error and also can use tracing. Believe me they are very helpful and worth a few extra lines of code.
For tracing have a look here, http://msdn.microsoft.com/en-us/magazine/ff714589.aspx