I'm struggling to enable automatic log collection for #azure/arm SDKs. Currently, I'm using #azure/arm-keyvault, #azure/arm-network and #azure/arm-resources. Is there some extra configuration needed? I thought HTTP logs are collected automatically.
#microsoft/microsoft-graph-client, #azure/cosmos and lots of other dependency logs are collected flawlessly. I use node 18LTS and tested 16LTS. applicationinsights package version is 2.3.6
appInsights
.setup(configService.get("APPLICATIONINSIGHTS_CONNECTION_STRING"))
.setDistributedTracingMode(DistributedTracingModes.AI_AND_W3C)
.setAutoCollectConsole(false, false)
.setAutoCollectExceptions(true)
.setAutoCollectPerformance(true, true)
.setAutoCollectPreAggregatedMetrics(true)
.setAutoCollectHeartbeat(true)
.setAutoCollectRequests(true)
.setAutoCollectDependencies(true)
.setAutoDependencyCorrelation(true)
.setUseDiskRetryCaching(true)
.setInternalLogging(false, true)
.setSendLiveMetrics(true)
.start();
In this screenshot one can see the missing dependency logs. The health requests takes 3 second and there are no correlated logs for the last 2 seconds, when arm SDKs are making requests. The logs are missing in the transactions list as well.
Originally posted on AI repo: https://github.com/microsoft/ApplicationInsights-node.js/issues/1055
Related
I am generating logs for my client application where there is very limited internet connectivity. I am storing the offline logs and generating it to application insights once the user is back online. The problem I am facing is out of all the logs only request logs are coming rest are getting discarded. This is happening because of sampling even though I have already disabled the sampling from Startup.cs. Here is my code:
var aiOptions = new Microsoft.ApplicationInsights.AspNetCore.Extensions.ApplicationInsightsServiceOptions();
aiOptions.EnableAdaptiveSampling = false;
services.AddApplicationInsightsTelemetry(aiOptions);
Any Suggestions how to completely remove the sampling so that I can have all the logs in application insight.
Check this document to see different log levels. if you have latest version of sdk than ILogger Can capture without required action.
It will capture log level.
Here is the configuration of logging level.
.ConfigureLogging(
builder =>
{
builder.AddApplicationInsights("ikey");
builder.AddFilter<Microsoft.Extensions.Logging.ApplicationInsights.ApplicationInsightsLoggerProvider>("", LogLevel.Information); // this will capture Info level traces and above.
}
For complete information check this SO thread.
I'm working on upgrading our service to use 3.63.0 (upgrading from 3.57.0) and I've noticed the following warning (with stack trace) shows up in the logs that wasn't there on the previous version:
2022-02-18 14:03:41.038 WARN 1088 --- [ main] c.s.c.s.c.c.AbstractHttpClientCache : Could not get HttpClient cache.
com.sap.cloud.sdk.cloudplatform.thread.exception.ThreadContextAccessException: No ThreadContext available for thread id=1.
at com.sap.cloud.sdk.cloudplatform.thread.ThreadLocalThreadContextFacade.lambda$tryGetCurrentContext$0(ThreadLocalThreadContextFacade.java:39) ~[cloudplatform-core-3.63.0.jar:na]
at io.vavr.Value.toTry(Value.java:1414) ~[vavr-0.10.4.jar:na]
at com.sap.cloud.sdk.cloudplatform.thread.ThreadLocalThreadContextFacade.tryGetCurrentContext(ThreadLocalThreadContextFacade.java:37) ~[cloudplatform-core-3.63.0.jar:na]
at io.vavr.control.Try.flatMapTry(Try.java:490) ~[vavr-0.10.4.jar:na]
at io.vavr.control.Try.flatMap(Try.java:472) ~[vavr-0.10.4.jar:na]
at com.sap.cloud.sdk.cloudplatform.thread.ThreadContextAccessor.tryGetCurrentContext(ThreadContextAccessor.java:84) ~[cloudplatform-core-3.63.0.jar:na]
at com.sap.cloud.sdk.cloudplatform.connectivity.RequestScopedHttpClientCache.getCache(RequestScopedHttpClientCache.java:28) ~[cloudplatform-connectivity-3.63.0.jar:na]
at com.sap.cloud.sdk.cloudplatform.connectivity.AbstractHttpClientCache.tryGetOrCreateHttpClient(AbstractHttpClientCache.java:78) ~[cloudplatform-connectivity-3.63.0.jar:na]
at com.sap.cloud.sdk.cloudplatform.connectivity.AbstractHttpClientCache.tryGetHttpClient(AbstractHttpClientCache.java:46) ~[cloudplatform-connectivity-3.63.0.jar:na]
at com.sap.cloud.sdk.cloudplatform.connectivity.HttpClientAccessor.tryGetHttpClient(HttpClientAccessor.java:153) ~[cloudplatform-connectivity-3.63.0.jar:na]
at com.sap.cloud.sdk.cloudplatform.connectivity.HttpClientAccessor.getHttpClient(HttpClientAccessor.java:131) ~[cloudplatform-connectivity-3.63.0.jar:na]
at com.octanner.mca.service.MarketingCloudApiContactService.uploadContacts(MarketingCloudApiContactService.java:138) ~[classes/:na]
...
This happens when the following calls are made...
Using the lower level API
HttpClient httpClient = HttpClientAccessor.getHttpClient(destination); // warning happens here
ODataRequestResultMultipartGeneric batchResult = requestBatch.execute(httpClient);
Using the higher level API
service
.getAllContactOriginData()
.withQueryParameter("$expand", "AdditionalIDs")
.top(size)
.filter(filter)
.executeRequest(destination)); // warning happens here
Even though this warning shows up in the logs the service requests do continue to work as expected. It's just a little concerning to see this and I'm wondering if maybe I have something misconfigured. I reviewed all of the java docs and the troubleshooting page and didn't see anything out of the ordinary other than how I am fetching my destination, but even using the DestinationAccessor didn't seem to make a difference. Also, I'm not doing any asynchronous or multi-tenant processing.
Any help you or guidance you can give on this would be appreciated!
Cheers!
Such an issue is often the result of missing Spring Boot annotations - especially in synchronous executions.
Please refer to our documentation to learn more about the SAP Cloud SDK Spring Boot integration.
Edit Feb. 28th 2022
It is safe to ignore the logged warning if your application does not need any of the SAP Cloud SDK's multitenancy features.
Error Cause
The SAP Cloud SDK for Java recently (in version 3.63.0) introduced a change to the thread propagation behavior of the HttpClientCache.
With that change, we also adapted the logging in case the propagation didn't work as expected - this is often caused by not using the ThreadContextExecutor for wrapping asynchronous operations.
This is the reason for logs like the one described by the issue author.
Planned Mitigation
In the meanwhile, we realized that these WARN logs are causing confusion on the consumer side.
We are working on improving the situation by degrading the log level to INFO for the message and to DEBUG for the exception.
I have an Azure Function App with multiples functions connected with Application Insights.
For some reason that I don't know sometimes, some requests and traces get lost and it's like they never happen, but I can see the data in our DB and also in others systems.
Here is a new function with just one call, in the azure function dashboard I can see the log:
But in Application Insights, when I try to search for the logs of the trace or the request, there is not info retrived.
This's not happening everytime, but there's not the first time I saw this issue. I can see the logs for others requests but I don't know why sometimes logs are lost.
Azure function info:
Runtime Version: 3
Stack: NodeJS
Have you configured sampling? This can appear as data loss.
You can control it as follows, as per the documentation:
const appInsights = require("applicationinsights");
appInsights.setup("<instrumentation_key>");
appInsights.defaultClient.config.samplingPercentage = 33; // 33% of all telemetry will be sent to Application Insights
appInsights.start();
I have a node server running on Google Cloud Run. Now I want to enable stackdriver tracing. When I run the service locally, I am able to get the traces in the GCP. However, when I run the service as Google Cloud Run, I am getting an an error:
"#google-cloud/trace-agent ERROR TraceWriter#publish: Received error with status code 403 while publishing traces to cloudtrace.googleapis.com: Error: The request is missing a valid API key."
I made sure that the service account has tracing agent role.
First line in my app.js
require('#google-cloud/trace-agent').start();
running locally I am using .env file containing
GOOGLE_APPLICATION_CREDENTIALS=<path to credentials.json>
According to https://github.com/googleapis/cloud-trace-nodejs These values are auto-detected if the application is running on Google Cloud Platform so, I don't have this credentials on the gcp image
There are two challenges to using this library with Cloud Run:
Despite the note about auto-detection, Cloud Run is an exception. It is not yet autodetected. This can be addressed for now with some explicit configuration.
Because Cloud Run services only have resources until they respond to a request, queued up trace data may not be sent before CPU resources are withdrawn. This can be addressed for now by configuring the trace agent to flush ASAP
const tracer = require('#google-cloud/trace-agent').start({
serviceContext: {
service: process.env.K_SERVICE || "unknown-service",
version: process.env.K_REVISION || "unknown-revision"
},
flushDelaySeconds: 1,
});
On a quick review I couldn't see how to trigger the trace flush, but the shorter timeout should help avoid some delays in seeing the trace data appear in Stackdriver.
EDIT: While nice in theory, in practice there's still significant race conditions with CPU withdrawal. Filed https://github.com/googleapis/cloud-trace-nodejs/issues/1161 to see if we can find a more consistent solution.
I have a 3-week-old Function on the v2 SDK and uses the out-box App Insights support, i.e. just set the environment variable and ...magic.
I did read that by default it doesn't send Information level logs to AppInsights and this looks correct because I can't see any of my information tracing but I do see occasional errors I've logged out with my custom message.
But for metrics, its broken. I can see some semblance of my telemetry in the logs, sort of. It's strange.
Look.
Now look at these curious logs.
...see those n/a that match the LogMetric lines in my code?
And when I search explicitly for one of them, nothing is found.
And if I drill down on those n/a logs.
...they're pretty empty.
Is it me or is this all rather fishy?
How do I log metrics from a Function in a way that works?
First, for the metric, do you mean custom metrics? If yes, then you should find it in the customMetrics table in your app insights.
Second, what's the method are you using to send metrics? From your code, what's the implementation of this.Logger?.LogMetric method?
For sending metrics, you can use TrackMetric(just tested with this method, and works) or GetMetric method.
A sample code use TrackMetric(deprecated, but still can use) / GetMetric(recommended) method:
TelemetryConfiguration.Active.InstrumentationKey = "your key";
TelemetryClient client = new TelemetryClient();
client.TrackMetric("my metric", 60);
client.GetMetric("my metric").TrackValue(89);
client.Flush();
In my case, the settings in the host.json file were excluding the metrics from getting to Application Insights. The metrics are being logged at the Information level and I originally had mine configured for the Warning level.
"logging": {
"logLevel": {
"Function.<function name>": "Information", //<--won't see metrics if set to Warning
...
See here for more information on configuring the logging levels.