I have have tried pushing custom metrics to splunk apm using below dependency and setting the properties using springboot application
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-signalfx</artifactId>
</dependency>
Properties
management.metrics.export.signalfx.access-token= <token>
management.metrics.export.signalfx.enabled=true
management.metrics.export.signalfx.uri=<uri>
management.metrics.export.signalfx.source=testservice
Now there is a requirement to try pushing from spring mvc application and I have added same dependency in the pom and created custom metric code as below but I am getting error while spring application is deployed
MeterRegistry resgRegistry = new SignalFxMeterRegistry(new SignalFxConfig() {
#Override
public String get(String key) {
// TODO Auto-generated method stub
return null;
}
}, Clock.SYSTEM);
Timer myTimer = Timer.builder("surya_timer").register(Metrics.globalRegistry);
Timer timer = Timer.builder("test").register(resgRegistry);
timer.record(Duration.ofMillis(123));
myTimer.record(Duration.ofMillis(567));
Error
io.micrometer.core.instrument.config.validate.ValidationException: signalfx.accessToken was 'null' but it is required
io.micrometer.core.instrument.config.validate.Validated$Either.orThrow(Validated.java:375)
io.micrometer.core.instrument.config.MeterRegistryConfig.requireValid(MeterRegistryConfig.java:49)
io.micrometer.core.instrument.push.PushMeterRegistry.<init>(PushMeterRegistry.java:42)
io.micrometer.core.instrument.step.StepMeterRegistry.<init>(StepMeterRegistry.java:43)
io.micrometer.signalfx.SignalFxMeterRegistry.<init>(SignalFxMeterRegistry.java:78)
io.micrometer.signalfx.SignalFxMeterRegistry.<init>(SignalFxMeterRegistry.java:74)
Please help me How to set this access token in the spring application
Related
I am looking into Azure AppInsights for my telemetry correlation requirement. I have created 3 simple APIs that call one another in succession as below:
First Api -----> Middle Api -----> Another Api
The Api calls are made using Typed HttpClient through a simple service class. All the Api projects have Microsoft.ApplicationInsights.AspNetCore and Microsoft.Extensions.Logging.ApplicationInsights NuGets references added. I have program and service classes for all the APIs as below:
Program.cs
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddControllers();
...
//App Insights
builder.Services.AddSingleton(typeof(ITelemetryChannel),
new ServerTelemetryChannel() { StorageFolder = "/tmp/myfolder" });
builder.Services.AddApplicationInsightsTelemetry();
builder.Services.AddSingleton<IConfiguration>(builder.Configuration);
builder.Services.AddScoped<IWeatherService, DummyWeatherService>();
builder.Services.AddHttpClient<IWeatherService, DummyWeatherService>();
var app = builder.Build();
...
app.Run();
Service
using System.Net.Http.Headers;
using AppInsightsDemo.Api.Models;
namespace AppInsightsDemo.Api.Services;
public class DummyWeatherService : IWeatherService
{
private readonly IConfiguration _configuration;
private readonly HttpClient _httpClient;
public DummyWeatherService(
IConfiguration configuration,
HttpClient httpClient)
{
_configuration = configuration;
_httpClient = httpClient;
_httpClient.BaseAddress = GetMiddleApiBaseUri();
_httpClient.DefaultRequestHeaders.Accept.Clear();
_httpClient.DefaultRequestHeaders.Accept.Add(
new MediaTypeWithQualityHeaderValue("application/json"));
}
private Uri GetAnotherApiBaseUri()
{
var configurationSection = _configuration.GetRequiredSection("Dependencies");
var baseUri = configurationSection.GetValue<string>("MiddleApiUri")
?? throw new ArgumentException("Another Api base uri is empty");
return new Uri(baseUri);
}
public async Task<Weather?> GetWeatherAsync()
{
Weather? weather = null;
var response = await _httpClient.GetAsync("middle");
if (response.IsSuccessStatusCode)
{
weather = await response.Content.ReadAsAsync<Weather>();
}
return weather;
}
}
This is what I end up with in AppInsights sample. The third API event has the same operation id as the first two Api events have but the third event has a different parent id. I expect the third event to have the id of my middle (second) api event (localhost://7290) as its parent id and the three events show up accordingly in a hierarchy.
Can anyone please advise if I am missing some configuration or not using this SDK right? Thank you
This is rather silly of me. I configured the ApplicationInsights connection string for my first api(:7176) and last api(:7206) but missed to configure it for my middle api (:7290) though I have added ApplicationInsights service to all Api projects. It took me a while to figure out the missing connection string. Now I get a nice dependency hierarchy as below:
I guess a connection string validation might come handy. Sorry for the trouble. Thanks.
I have created the following TelemetryFilter:
public class TelemetryFilter : ITelemetryProcessor
{
private ITelemetryProcessor Next { get; set; }
public TelemetryFilter(ITelemetryProcessor next)
{
Next = next;
}
public void Process(ITelemetry item)
{
var dependency = item as DependencyTelemetry;
if (dependency != null && dependency.Success == true) return;
Next.Process(item);
}
}
And added TelemetryFilter to TelemetruyProcessors in ApplicationInsights.config. It works when I run the application on my machine but when it is deployed to test and production environments, dependencies are getting collected by Azure AppInsights. When I see them in Azure Portal they have the property Call status: true. Is Call status refers to dependency.Success? What's the best way to filter out all successful calls to decrease our AppInsights data ingress and lower our Azure bill?
Filter out all successful dependencies:
you can initialize the filter in code. In a suitable initialization class,
AppStart in Global.asax.cs, insert your processor into the chain:
var builder = TelemetryConfiguration.Active.DefaultTelemetrySink.TelemetryProcessorChainBuilder;
builder.Use((next) => new SuccessfulDependencyFilter(next));
// If you have more processors:
builder.Use((next) => new AnotherProcessor(next));
builder.Build();
Refer for filtering sampling
& for request filtering
To Reduce Application Insights cost
You need to optimize Telemetry with Application Insights check here
Check here for some more methods to reduce Application insights cost
I found that ApplicationInsights.config file wasn't set to be copied into the output folder by the build process. That's why it didn't work.
I am dealing with a message ordering issue, after having fixed it a while ago, now the fix does not work anymore.
Just for overview, I have the following environment:
The order is lost somewhere between tcpAdapter and the message receiver.
This issue I have fixed using:
on the producer side - using publisher confirms and returns
rabbitmq:
publisher-confirms: true
publisher-returns: true
on the consumer side - enforcing single thread executor:
The idea I found here: RabbitMQ - Message order of delivery, and I used a post processor for this.
#Component
public class RabbitConnectionFactoryPostProcessor implements BeanPostProcessor {
#Override
public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException {
if (bean instanceof CachingConnectionFactory) {
((CachingConnectionFactory) bean).setExecutor(Executors.newSingleThreadExecutor());
}
return bean;
}
}
And now, after some master-pom updates (we do not control the master pom, it is at project level) the fix suddenly does not work anymore. After checking the differences, I did not see any changes on the spring-rabbit or spring-amqp, I do not understand why there is an impact.
Here are more details if you want concrete examples:
Producer.
The TCP Server sends a message to the tcpAdapter app, which uses spring-integration flow to take the message from TCP and send it to rabbitmq.
Here is the code that does this (inboundAdapterClient I did not post here because I do not think it is important):
#Bean
public IntegrationFlow tcpToRabbitFlowClient() {
return IntegrationFlows.from(inboundAdapterClient())
.transform(tcpToRabbitTransformer)
.channel(TCP_ADAPTER_SOURCE);
.get();
}
Message are received by tcpAdapter app from TCP in the right order, but then the tcpAdapter rabbitmq stack does not send them in the correct order every time (80% of the time ok, 20% wrong order)
Here is the spring boot yml configuration (only relevant info):
spring:
rabbitmq:
publisher-confirms: true
publisher-returns: true
cloud:
stream:
bindings:
tcpAdapterSource:
binder: rabbit
content-type: application/json
destination: tcpadapter.messagereceiver
Consumer.
The message receiver has the single thread executor enforced plus the configuration as below.
Here is the spring boot yml configuration (only relevant info)
spring:
cloud:
stream:
bindings:
fromTcpAdapter:
binder: rabbit
content-type: application/json
destination: tcpadapter.messagereceiver
rabbit:
default:
producer:
exchangeDurable: false
exchangeAutoDelete: true
consumer:
exchangeDurable: false
exchangeAutoDelete: true
Note: There is only one producer and one consumer.
Some versions from pom, maybe it helps:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot</artifactId>
<version>2.2.4.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.amqp</groupId>
<artifactId>spring-amqp</artifactId>
<version>2.2.3.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.amqp</groupId>
<artifactId>spring-rabbit</artifactId>
<version>2.2.3.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream</artifactId>
<version>3.0.1.RELEASE</version>
</dependency>
Solved by removing yml configuration and using explicit bean declarations and factory configurations as described below. The only issue is that is slow performance wise, but this is expected with publisher confirms.
So indeed it was a producer problem.
#Bean
public CachingConnectionFactory connectionFactory() {
com.rabbitmq.client.ConnectionFactory connectionFactoryClient = new com.rabbitmq.client.ConnectionFactory();
connectionFactoryClient.setUsername(username);
connectionFactoryClient.setPassword(password);
connectionFactoryClient.setHost(hostname);
connectionFactoryClient.setVirtualHost(vhost);
return new CachingConnectionFactory(connectionFactoryClient);
}
#Bean("rabbitTemplateAdapter")
#Primary
public RabbitTemplate rabbitTemplate(CachingConnectionFactory connectionFactory) {
connectionFactory.setPublisherConfirmType(CORRELATED);
connectionFactory.setPublisherReturns(true);
RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setMandatory(true);
rabbitTemplate.setConfirmCallback((correlationData, ack, cause)
-> log.debug("correlationData({}),ack({}),cause ({})", correlationData, ack, cause));
rabbitTemplate.setReturnCallback((message, replyCode, replyText, exchange, routingKey)
-> log.debug("exchange({}),route({}),replyCode({}),replyText({}),message:{}",
exchange, routingKey, replyCode, replyText, message));
return rabbitTemplate;
}
And for sending messages:
rabbitTemplateAdapter.invoke(t -> {
t.convertAndSend(
exchange,
DESTINATION,
jsonMessage.getPayload(),
m -> {outboundMapper().fromHeadersToRequest(jsonMessage.getHeaders(), m.getMessageProperties());
return m;
});
t.waitForConfirmsOrDie(10_000);
return true;
});
I did this using the spring rabbit and amqp versions:
<dependency>
<groupId>org.springframework.amqp</groupId>
<artifactId>spring-rabbit</artifactId>
<version>2.2.3.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.amqp</groupId>
<artifactId>spring-amqp</artifactId>
<version>2.2.3.RELEASE</version>
</dependency>
Spring amqp documentation helped a lot, the technique used is called "Scoped Operations":
https://docs.spring.io/spring-amqp/docs/2.2.7.RELEASE/reference/html/#scoped-operations
Is it possible to tell Application Insights to use a different InstrumentationKey depending on the request URL?
Our application works with different clients and we want to separate the logs for them in different instances of Application Insights.
Url format: https://webapi.com/v1/{client_name}/bla/bla
It would be great to setup configuration to select InstrumentationKey by client_name from request.
If the goal is to send different telemetry items to different instrumentation key, the correct way to achieve that is by modifying the individual item with a TelemetryInitializer to have the correct ikey.
An initializer like the following:
item.Context.InstrumentationKey = ikey.
This initializer should access HttpContext and decide the ikey dynamically from request route/other params.
Modifying TC.Active is not recommended for this purpose as its a global shared setting.
(This is not a very common use case - but there are teams inside Microsoft who does this for PROD scale apps)
You can do that. If you have a logger, have the ApplicationInsightsKey parameter-ized and pass the Key for the client on every call, or inject it on load if your application is tenant based.
Checkout the Docs here: Separating telemetry from Development, Test, and Production
Microsoft.ApplicationInsights.Extensibility.
TelemetryConfiguration.Active.InstrumentationKey = <App-Insights-Key-for-the-client>
Just change the Application Insights key before logging and it will do the job.
It would be great to setup configuration to select InstrumentationKey
by client_name from request.
You can dynamically select the ikey as per the client_name from the request. First, you need to get the request url, then check the client_name.
To do that, you can add the following code to the Global.asax file:
void Application_BeginRequest(Object source, EventArgs e)
{
var app = (HttpApplication)source;
//get the request url
var uriObject = app.Context.Request.Url.ToString();
if (uriObject.Contains("/client_name_1"))
{
Microsoft.ApplicationInsights.Extensibility.
TelemetryConfiguration.Active.InstrumentationKey = "ikey_1";
}
else if (uriObject.Contains("/client_name_2"))
{
Microsoft.ApplicationInsights.Extensibility.
TelemetryConfiguration.Active.InstrumentationKey = "ikey_2";
}
else
{
Microsoft.ApplicationInsights.Extensibility.
TelemetryConfiguration.Active.InstrumentationKey = "ikey_3";
}
}
The test result:
But I want to say we rarely use 1 more ikeys in one environment. If your goal is to make the data not being cluttered, I suggest you can use only 1 ikey, and then use Kusto query for your purpose.
Thanks to the answers from #cijothomas and #danpop (link) I was able to understand the whole picture.
Step 1: Create custom ITelemetryInitializer (Microsoft Documentation):
public class MyTelemetryInitializer : ITelemetryInitializer
{
public void Initialize(ITelemetry telemetry)
{
var appKey = CallContext.LogicalGetData("ApplicationKey")?.ToString();
switch (appKey)
{
case "App1":
telemetry.Context.InstrumentationKey = "d223527b-f34e-4c47-8aa8-1f21eb0fc349";
return;
default:
telemetry.Context.InstrumentationKey = "f8ceb6cf-4357-4776-a2b6-5bbed8d2561c";
return;
}
}
}
Step 2: Register custom initializer:
<ApplicationInsights xmlns="http://schemas.microsoft.com/ApplicationInsights/2013/Settings">
<TelemetryInitializers>
<Add Type="Application.WebAPI.MyTelemetryInitializer, Application.WebAPI"/>
</TelemetryInitializers>
<!--<InstrumentationKey>f8ceb6cf-4357-4776-a2b6-5bbed8d2561c</InstrumentationKey>-->
</ApplicationInsights>
OR
protected void Application_Start()
{
// ...
TelemetryConfiguration.Active.TelemetryInitializers.Add(new MyTelemetryInitializer());
}
Step 3: Make some adjustments to the logger (source code taken from #danpop answer Logger target configuration):
var config = new LoggingConfiguration();
ConfigurationItemFactory.Default.Targets.RegisterDefinition("ai", typeof());
ApplicationInsightsTarget aiTarget = new ApplicationInsightsTarget();
aiTarget.InstrumentationKey = "your_key";
aiTarget.Name = "ai";
config.AddTarget("ai", aiTarget);
LogManager.Configuration = config;
ILogger configuration exmples: Log4Net, NLog, System.Diagnostics
I'm attempting to create a second App host for self-hosting, so my unit tests are in the same process as the service, to aid debugging. I created the new app host as follows. When my Unit test calls the .Init() method, I receive the following error:
ServiceStack.LicenseException was unhandled by user code
HResult=-2146233088
Message=The free-quota limit on '10 ServiceStack Operations' has been reached. Please see https://servicestack.net to upgrade to a commercial license or visit https://github.com/ServiceStackV3/ServiceStackV3 to revert back to the free ServiceStack v3.
Source=ServiceStack.Text
The class below is in the same assembly as my real AppHost (my main ASP.NET service project)., so there is definitely a license key in the web.config. file.
public class ServiceTestAppHost : AppSelfHostBase
{
public const string BaseUrl = "http://localhost/dvsvc";
public ServiceTestAppHost()
: base("Test Web Services", typeof(DV.Svc.Interface.HelloService).Assembly) { }
public override void Configure(Funq.Container container)
{
ServiceStack.Text.JsConfig.IncludeNullValues = true;
ServiceStack.Text.JsConfig.DateHandler = ServiceStack.Text.DateHandler.ISO8601;
ServiceStack.Text.JsConfig.ExcludeTypeInfo = true; //exclude the type specification
ServiceStack.Formats.HtmlFormat.Humanize = false;
//most apps use credentials auth. the TVTI player uses Basic auth
Plugins.Add(new AuthFeature(() =>
new DVAuthUserSession(),
new ServiceStack.Auth.IAuthProvider[] { new DVCredentialsAuthProvider(), new DVBasicAuthProvider() })
/*{ HtmlRedirect = null }*/
);
//in memory cache
container.RegisterAs<MemoryCacheClient, ICacheClient>();
SetConfig(new HostConfig { DebugMode = true });
}
}
Self hosted applications don't read from Web.config, they read from the app config App.config, so you would have to create an appropriate config file for the host executable.
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<add key="servicestack:license" value="{licenseKeyText}" />
</configuration>