I have a web application acting as a gateway to a variety of internal services. These services are consumed by using a single instance of HttpClient, instantiated at startup (i.e. Startup.cs)
After a certain period of time, something is causing our HttpClient to stop hitting our APIs and immediately fail with HTTP 502 errors for every call using that client. (Note that I can still hit our APIs using other means, such as Postman)
Also be aware that this is all deployed to a variety of AppServices in Azure.
Any ideas as to what could corrupt HttpClient in this manner?
Thanks,
-Tim
This error is usually given when you are behind a proxy server.
If you are using a proxy you will probably have to authenticate the http client again.
Another possibility is to generate the Singleton instance again when it gives an error, for Example Singleton.killInstance(); If your implementation is like this:
public class Singleton
{
private static Singleton instance;
private Singleton() {}
public static Singleton instance
{
get
{
if (instance == null)
{
instance = new Singleton();
}
return instance;
}
}
public static void Singleton killInstance()
{
instance == null
}
}
Whatever the error might be, your httpclient request is causing your web server to crash. In order to investigate it, you need to go to to the Event Viewer of your AppService.
Here are the steps to launch the event viewer:
Go to Kudu management site of your website (ie https://{sitename}.scm.azurewebsites.net
Open menu item: Tools => Support
Choose relevant Azure AD Directory of your website
Click on Analyze => Event Viewer.
Check the error messages
Related
I have a web application that sometimes undergoes "Platform (Infrastructure Upgrade)" events.
The only way that I can detect these is by going to the Azure portal and drilling down
Diagnose and Solve Problems > Web App Restarted
and looking for errors such as below
My question really, is can I use Application Insights to query Azure to find out when these events happen?
Seems like a simple thing to do, but can't figure it out to do it.
can I use Application Insights to query Azure to find out when these events happen?
No, but your application is maybe able to log an event upon start or close? For example, in a .Net Core app you can listen to stop/start events like this:
// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public static void Configure(IApplicationBuilder app, IWebHostEnvironment env, IHostApplicationLifetime hostApplicationLifetime, TelemetryClient telemetryClient)
{
hostApplicationLifetime.ApplicationStarted.Register(() => { telemetryClient.TrackEvent("App Started"); });
hostApplicationLifetime.ApplicationStopping.Register(() => { telemetryClient.TrackEvent("App Stopping"); });
hostApplicationLifetime.ApplicationStopped.Register(() => {
telemetryClient.TrackEvent("App Stopped");
telemetryClient.Flush();
Thread.Sleep(TimeSpan.FromSeconds(5));
});
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
app.UseMvc();
}
It uses the IHostApplicationLifetime interface to get notified of application lifetime events.
We finally got EventSource and ElasticSearch correctly configured in our service fabric cluster. Now that we have that we want to add EventSources to our web applications that interact with our service fabric applications so that we can view all events (application logs) in one location and filter / query via Kibana.
Our issue seems to be related to the differences between a service fabric app which is an exe and a .NET 4.6 (not .net CORE) web app which is stateless. In service Fabric we place the using statement that instantiates the pipeline in Program.cs and set an infinite sleep.
private static void Main()
{
try
{
using (var diagnosticsPipeline = ServiceFabricDiagnosticPipelineFactory.CreatePipeline("CacheApp-CacheAPI-DiagnosticsPipeline"))
{
ServiceEventSource.Current.ServiceTypeRegistered(Process.GetCurrentProcess().Id, typeof(Endpoint).Name);
// Prevents this host process from terminating so services keeps running.
Thread.Sleep(Timeout.Infinite);
}
How do I do this in a web app? This is the pipeline code we are using for a non ServiceFabric implementation of the EventSource. This is what we are using:
using (var pipeline = DiagnosticPipelineFactory.CreatePipeline("eventFlowConfig.json"))
{
IEnumerable ie = System.Diagnostics.Tracing.EventSource.GetSources();
ServiceEventSource.Current.Message("initialize eventsource");
}
We are able to see the pipeline and send events to ElasticSearch from within the using statement but not outside of it. So the question is:
how/where do we place our pipeline using statement for a web app?
Do we need to instantiate and destroy the pipeline that every time we log or is there a way to reuse that pipeline across the stateless web events? It would seem like that would be very expensive and hurt performance. Maybe we can we cache a pipeline?
That’s the jist, let me know if you need clarification. I see lots of doco out there for client apps but not much for web apps.
Thanks,
Greg
UPDATE WITH SOLUTION CODE
DiagnosticPipeline pipeline;
protected void Application_Start(Object sender, EventArgs e)
{
try
{
pipeline = DiagnosticPipelineFactory.CreatePipeline("eventFlowConfig.json");
IEnumerable ie = System.Diagnostics.Tracing.EventSource.GetSources();
AppEventSource.Current.Message("initialize eventsource");
}
}
protected void Application_End(Object sender, EventArgs e)
{
pipeline.Dispose();
}
Assuming ASP.NET Core the simplest way to initialize EventFlow pipeline would be in the Program.cs Main() method, for example:
public static void Main(string[] args)
{
using (var pipeline = DiagnosticPipelineFactory.CreatePipeline("eventFlowConfig.json"))
{
var host = new WebHostBuilder()
.UseKestrel()
.UseContentRoot(Directory.GetCurrentDirectory())
.UseIISIntegration()
.UseStartup<Startup>()
.UseApplicationInsights()
.Build();
host.Run();
}
}
This takes advantage of the fact that host.Run() will block until the server is shut down, and so the pipeline will exist during the time when requests are received and served.
Depending on the web framework you use things might vary. E.g. if the one you use offers "setup" and "cleanup" hooks, you could create a diagnostic pipeline during setup phase (and store a reference to it in some member variable), then dispose of it during cleanup phase. For example, in ASP.NET classic you'd put the code in global.asax.cs and leverage Application_OnStart and Application_OnEnd methods. See Application Instances, Application Events, and Application State in ASP.NET for details.
Creating a pipeline instance every time a request is served is quite inefficient, like you said. There is really no good reason to do that.
We are moving from an on premise-like application to a multi tenant cloud application.
for my web application we made a very simple interface based on IPlugin, to create a plugin architecture. (customers can have/install different plugins)
public interface IWebPlugin : IPlugin
{
string ContentBaseUrl { set; get; }
}
We have some plugins that would normally be loaded in on startup. Now i'm migrating the code to load at the beginning of a request (the Register function is called on request start), and scope everything inside this request.
It's not ideal but it would bring the least impact on the plugin system for now.
I could scope the Container by making an AppHost child container which would stick to the request:
Container IHasContainer.Container
{
get
{
if (HasStarted)
return ChildContainer;
return base.Container;
}
}
public Container ChildContainer
{
get { return HttpContext.Current.Items.GetOrAdd<Container>("ChildContainer", c => Container.CreateChildContainer()); }
}
problem case
Now im trying to make plugins work that actually add API services.
appHost.Routes.Add<GetTranslations>("/Localizations/translations", ApplyTo.Get);
But this service is unreachable (and not visible in metadata). How do i make it reachable?
I see you execute the following in ServiceController AfterInit. Re-executing this still wouldnt make it work.
//Copied from servicestack repo
public void AfterInit()
{
//Register any routes configured on Metadata.Routes
foreach (var restPath in appHost.RestPaths)
{
RegisterRestPath(restPath);
//Auto add Route Attributes so they're available in T.ToUrl() extension methods
restPath.RequestType
.AddAttributes(new RouteAttribute(restPath.Path, restPath.AllowedVerbs)
{
Priority = restPath.Priority,
Summary = restPath.Summary,
Notes = restPath.Notes,
});
}
//Sync the RestPaths collections
appHost.RestPaths.Clear();
appHost.RestPaths.AddRange(RestPathMap.Values.SelectMany(x => x));
appHost.Metadata.AfterInit();
}
solution directions
Is there a way i could override the route finding? like extending RestHandler.FindMatchingRestPath(httpMethod, pathInfo, out contentType);
Or could i restart the path compilation/caching? (would be enough for now that the service would be reachable tenant wide )
All configuration in ServiceStack should be contained within AppHost.Configure() and remain immutable thereafter. It's not ThreadSafe to modify ServiceStack's Static Configuration at runtime like trying to modify registered routes or Service Metadata which needs to be registered once at StartUp in AppHost.Configure().
It looks as though you'll need to re-architect your solution so all Routes are registered on Startup. If it helps Plugins can implement IPreInitPlugin and IPostInitPlugin interfaces to execute custom logic before and after Plugins are registered. They can also register a appHost.AfterInitCallbacks to register custom logic after ServiceStack's AppHost has been initialized.
Not sure if it's applicable but at runtime you can "hi-jack Requests" in ServiceStack by registering a RawHttpHandler or a PreRequestFilter, e.g:
appHost.RawHttpHandlers.Add(httpReq =>
MyShouldHandleThisRoute(httpReq.PathInfo)
? new CustomActionHandler((req, res) => {
//Handle Route
});
: null);
Simple answer seems to be, no. The framework wasn't build to be a run-time plugable system.
You will have to make this architecture yourself on top of ServiceStack.
Routing solution
To make it route to these run-time loaded services/routes it is needed to make your own implementation.
The ServiceStack.HttpHandlerFactory checks if a route exist (one that is registered on init). so here is where you will have to start extending. The method GetHandlerForPathInfo checks if it can find the (service)route and otherwise return a NotFoundHandler or StaticFileHandler.
My solution consists of the following code:
string contentType;
var restPath = RestHandler.FindMatchingRestPath(httpMethod, pathInfo, out contentType);
//Added part
if (restPath == null)
restPath = AppHost.Instance.FindPluginServiceForRoute(httpMethod, pathInfo);
//End added part
if (restPath != null)
return new RestHandler { RestPath = restPath, RequestName = restPath.RequestType.GetOperationName(), ResponseContentType = contentType };
technically speaking IAppHost.IServiceRoutes should be the one doing the routing. Probably in the future this will be extensible.
Resolving services
The second problem is resolving the services. After the route has been found and the right Message/Dto Type has been resolved. The IAppHost.ServiceController will attempt to find the right service and make it execute the message.
This class also has init functions which are called on startup to reflect all the services in servicestack. I didn't found a work around yet, but ill by working on it to make it possible in ServiceStack coming weeks.
Current version on nuget its not possible to make it work. I added some extensibility in servicestack to make it +- possible.
Ioc Solution out of the box
For ioc ServiceStack.Funq gives us a solution. Funq allows making child containers where you can register your ioc on. On resolve a child container will, if it can't resolve the interface, ask its parent to resolve it.
Container.CreateChildContainer()
I have successfully implemented an enterprise SharePoint solution using Ninject dependency injection and other infrastructure such as NLog logging etc using an Onion architecture. With a HttpModule as an Composition Root for the injection framework, it works great for normal web requests:
public class SharePointNinjectHttpModule: IHttpModule, IDisposable
{
private readonly HttpApplication _httpApplication;
public void Init(HttpApplication context)
{
if (context == null) throw new ArgumentException("context");
Ioc.Container = IocContainerFactory.CreateContainer();
}
public void Dispose()
{
if(_httpApplication == null) return;
_httpApplication.Dispose();
Ioc.Container.Dispose();
}
}
The CreateContainer method loads the Ninject modules from a separate class library and my ioc container is abstracted.
For normal web application requests I used a shared static class for the injector called Ioc. The UI layer has a MVP pattern implementation. E.g in the aspx page the presenter is constructed as follows:
presenter = Ioc.Container.Get<SPPresenter>(new Ninject.Parameters.ConstructorArgument("view", this));
I'm still reliant on a Ninject reference for the parameters. Is there any way to abstract this, other than mapping a lot of methods in a interface? Can't I just pass in simple types for arguments?
The injection itself works great, however my difficulty comes in when using external processes such as SharePoint Timer Jobs. It would obviously be a terrible idea to reuse the ioc container from here, so it needs to bootstrap the dependencies itself. In addition, it needs to load the configuration from the web application pool, not the admin web application. Else the job would only be able to run on the application server. This way the job can run on any web server, and your SharePoint feature only has to deploy configurations etc. to the web apllication.
Here is the execute method of my timer job, it opens the associated web application configuration and passes it to the logging service (nlog) and reads it's configuration from the external web config service. I have written code that reads a custom section in the configuration file and initializes the NLog logging infrastructure.
public override void Execute(Guid contentDbId)
{
try
{
using (var ioc = IocContainerFactory.CreateContainer())
{
// open configuration from web application
var configService = ioc.Get<IConfigService>(new ConstructorArgument("webApplicationName", this.WebApplication.Name));
// get logging service and set with web application configuration
var logginService = ioc.Get<ILoggingService>();
logginService.SetConfiguration(configService);
// reapply bindings
ioc.Rebind<IConfigService>().ToConstant(configService);
ioc.Rebind<ILoggingService>().ToConstant(logginService);
try
{
logginService.Info("Test Job started.");
// use services etc...
var productService = ioc.Get<IProductService>();
var products = productService.GetProducts(5);
logginService.Info("Got products: " + products.Count() + " Config from web application: " + configService.TestConfigSetting);
logginService.Info("Test Job completed.");
}
catch (Exception exception)
{
logginService.Error(exception);
}
}
}
catch (Exception exception)
{
EventLog.WriteError(exception, "Exception thrown in Test Job.");
}
}
This does not make the timer jobs robust enough, and there is a lot of boiler plate code. My question is how do I improve on this design? It's not the most elegant, I'm looking for a way to abstract the timer job operation code and have it's dependencies injected into it for each timer job. I would just like to hear your comments if you think this is a good approach. Or if someone has faced similar problems like this? Thanks
I think I've answered my own question with the presenter construction code above. When using dependency injection in a project, the injection itself is not that important, but the way it changes the way you write code is far more significant. I need to use a similar pattern such as command for my SharePoint timer job operations. I'd just like the bootstrapping to be handled better.
I have written a custom Windows Service that writes data to a custom Event Log (in the Windows Event Viewer).
For dev'ing the biz logic that the service uses, I created a Windows Form which simulates the Start/Stop methods of the Windows Service.
When executing the biz logic via the Windows Forms, info is successfully written to my custom Event Log. However, when I run the same biz logic from the custom Windows Service, information is failing to be written to the Event Log.
To be clear, I have written a library (.dll) that does all the work that I want my custom service to do - including the create/write to the custom Event Log. My Form application references this library as does my Windows Service.
Thinking the problem is a security issue, I manually set the custom Windows Service to "Log on" as "Administrator", but the service still did not write to the Event Log.
I'm stuck on how to even troubleshoot this problem since I can't debug and step into the code when I run the service (if there is a way to debug a service, please share).
Do you have any ideas as to what could be causing my service to fail to write to the event log?
I use it like this. There can be some typos. Writed it on my phone browser...
public class MyClass
{
private EventLog eventLog = new EventLog();
public void MyClass()
{
if (!System.Diagnostics.EventLog.SourceExists("MyLogSource"))
System.Diagnostics.EventLog.CreateEventSource("MyLogSource", "MyLogSource_Log");
eventLog.Source = "MyLogSource";
eventLog.Log = "MyLogSource_Log";
}
private void MyLogWrite()
{
eventLog.WriteEntry(ex.ToString(), EventLogEntryType.Error);
}
}
To debug a running service you need to attach to the process. See here for the steps.
You could also add parameter checking to the Main entry point and have a combination service and console app which would start based on some flag. See this SO post for a good example but here's a snippet:
using System;
using System.ServiceProcess;
namespace WindowsService1
{
static class Program
{
static void Main(string[] args)
{
if (args == null)
{
Console.WriteLine("Starting service...");
ServiceBase.Run(new ServiceBase[] { new Service1() });
}
else
{
Console.WriteLine("Hi, not from service: " + args[0]);
}
}
}
}
The above starts the app in console mode if there any parameters exist and in service mode if there are no parameters. Of course it can be much fancier but that's the gist of the switch.
I discovered why my service wasn't writing to the Event Log.
The problem had nothing to do with any part of the code/security/etc that was attempting to write to the EL. The problem was that my service wasn't successfully collecting the information that is written to the EL - therefore, the service wasn't even attempting to write the log.
Now that I fixed the code that collects the data, data is successfully writing to the event log.
I'm open to having this question closed since the question was amiss to the real problem.