.net core webapi causes iis application pool to shutdown - iis

Background:
I'm building a .net core webapi does practically nothing more than checking if a given URL exists and returns the result. If a URL exists and is a redirect (301, 302), the api follows the redirect and returns that result as well. The webapi is called by an SPA which does an api-call for every given url in a checkrequest-queue. So, if someone adds 500 urls to the queue the SPA will loop through it and will send 500 calls to the API – something I could improve upon.
The problem:
My IIS application pool is being shut down on a regular basis due to high CPU usage and/or memory usage:
A worker process serving application pool 'api.domain.com(domain)(4.0)(pool)' has requested a recycle because it reached its private bytes memory limit.
The only way to get my API going again is to manually restart the application. I don't think the operations performed by the API are that demanding, but I surely must be doing something wrong here. Can somebody help me please? The code called by the SPA is:
var checkResponse = new CheckResponse();
var httpMethod = new HttpMethod(request.HttpMethod.ToUpper());
var httpRequestMessage = new HttpRequestMessage(httpMethod, request.Url);
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
var httpResponseMessage = await httpClient.SendAsync(httpRequestMessage);
checkResponse.RequestMessage = httpResponseMessage.RequestMessage;
checkResponse.Headers = httpResponseMessage.Headers;
checkResponse.StatusCode = httpResponseMessage.StatusCode;
switch (httpResponseMessage.StatusCode)
{
case HttpStatusCode.Ambiguous:
case HttpStatusCode.Found:
case HttpStatusCode.Moved:
case HttpStatusCode.NotModified:
case HttpStatusCode.RedirectMethod:
case HttpStatusCode.TemporaryRedirect:
case HttpStatusCode.UseProxy:
var redirectRequest = new CheckRequest
{
Url = httpResponseMessage.Headers.Location.AbsoluteUri,
HttpMethod = request.HttpMethod,
CustomHeaders = request.CustomHeaders
};
checkResponse.RedirectResponse = await CheckUrl(redirectRequest);
break;
}
The Action on my ApiController:
[HttpPost]
public async Task<IActionResult> Post([FromBody] CheckRequest request)
{
if (!ModelState.IsValid)
{
return BadRequest(ModelState);
}
var result = await CheckService.CheckUrl(request);
return Ok(result);
}

Related

REST Api is not returning expected data

I wrote an Auth API where it should retrieve the details from my user, but I'm getting error 404 instead. All my users are stored in an Azure Storage Account, and I was using the TableClient class to handle with my table. However I am not able to go any further when I started to do this Auth. I spent over one week only on this function, and I got no progress on this, here is my code:
[FunctionName(nameof(Auth))]
public static async Task<IActionResult> Auth(
[HttpTrigger(AuthorizationLevel.Admin, "POST", Route = "auth")] HttpRequest req,
[Table("User", Connection = "AzureWebJobsStorage")] TableClient tdClient,
ILogger log)
{
string url = String.Format("http://localhost:7235/api/");
HttpMessageHandler handler = new HttpClientHandler()
{
};
var httpClient = new HttpClient(handler)
{
BaseAddress = new Uri(url),
Timeout = new TimeSpan(0, 2, 0)
};
httpClient.DefaultRequestHeaders.Add("ContentType", "application/json");
var plainTextBytes = System.Text.Encoding.UTF8.GetBytes("roy.mitchel#somecompany.com:pass1234");
string val = System.Convert.ToBase64String(plainTextBytes);
httpClient.DefaultRequestHeaders.Add("Authorization", "Basic " + val);
HttpResponseMessage response = httpClient.GetAsync(url).Result;
return new OkObjectResult(response);
}
How can auth my user using this class? Iam doing on the right way?
Thanks.
Debugging any API issues by looking at just the code, is (as you have discovered) a horribly painful process.
I'd strongly recommend using a MITM proxy (like burp) to get visibility of exactly what is sent to, and received from the API. By using this approach, it generally becomes really obvious what is wrong.
If you can't use this approach, then enable logging for the raw HTTP request and response, as outlined in this guide.

How to resolve external API latency while calling from Azure deployed application?

Info :
I have below 2 method which is part of Web API (not core API) and it is deployed in Azure
Method 1 :
public async Task<bool> ProcessEmployee(list<employee> EmployeeList)
var tasks = new List<Task<EmployeeResponseModel>>();
HttpClient localHttpClient = new HttpClient();
localHttpClient.Timeout = TimeSpan.FromSeconds(100);
foreach (var employee in EmployeeList) // **having 1000 calls**
{
tasks.Add(GetAddressResponse(employee.URL,localHttpClient));
}
var responses = await Task.WhenAll(tasks);
}
Method 2 :
private async Task<EmployeeResponseModel> GetAddressResponse(url, HttpClient client)
{
var response = new EmployeeResponseModel();
try
{
using (HttpResponseMessage apiResponse = await client.GetAsync(**url**))
{
if (apiResponse.IsSuccessStatusCode)
{
var res= await apiResponse.Content.ReadAsStringAsync();
response = JsonConvert.DeserializeObject<EmployeeResponseModel>(res);
}
}
return response;
}
catch (Exception ex)
{
}
return response;
}
If i monitor from Azure -> Diagnose and Solve Problem -> Web App Slow all external API calls is showing latency issue
But if i am calling same external API from Postman is is quite fast and having less latency
method 1 and method 2 is part of one web api and it is deployed on Azure AppService.
getAddress is external API which is been deployed in other environment and don't have much information
if we are calling external API i.e 'getAddress' from 1) we are facing high latency more than 5 sec.
if we are calling external API i.e 'getAddress' from Postman we receive response in 303 ms.
I guess it results from the location of the service plan.
If the location of the service plan is far away from you position, it may cause the latency. But it can't rule out other possibilities, so my suggestion is debug in localhost first to rule out the possibility of the code.

Azure Durable Function (external functions)

I developed a couple of microservice using Azure functions, every service has independent use case and different programming language.
Now I have a use case to use all service in below order, So I developed one more Azure function to use all service in given order. below code running well.
public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = null)]HttpRequestMessage req, TraceWriter log)
{
string returnValue = string.Empty;
dynamic data = await req.Content.ReadAsStringAsync();
if (data == null)
{
return req.CreateResponse(HttpStatusCode.BadRequest, "Please pass a value in the request body");
}
else
{
string body = data.ToString();
var transformResult = await HttpRestHelper.CreateRequestAsync(AppConfiguration.TransformServiceEndPoint, body, HttpMethod.POST);
var validationResult = await HttpRestHelper.CreateRequestAsync(AppConfiguration.ValidationServiceEndPoint, transformResult.Result.ToString(), HttpMethod.POST);
if (validationResult.Result != null && Convert.ToBoolean(validationResult.Result))
{
var encryptionResult = await HttpRestHelper.CreateRequestAsync(AppConfiguration.EncryptionServiceEndPoint, transformResult.Result.ToString(), HttpMethod.POST);
var storageResult = await HttpRestHelper.CreateRequestAsync(AppConfiguration.StorageServiceEndPoint, encryptionResult.Result.ToString(), HttpMethod.POST);
returnValue = storageResult.Result.ToString();
}
else
{
returnValue = "Validation Failed";
}
return req.CreateResponse(HttpStatusCode.OK, returnValue, "text/plain");
}
}
Question
If every microservice takes 1 min to execution, I have to wait ~4min in my Super Service and billed for 4+ min. (We don't need to pay for waiting time :) https://www.youtube.com/watch?v=lVwWlZ-4Nfs)
I want to use Azure Durable functions here but didn't get any method to call external url.
Please help me or suggest a better solution.
Thanks In Advance
Durable Orchestration Functions don't work with arbitrary HTTP endpoints. Instead, you need to create individual functions as Activity-triggered.
Orchestration will use message queues behind the scenes rather than HTTP. HTTP is request-response in nature, so you have to keep the connection and thus pay for it.
Queue-based orchestrator can also give you some extra resilience in face of intermittent failures.

ServiceStack/Funq not disposing RavenDB document session after request is complete

In trying to integrate RavenDB usage with Service Stack, I ran across the following solution proposed for session management:
A: using RavenDB with ServiceStack
The proposal to use the line below to dispose of the DocumentSession object once the request is complete was an attractive one.
container.Register(c => c.Resolve<IDocumentStore>().OpenSession()).ReusedWithin(ReuseScope.Request);
From what I understand of the Funq logic, I'm registering a new DocumentSession object with the IoC container that will be resolved for IDocumentSession and will only exist for the duration of the request. That seemed like a very clean approach.
However, I have since run into the following max session requests exception from RavenDB:
The maximum number of requests (30) allowed for this session has been
reached. Raven limits the number of remote calls that a session is
allowed to make as an early warning system. Sessions are expected to
be short lived, and Raven provides facilities like Load(string[] keys)
to load multiple documents at once and batch saves.
Now, unless I'm missing something, I shouldn't be hitting a request cap on a single session if each session only exists for the duration of a single request. To get around this problem, I tried the following, quite ill-advised solution to no avail:
var session = container.Resolve<IDocumentStore>().OpenSession();
session.Advanced.MaxNumberOfRequestsPerSession = 50000;
container.Register(p => session).ReusedWithin(ReuseScope.Request);
Here is a sample of how I'm using the resolved DocumentSession instance:
private readonly IDocumentSession _session;
public UsersService(IDocumentSession session)
{
_session = session;
}
public ServiceResponse<UserProfile> Get(GetUser request)
{
var response = new ServiceResponse<UserProfile> {Successful = true};
try
{
var user = _session.Load<UserProfile>(request.UserId);
if (user == null || user.Deleted || !user.IsActive || !user.IsActive)
{
throw HttpError.NotFound("User {0} was not found.".Fmt(request.UserId));
}
response.Data = user;
}
catch (Exception ex)
{
_logger.Error(ex.Message, ex);
response.StackTrace = ex.StackTrace;
response.Errors.Add(ex.Message);
response.Successful = false;
}
return response;
}
As far as I can see, I'm implementing SS + RavenDB "by the book" as far as the integration point goes, but I'm still getting this max session request exception and I don't understand how. I also cannot reliably replicate the exception or the conditions under which it is being thrown, which is very unsettling.

Is there a more elegant way to build URIs in ServiceStack?

I'm building a Request/Acknowledge/Poll style REST service with NServiceBus underneath to manage queue processing. I want to give the client a URI to poll for updates.
Therefore I want to return a location header element in my web service as part of the acknowledgement. I can see that it is possible to do this:
return new HttpResult(response, HttpStatusCode.Accepted)
{
Location = base.Request.AbsoluteUri.CombineWith(response.Reference)
}
But for a Url such as: http://localhost:54567/approvals/?message=test, which creates a new message (I know I should probably just use a POST), the location will be returned as: http://localhost:54567/approvals/?message=test/8f0ab1c1a2ca46f8a98b75330fd3ac5c.
The ServiceStack request doesn't expose the Uri fragments, only the AbsouteUri. This means that I need to access the original request. I want this to work regardless of whether this is running in IIS or in a self hosted process. The closest I can come up with is the following, but it seems very clunky:
var reference = Guid.NewGuid().ToString("N");
var response = new ApprovalResponse { Reference = reference };
var httpRequest = ((System.Web.HttpRequest)base.Request.OriginalRequest).Url;
var baseUri = new Uri(String.Concat(httpRequest.Scheme, Uri.SchemeDelimiter, httpRequest.Host, ":", httpRequest.Port));
var uri = new Uri(baseUri, string.Format("/approvals/{0}", reference));
return new HttpResult(response, HttpStatusCode.Accepted)
{
Location = uri.ToString()
};
This now returns: http://localhost:55847/approvals/8f0ab1c1a2ca46f8a98b75330fd3ac5c
Any suggestions? Does this work regardless of how ServiceStack is hosted? I'm a little scared of the System.Web.HttpRequest casting in a self hosted process. Is this code safe?
Reverse Routing
If you're trying to build urls for ServiceStack services you can use the RequestDto.ToGetUrl() and RequestDto.ToAbsoluteUri() to build relative and absolute urls as seen in this earlier question on Reverse Routing. e.g:
[Route("/reqstars/search", "GET")]
[Route("/reqstars/aged/{Age}")]
public class SearchReqstars : IReturn<ReqstarsResponse>
{
public int? Age { get; set; }
}
var relativeUrl = new SearchReqstars { Age = 20 }.ToUrl("GET");
var absoluteUrl = HostContext.Config.WebHostUrl.CombineWith(relativeUrl);
relativeUrl.Print(); //= /reqstars/aged/20
absoluteUrl.Print(); //= http://www.myhost.com/reqstars/aged/20
For creating Urls for other 3rd Party APIs look at the Http Utils wiki for example extension methods that can help, e.g:
var url ="http://api.twitter.com/user_timeline.json?screen_name={0}".Fmt(name);
if (sinceId != null)
url = url.AddQueryParam("since_id", sinceId);
if (maxId != null)
url = url.AddQueryParam("max_id", maxId);
var tweets = url.GetJsonFromUrl()
.FromJson<List<Tweet>>();
You can also use the QueryStringSerializer to serialize a number of different collection types, e.g:
//Typed POCO
var url = "http://example.org/login?" + QueryStringSerializer.SerializeToString(
new Login { Username="mythz", Password="password" });
//Anonymous type
var url = "http://example.org/login?" + QueryStringSerializer.SerializeToString(
new { Username="mythz", Password="password" });
//string Dictionary
var url = "http://example.org/login?" + QueryStringSerializer.SerializeToString(
new Dictionary<string,string> {{"Username","mythz"}, {"Password","password"}});
You can also serialize the built-in NameValueCollection.ToFormUrlEncoded() extension, e.g:
var url = "http://example.org/login?" + new NameValueCollection {
{"Username","mythz"}, {"Password","password"} }.ToFormUrlEncoded();

Resources