I am currently looking to monitor the Current Queue Size counter and have been poking about in PowerShell via Get-Counter -Counter "\HTTP Service Request Queues(*)\CurrentQueueSize" which returns the values I am interested in.
The returned value are clearly organised by AppPool but I don't recognise one of them.
\\computername\http service request queues(defaultapppool)\currentqueuesize : 0
\\computername\http service request queues(---2)\currentqueuesize : 0
What is ---2 representing? I have run the command against other servers and they return similar results but a different number, I've seen a ---1,---42 and a few other different values.
I test the command in my server and it also responses different value of http service.
So we cannot figure out what they mean but the docs has explained this result.
However, some server applications may have unnamed Request Queues that cannot be matched to a performance counter instance ID.
Related
We have a service which pings our EP1 Premium service and yesterday we received 3 client side timeout errors after 2 minutes of waiting. When opening the trace in App insights, these requests which time out are not even logged and have no trace of ever being received Azure side, and therefore stay unanswered. By looking at the metrics provided in the Azure Functions app, I found out that 1-2 minutes after the request has been sent, the app loses all its ability to work as its Total App Domains falls to 0 as well as all connections, threads and so on and this state lasts until the next request is received, therefore "skipping" the request that happened beforehand. This is a big issue as I need to make sure requests get answered in a timely manner.
The client service sent HTTP requests to the Azure Functions app expecting an answer, only to time out while the Azure-side doesn't have any record of ever receiving the request.
I believe this issues is related to Consumption Plan of Azure Functions called Cold Start behaviour. The "skipping" mechanism is explained below:
Apps may scale to zero when idle, meaning some requests may have additional latency at startup. The consumption plan does have some optimizations to help decrease cold start time, including pulling from pre-warmed placeholder functions that already have the function host and language processes running.https://learn.microsoft.com/en-us/azure/azure-functions/functions-scale#cold-start-behavior
Please also consider of having look on this article, which explains the behaviour. https://azure.microsoft.com/en-us/blog/understanding-serverless-cold-start/
I am using an IIS web garden for long running requests with 15 worker processes.
With, for example, 3 browsers, typically multiple worker processes are used.
With Apache jMeter, all requests are using the same worker process.
Is there a way to force the use of multiple worker processes?
This may have at least 2 explanations:
You have some hard coded ID or session ID in your test plan. Check for their presence and remove them, add Cookie Manager to your test
You have a load balancer that work in Source IP mode, in this case you need to either change policy to Round Robin or add 2 other machines
If you are using 1 thread with X iterations and expecting different workers then check that:
Cookie Manager is configured this way:
And Thread Group this way (notice "Same User on each iteration is unchecked"):
If issue persists, then please share you plan and check that you don't have somewhere in Header Manager a hardcoded id leading to using 1 worker
Well-behaved JMeter script should produce the same network footprint as the real browser do so if you're observing inconsistencies most probably your JMeter configuration is not matching requests which are being sent by the real browser.
Make sure that your JMeter test is doing what it is supposed to be doing by inspecting requests/responses details using View Results Tree listener
Use a 3rd-party tool like Wireshark or Fiddler to capture the requests originating from browser/JMeter, detect the differences and amend your JMeter configuration to eliminate them
More information: How to make JMeter behave more like a real browser
In the absolute majority of cases JMeter script is not working as expected due to missing or improperly implemented correlation of the dynamic values
I'm running an Azure Function app on Consumption Plan and I want to monitor the amount of instances currently running. Using REST API endpoint of format
https://management.azure.com/subscriptions/{subscr}/resourceGroups/{rg}
/providers/Microsoft.Web/sites/{appname}/instances?api-version=2015-08-01
I'm able to retrieve the instances. However, the result doesn't match the information that I see in Application Insights / Live Metrics Stream.
For example, right now App Insights shows 4 servers online, while API call returns just one (the GUID of this 1 instance is also among App Insights guids).
Who can I trust? Is there a better way to get instance count (e.g. from App Insights)?
UPDATE: It looks like data from REST API are wrong.
I was sending 10000 messages to the queue, logging each function call with respective instance ID which processed the request.
While messages keep coming in and the backlog grows, instance count from REST API seems to be correct (scaled from 1 to 12). After sending stops, the reported instance count rapidly goes down (eventually back to 1, while processors are still busy).
But based on the speed and the execution logs I can tell that the actual instance count kept growing and ended up at 15 instances at the moment of last message processed.
UPDATE2: It looks like SDK refuses to report more than 20 servers. The metric flats out at 20, while App Insights kept steady growth and is already showing 41.
Who can I trust? Is there a better way to get instance count (e.g. from App Insights)?
Based on my understanding we need to use Rest API endpoint to retrieve the instance, App Insights could be configured for multiple WebApps, so the number of servers online in the App Insights may be for multiple WebApps.
Updated:
Based on my test, the number of the application insight may be not real time.
During my test if the WebApp Function scale out then I could get multiple instances with Rest API, and I also can check the number of servers online in the App Insights.
https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourcegroup}/providers/Microsoft.Web/sites/{functionname}/instances?api-version=2016-08-01
But after I finished the test, I could get the number of the instance with Rest API is 1, based on my understanding, it is right result.
At the same time I check it in the Application Insight the number of the servers online is the max number during my test.
And after a while, the number of server online in the application insight also became 1.
So If we want to get the number of intance for Azure function, my suggestion is that using REST API to do that.
Update2：
According to the DavidEbbo mentioned that the REST API is not always reliable.
Unfortunately, the REST API is not always reliable. Specifically, when a Function App scales across multiple scale units, only the instances from the 'home' scale unit are reflected. You probably will not see this in a smallish test, but likely will if you start scaling out widely (say over 20 instances).
I have a question about Azure App Insights Sampling.
If itemCount field is greater than 1 for a log item, does it mean that there was an exactly the SAME request and it was sampled?
My logs have one request that sends this message with itemCount = 2. And this request has ended with OptimisticConcurrencyException, so my transaction has been roll-backed. In this transaction I send a message to 3rd party service.
The most interesting is that they told me they've got 2 messages from my service and my database has been updated (so transaction has been committed).
All of it became clear, if there were 2 requests and one of them returned 200 code, and another returned 500. But app insights log item abot OptimisticConcurrencyException has value itemCount = 2, which means that this exception was thrown twice (for both requests).
Furthermore Beside this I don't see any other requests, that could change data, that request was changing.
So could anybody explain me how app insights samples requests and errors?
This really depends on how/where your sampling occurred, as sampling could have occurred at 3 different places depending on how you have your app configured.
There's a fair amount of documentation about the various layers of sampling, but hypothetically:
The sampling algorithm decides which telemetry items to drop, and which ones to keep (whether it's in the SDK or in the Application Insights service). The sampling decision is based on several rules that aim to preserve all interrelated data points intact, maintaining a diagnostic experience in Application Insights that is actionable and reliable even with a reduced data set. For example, if for a failed request your app sends additional telemetry items (such as exception and traces logged from this request), sampling will not split this request and other telemetry. It either keeps or drops them all together. As a result, when you look at the request details in Application Insights, you can always see the request along with its associated telemetry items.
Update:
I got some more details from people on the team that do the sampling, and it works like this:
Sampling ratio is determined by the number of events per second occurring in the app
The AI SDK randomly selects requests to be sampled when the request begins (so, it is not known whether it will fail or succeed)
AI SDK assigns itemCount=<sampling ratio>
This would then explain the behavior you are seeing, when two requests (success + failure) were counted as two failures: the failed request was sampled "in", and so in telemetry, you'd have 2 failed requests (one request with itemCount=2) instead of a failed and a successful, because the successful one got sampled away.
I'm code reviewing a WCF service.
In the header of each message we inject data that the service is going to use later to build a connection string to a DB.
That's because the service is going to be used by a number of different sites, each with its own DB that the service has to query.
We use wcf extensibility. We have a custom MessageInspector that, after receiving the request, extracts the data from the message header, creates a context (that implements IExtension) and adds it to OperationContext.Current.Extensions.
Before sending the reply the custom context is removed from the Extencions collection.
This is a fairly common pattern, as discussed here:
Where to store data for current WCF call? Is ThreadStatic safe?
and here:
http://social.msdn.microsoft.com/Forums/vstudio/en-US/319cac66-66e8-4dfe-9a82-dfd289c9df1f/wcf-doesnt-have-session-storage-so-where-should-one-store-call-specific-data?forum=wcf
This all works fine as long as the service receives a request, processes it, sends the reply and receives the next request.
But what if the service receives a request and before being able to reply it gets a second request? I built a small console application to test it. I send 2 messages from 2 different threads, I made the wcf service wait for 2 seconds, to ensure the second request comes in before the first one is completed and this is what I get:
Site Id : test1450 ; Session: uuid:2caf47cf-7d46-4d72-9275-d9c037fa0e70;id=2 : Thread Id: 6
Site Id : test1450 ; Session: uuid:2caf47cf-7d46-4d72-9275-d9c037fa0e70;id=3 : Thread Id: 22
It looks like wcf creates 2 sessions executing on 2 different threads, but Site Id is the same. It shouldn't. Judging from this it looks like OperationContext.Current.Extensions is a collection shared between threads.
Right now I'm inclined to think my test is wrong and I missed something.
Has anyone tried something similar and found out that OperationContext.Current is not thread safe?
OperationContext.Current like other similar properties such as HttpContext.Current have thread affine (or thread static) values. So they are thread safe in the sense that multiple threads can read them, but different threads will get different instances. They can be thought of as dictionaries between specific threads and instances.
So in this context they are not thread safe.
Requests are served by a thread pool so concurrent requests will have different thread ids. (up to a point where the thread pool is full, then requests will be put on hold)