I have a web app running on prem (IIS) and I am trying to collect performance counters from it, although I have everything setup on my local and I am seeing all other types of telemetry in my Azure resource (requests, exceptions, traces) performance Counters are just not there,
it is worth mentioning that I am running 2 more web applications (they all work together) under the same site and same applicationPool in IIS and they are collecting Performance counters just fine, they all use the same package that contains our Application Insights implementation. Something strange I have noticed is that, when I am serving the application from Visual Studio, all the telemetry goes through, even performance counters, but, when I try with an installed Instance of my webapplications (using our inhouse installer ) any request or perfCounter for this particular web application just won't work.
I don't know what else to check, all 3 web applications should be collecting performance counters the same way since they are pretty much alike, and running under the same AppPool, same goes for requests.
this is what my appInsights.config looks like:
<Add Type="Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.PerformanceCollectorModule, Microsoft.AI.PerfCounterCollector">
<Counters>
<Add PerformanceCounter="\Processor(_Total)\% Processor Time" ReportAs="Processor Total - % Processor Time"/>
<Add PerformanceCounter="\Process(??APP_W3SVC_PROC??)\% Processor Time" ReportAs="W3SVC - % Processor Time"/>
<Add PerformanceCounter="\Process(??APP_WIN32_PROC??)\% Processor Time" ReportAs="Win32 - % Processor Time"/>
<Add PerformanceCounter="\Process(??APP_CLR_PROC??)\% Processor Time" ReportAs="CLR - % Processor Time"/>
<Add PerformanceCounter="\Process(_Total)\Private Bytes" ReportAs="Process Total - Private Bytes"/>
<Add PerformanceCounter="\Process(??APP_W3SVC_PROC??)\Private Bytes" ReportAs="W3SVC - Private Bytes"/>
<Add PerformanceCounter="\Process(??APP_WIN32_PROC??)\Private Bytes" ReportAs="Win32 - Private Bytes"/>
<Add PerformanceCounter="\Process(??APP_CLR_PROC??)\Private Bytes" ReportAs="CLR - Private Bytes"/>
</Counters>
<!--
Use the following syntax here to collect additional performance counters:
<Counters>
<Add PerformanceCounter="\Process(??APP_WIN32_PROC??)\Handle Count" ReportAs="Process handle count" />
...
</Counters>
PerformanceCounter must be either \CategoryName(InstanceName)\CounterName or \CategoryName\CounterName
NOTE: performance counters configuration will be lost upon NuGet upgrade.
The following placeholders are supported as InstanceName:
??APP_WIN32_PROC?? - instance name of the application process for Win32 counters.
??APP_W3SVC_PROC?? - instance name of the application IIS worker process for IIS/ASP.NET counters.
??APP_CLR_PROC?? - instance name of the application CLR process for .NET counters.
-->
</Add>
few more details: AppInsights sdk 2.4
.NetFramework 4.6.1
Thanks, all help is welcome
Related
we have an application pool that has a slower initialization time after an app pool recycle in IIS 10. (around 5-7 seconds after recycle then 30-50 ms after the first request.)
I have done some research and found that the "Application Initialization." module should do the trick.
I installed it onto the server and set the application pool to "AlwaysRunning" and the corresponding site to "PreloadEnabled == True." After making those changes we tested by recycling and the response times seemed a bit better...down to 3 to 4 seconds after recycle. I tried to then disable the "overlapped recycle" to see if that helped and again it did a bit better 1.5 to 2 seconds after recycle and then 20 to 30 ms after the first request.
Question is: Is that the best we can expect? I was hoping there would be away to fully pre-warm the app pool so that even the first request is around a few ms. The issue is that test messages we are sending to the API are small and the ones in Prod would be much larger so an initialization of 3-4 seconds could be much much longer in Prod.
Following are the steps you can perform to Auto Initialize application hosted on IIS –
• Installed Application Initialization feature - IIS 8.0 Application
https://learn.microsoft.com/en-us/iis/get-started/whats-new-in-iis-8/iis-80-application-initialization
• Make sure the warmup.dll (which should load from C:\Windows\SysWOW64\inetsrv\warmup.dll or from C:\Windows\system32\inetsrv\warmup.dll depending on the bitness of your process) present
• Configure the app pool to be always running (from the advanced properties)
eg. In the applicationHost.config (%WINDIR%\system32\inetsrv\config\applicationHost.config) file the application pool setting looks like this –
<add name="PreLoadApp" autoStart="true" managedRuntimeVersion="" startMode="AlwaysRunning">
<processModel idleTimeout="00:00:00" />
</add>
• Scroll down a little more in applicationHost.config to the configuration element. Within that section there will be an entry, modify your application as below
<site name="PreLoadApp" id="5">
<application path="/" applicationPool="PreLoadApp" preloadEnabled="true">
<virtualDirectory path="/" physicalPath="C:\inetpub\wwwroot\PreLoadApp" />
</application>
• Then selected the site from the IIS manager tree view on the left-hand side and go to the configuration editor.
this time underneath the <system.WebServer/applicationInitialization>
tag, and look at the list of requests
set the request to only target a page called (host header is optional) and also you can provide a query string
q=abhi // to identify if request is coming from preload only.
set the doAppInitAfterRestart parameter to true and apply the
settings
And you should be good by now, try recycling your application pool, it should Initialize and warmup automatically.
You can refer these MS docs to know more about Application Initialization and configuration steps –
https://learn.microsoft.com/en-us/iis/configuration/system.webServer/applicationInitialization/
https://learn.microsoft.com/en-us/iis/get-started/whats-new-in-iis-8/iis-80-application-initialization
IIS saves doAppInitAfterRestart in the web.config of the application, which might be overwritten by future deployments. Therefore I'd put the web.config under source control and make it part of the deployed artifact.
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.webServer>
<handlers>
<add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
</handlers>
<aspNetCore processPath="%LAUNCHER_PATH%" arguments="%LAUNCHER_ARGS%" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" hostingModel="inprocess" />
<!-- Add this node to web.config -->
<applicationInitialization doAppInitAfterRestart="true">
<!-- Only needed when website contains multiple child apps -->
<add initializationPage='/hangfire',hostname='' />
<applicationInitialization />
</system.webServer>
</configuration>
After successfully developing an application with multiple ServiceStack services, we are moving to other testing environments, lots of them due to us running a SAAS model (aka multi-tenant). I'd like to reuse some of the base infrastructure services, primarily Redis and RabbitMQ across a few of these environments.
We're using the IAppSetting interface to pull our configuration from multiple sources into one cohesive settings object at run-time, which is then filtered tier. Since tier drives the configuration per environment it made sense to use Tier to prefix any RabbitMQ messages queues, and prefix any generated cache keys that will be used by Redis, thus providing collision protection per environment.
Below is an example:
RabbitMQ => "Some MQ method here" => "mq:qa1.Outbound.inq"
Redis => "Some Redis method here" => "urn:qa1.somePoco:123"
Here is an example configuration and the various enviroments
<appSettings>
<add key="Tier" value="qa1" />
<!--<add key="Tier" value="dev" />-->
<!--<add key="Tier" value="tst" />-->
<!--<add key="Tier" value="stg" />-->
<!--<add key="Tier" value="prod" />-->
</appSettings>
Thank you,
Stephen
Some examples on how to modify Queue Names are in MqNameTests, e.g:
QueueNames.SetQueuePrefix("site1.");
Will add a prefix on QueueNames, e.g:
site1.mq:TestPrefix.inq
Otherwise you can use QueueNames.ResolveQueueNameFn to have complete control over the MQ name, e.g:
QueueNames.ResolveQueueNameFn = (typeName, suffix) =>
"SITE.{0}{1}".Fmt(typeName, suffix.ToUpper());
QueueNames<TestFilter>.In.Print(); // SITE.TestFilter.INQ
Note the same configuration also needs to applied on the client so the same MQ names gets used.
Configuring ServiceStack with AppSettings
ServiceStack is a code-first framework which means all configuration is done in code, but has a rich and versatile configuration model where you can get the behavior your after by reading App Settings in AppHost.Configure():
QueueNames.SetQueuePrefix(AppSettings.Get("Tier","dev"));
Where if Tier doesn't exist in your Web.config (e.g. in Unit Tests) it will use dev otherwise will use the value in your appSettings:
<appSettings>
<add key="Tier" value="qa1" />
</appSettings>
This is a follow up on this question. I used Cerebrata Diagnostics Manager Remote Diagnostics to try to turn on IIS logs. I hadn't deployed with it on. It seemed to work and a few files were copied. Then it never seemed to work again. I tweaked settings again. I tried deleting the iis related blob and table storage entries to see if that would get it to start over. Here is what the config looks like in the wad-control-container that shows that it seemed to be updated based on Cerebrata tool.
<?xml version="1.0"?>
<ConfigRequest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<OnDemandTransfers />
<DataSources>
<OverallQuotaInMB>4096</OverallQuotaInMB>
<Logs>
<BufferQuotaInMB>1024</BufferQuotaInMB>
<ScheduledTransferPeriodInMinutes>1</ScheduledTransferPeriodInMinutes>
<ScheduledTransferLogLevelFilter>Undefined</ScheduledTransferLogLevelFilter>
</Logs>
<DiagnosticInfrastructureLogs>
<BufferQuotaInMB>0</BufferQuotaInMB>
<ScheduledTransferPeriodInMinutes>0</ScheduledTransferPeriodInMinutes>
<ScheduledTransferLogLevelFilter>Undefined</ScheduledTransferLogLevelFilter>
</DiagnosticInfrastructureLogs>
<PerformanceCounters>
<BufferQuotaInMB>0</BufferQuotaInMB>
<ScheduledTransferPeriodInMinutes>0</ScheduledTransferPeriodInMinutes>
<Subscriptions />
</PerformanceCounters>
<WindowsEventLog>
<BufferQuotaInMB>0</BufferQuotaInMB>
<ScheduledTransferPeriodInMinutes>0</ScheduledTransferPeriodInMinutes>
<Subscriptions />
<ScheduledTransferLogLevelFilter>Undefined</ScheduledTransferLogLevelFilter>
</WindowsEventLog>
<Directories>
<BufferQuotaInMB>0</BufferQuotaInMB>
<ScheduledTransferPeriodInMinutes>1</ScheduledTransferPeriodInMinutes>
<Subscriptions>
<DirectoryConfiguration>
<Path>C:\Resources\directory\8973cd09642f4dfeafe830612cc8c1fe.AllRole.DiagnosticStore\FailedReqLogFiles</Path>
<Container>wad-iis-failedreqlogfiles</Container>
<DirectoryQuotaInMB>1024</DirectoryQuotaInMB>
</DirectoryConfiguration>
<DirectoryConfiguration>
<Path>C:\Resources\directory\8973cd09642f4dfeafe830612cc8c1fe.AllRole.DiagnosticStore\LogFiles</Path>
<Container>wad-iis-logfiles</Container>
<DirectoryQuotaInMB>1024</DirectoryQuotaInMB>
</DirectoryConfiguration>
<DirectoryConfiguration>
<Path>C:\Resources\directory\8973cd09642f4dfeafe830612cc8c1fe.AllRole.DiagnosticStore\CrashDumps</Path>
<Container>wad-crash-dumps</Container>
<DirectoryQuotaInMB>1024</DirectoryQuotaInMB>
</DirectoryConfiguration>
</Subscriptions>
</Directories>
</DataSources>
<IsDefault>false</IsDefault>
</ConfigRequest>
Any ideas on why it won't seem to work?
UPDATE
We redeployed today with the following diagnostics.wadcfg and still no IISLogs. Trace logs are working. We don't have any code that calls the diagnostics because it was my understanding that the file could handle it all. Am I missing something?
<DiagnosticMonitorConfiguration xmlns="http://schemas.microsoft.com/ServiceHosting/2010/10/DiagnosticsConfiguration"
configurationChangePollInterval="PT1M"
overallQuotaInMB="4096">
<Logs bufferQuotaInMB="1024"
scheduledTransferLogLevelFilter="Verbose"
scheduledTransferPeriod="PT1M" />
<Directories bufferQuotaInMB="1024"
scheduledTransferPeriod="PT1M">
<!-- These three elements specify the special directories
that are set up for the log types -->
<CrashDumps container="wad-crash-dumps" directoryQuotaInMB="256" />
<FailedRequestLogs container="wad-frq" directoryQuotaInMB="256" />
<IISLogs container="wad-iis" directoryQuotaInMB="256" />
</Directories>
</DiagnosticMonitorConfiguration>
Could it be that the web roles are XS instances (since we're just testing right now)? Again, it did work once, but seems to be dead now.
Few suggestions:
Lower the OverallQuotaInMB to something like 4000
Increase
BufferQuotaInMB under the Directories node to a number (say 1 gig)
Lower the other individual Directory quotas so that they add up
to something slightly less than BufferQuotaInMB in #2 and so that
ALL of the quotas (including the overall Directories and and
individual Folders are under OverallQuotaInMB). IE: Logs - 1gig
(this is trace data), Directories: 1 gig, FailedRequests: 256mb,
IISLogs: 256mb, CrashDumps: 256mb
Reboot your servers (just in
case)
Good luck
Basically, I've seen diagnostics act fussy when the overall quota is set to the max space that Azure allocates for diagnostics storage (4gig). Lowering individual quotas so that they add up to something less than total quota also helps because if Azure Diagnostics ever fills up, there is breathing room before Azure removes old data.
Overall, setting up Azure Diagnostics is something of a black magic art. I've been helping AzureWatch customers do this for two years now and I still feel like I'm fumbling with the quotas. Wish they would just let users turn the thing on or off have the entire config be driven by convention vs. configuration. Almost noone ever cares to capture the data onto their VM's and not transfer it to azure storage and thus small quotas are totally fine for majority of the cases as majority of folks transfer their data to storage every few minutes.
HTH
This is my first post to Stack Overflow so please apologies if there is any non-conformity in it.
Question
I have developed a Windows Azure based site (similar to eBay) and hosted it on Azure platform. I have deployed multiple instances of web role with Azure caching enabled. Till last week everything was going fine but suddenly product search page started freezing while loading the data from db. It hangs only for specific categories which returns huge amount of data.
I read somewhere that we should enable localCache and transportProperties if we are expecting large messages. Hence I modified datacache item in my web.config as below but no luck. The page still hangs for those categories!
Could somebody please tell me what is wrong in following and show me some pointers?
<dataCacheClient name="default" channelOpenTimeout="20000" maxConnectionsToServer="4" requestTimeout="30000">
<localCache isEnabled="true" sync="TimeoutBased" ttlValue="300" objectCount="10000"/>
<clientNotification pollInterval="300" maxQueueLength="10000"/>
<transportProperties connectionBufferSize="64000" maxBufferPoolSize="5242880"
maxBufferSize="1242880" maxOutputDelay="2" channelInitializationTimeout="60000"
receiveTimeout="600000"/>
<hosts>
<host name="<<AZURE CACHE URL>>" cachePort="22233" />
</hosts>
<securityProperties mode="Message">
<messageSecurity
authorizationInfo="<<KEY>>">
</messageSecurity>
</securityProperties>
</dataCacheClient>
<dataCacheClient name="SslEndpoint" channelOpenTimeout="20000" maxConnectionsToServer="4" requestTimeout="30000">
<localCache isEnabled="true" sync="TimeoutBased" ttlValue="300" objectCount="10000"/>
<clientNotification pollInterval="300" maxQueueLength="10000"/>
<transportProperties connectionBufferSize="64000" maxBufferPoolSize="15242880"
maxBufferSize="5242880" maxOutputDelay="2" channelInitializationTimeout="60000"
receiveTimeout="600000"/>
<hosts>
<host name="<<AZURE CACHE URL>>" cachePort="22243" />
</hosts>
<securityProperties mode="Message" sslEnabled="true">
<messageSecurity
authorizationInfo="<<KEY>>">
</messageSecurity>
</securityProperties>
</dataCacheClient>
My dev env,
Azure SDK 1.8 (Oct 12), SQL Server 2008 R2, ASP.Net MVC 3
UPDATE
Today I deployed a build with CustomerErrors off to see the if it throws any exception, and this is what I got.
Thanks in advance
ND
I would advice to first find out which component is truly causing your intermittent slowdowns. Is it cache or is it SQL Azure?
If it is indeed cache, and since you're using Azure Shared Cache (previously known as Azure AppFabric Cache)
I would suggest looking at Dedicated cache as a solution instead of Shared cache. Performance of Shared Cache can sometimes be... unpredictable since it is a multi-tenant service and data travels over a network.
I tried the new azure preview that came with the new sdk on my computer.
I put a worker role with cache preview and put co-located role with 30% cache size.
on my controller i put this code:
[OutputCache(Duration=int.MaxValue, VaryByParam="none")]
public ActionResult Index()
{
ViewBag.Message = "Welcome to ASP.NET MVC!";
ViewBag.Id = Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.CurrentRoleInstance.Id;
return View();
}
now i ran the worker role via the emulator with 4 instances. the result was that every time i saw a different id - which mean the output cache never work with all the 4 instances ( to be clear i configure the output cache to work with the cache preview).
Only when i put an extra cache worker role as dedicated role everything start to work like it should be.
My questions is:
Do i need the extra worker role to actually make the cache preview to work ok? - which mean the trade off of not working with azure appfabric cache is putting extra machine
Did i do something work and it should work with the web roles as co located roles?
thanks
edit:
this another section of my web.config
<dataCacheClients>
<tracing sinkType="DiagnosticSink" traceLevel="Error" />
<dataCacheClient name="default">
<autoDiscover isEnabled="true" identifier="NugetTest" />
<!--<localCache isEnabled="true" sync="TimeoutBased" objectCount="100000" ttlValue="300" />
</dataCacheClient>
if my identifier have NugetTest ( which is my web roles - which i have 4) every time i switch machine i get a different cache. if i change the identifier to my worker role i get the result
Can you add applicationName tag in the provider configuration in web.config of you app? If this is not added, instances will not share the cache across. Please note the applicationName tag.
This should be added for the web.config of webrole in both dedicated or colocated cache scenario.
Please reply if this solves your issue.
<caching>
<outputCache defaultProvider="DistributedCache">
<providers>
<add name="DistributedCache" type="Microsoft.Web.DistributedCache.DistributedCacheOutputCacheProvider, Microsoft.Web.DistributedCache" cacheName="<cacheName>" applicationName ="<anyName>" dataCacheClientName="<dataCacheClientName>" />
</providers>
</outputCache>
</caching>
I'm unable to reproduce this issue. I always see the same instance, and I'm using Ctrl+F5 in the browser (thus rule out browser cache). PLease make sure you've configured output cache provider as described on http://www.windowsazure.com/en-us/develop/net/how-to-guides/cache/.
<!-- If output cache content needs to be saved in a Windows Azure
cache, add the following to web.config inside system.web. -->
<caching>
<outputCache defaultProvider="DistributedCache">
<providers>
<add name="DistributedCache"
type="Microsoft.Web.DistributedCache.DistributedCacheOutputCacheProvider, Microsoft.Web.DistributedCache"
cacheName="default"
dataCacheClientName="default" />
</providers>
</outputCache>
</caching>
Best Regards,
Ming Xu.