This is my first post to Stack Overflow so please apologies if there is any non-conformity in it.
Question
I have developed a Windows Azure based site (similar to eBay) and hosted it on Azure platform. I have deployed multiple instances of web role with Azure caching enabled. Till last week everything was going fine but suddenly product search page started freezing while loading the data from db. It hangs only for specific categories which returns huge amount of data.
I read somewhere that we should enable localCache and transportProperties if we are expecting large messages. Hence I modified datacache item in my web.config as below but no luck. The page still hangs for those categories!
Could somebody please tell me what is wrong in following and show me some pointers?
<dataCacheClient name="default" channelOpenTimeout="20000" maxConnectionsToServer="4" requestTimeout="30000">
<localCache isEnabled="true" sync="TimeoutBased" ttlValue="300" objectCount="10000"/>
<clientNotification pollInterval="300" maxQueueLength="10000"/>
<transportProperties connectionBufferSize="64000" maxBufferPoolSize="5242880"
maxBufferSize="1242880" maxOutputDelay="2" channelInitializationTimeout="60000"
receiveTimeout="600000"/>
<hosts>
<host name="<<AZURE CACHE URL>>" cachePort="22233" />
</hosts>
<securityProperties mode="Message">
<messageSecurity
authorizationInfo="<<KEY>>">
</messageSecurity>
</securityProperties>
</dataCacheClient>
<dataCacheClient name="SslEndpoint" channelOpenTimeout="20000" maxConnectionsToServer="4" requestTimeout="30000">
<localCache isEnabled="true" sync="TimeoutBased" ttlValue="300" objectCount="10000"/>
<clientNotification pollInterval="300" maxQueueLength="10000"/>
<transportProperties connectionBufferSize="64000" maxBufferPoolSize="15242880"
maxBufferSize="5242880" maxOutputDelay="2" channelInitializationTimeout="60000"
receiveTimeout="600000"/>
<hosts>
<host name="<<AZURE CACHE URL>>" cachePort="22243" />
</hosts>
<securityProperties mode="Message" sslEnabled="true">
<messageSecurity
authorizationInfo="<<KEY>>">
</messageSecurity>
</securityProperties>
</dataCacheClient>
My dev env,
Azure SDK 1.8 (Oct 12), SQL Server 2008 R2, ASP.Net MVC 3
UPDATE
Today I deployed a build with CustomerErrors off to see the if it throws any exception, and this is what I got.
Thanks in advance
ND
I would advice to first find out which component is truly causing your intermittent slowdowns. Is it cache or is it SQL Azure?
If it is indeed cache, and since you're using Azure Shared Cache (previously known as Azure AppFabric Cache)
I would suggest looking at Dedicated cache as a solution instead of Shared cache. Performance of Shared Cache can sometimes be... unpredictable since it is a multi-tenant service and data travels over a network.
Related
Accordingly to this MSDN article (on AppFabric Caching, which is what Azure is run on), I should be able to find a DistributedCacheService.exe.config file located at \Windows\System32\AppFabric, but it doesn't exist on any of the instances.
When remoting into one of the instances and searching for configs, I find several cache-related config files in E:\plugins\Caching.
The CacheService.config.exe file looks very promising (similar to DistributedCacheService .exe.config), except that the dataCacheConfig is not initialized:
<dataCacheConfig cacheHostName="">
<!-- Comment/uncomment below line to disable/enable file sink logging.
Also location attribute is not honored. It is just specified since its mandatory. -->
<!--<log logLevel="3" location="" />-->
<clusterConfig connectionString="" />
</dataCacheConfig>
I need to confirm that certain data cache settings are being configured properly on the server side in order to solve a previous post of mine.
My client-side web.config looks something like this:
<dataCacheClients>
<dataCacheClient name="DataCache1">
<autoDiscover isEnabled="true" identifier="MyRoleName" />
<transportProperties maxBufferPoolSize="6400000" maxBufferSize="256" />
</dataCacheClient>
<dataCacheClient name="DataCache2">
<autoDiscover isEnabled="true" identifier="MyRoleName" />
<transportProperties maxBufferPoolSize="0" maxBufferSize="10485760" />
</dataCacheClient>
<dataCacheClient name="DataCache3">
<autoDiscover isEnabled="true" identifier="MyRoleName" />
<transportProperties maxBufferPoolSize="3276800" maxBufferSize="32768" />
</dataCacheClient>
</dataCacheClients>
Where do I find the cache host configuration file in Azure In-Role caching (colocated)?
The host property that you configure in on premise AppFabric cache is dynamically initialized in InRole Cache. You can check Caching.csplugin at Program Files\Microsoft SDKs\Windows Azure.NET SDK\v2.2\bin\plugins\Caching to see the endpoints for the cache server.
This is a follow up on this question. I used Cerebrata Diagnostics Manager Remote Diagnostics to try to turn on IIS logs. I hadn't deployed with it on. It seemed to work and a few files were copied. Then it never seemed to work again. I tweaked settings again. I tried deleting the iis related blob and table storage entries to see if that would get it to start over. Here is what the config looks like in the wad-control-container that shows that it seemed to be updated based on Cerebrata tool.
<?xml version="1.0"?>
<ConfigRequest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<OnDemandTransfers />
<DataSources>
<OverallQuotaInMB>4096</OverallQuotaInMB>
<Logs>
<BufferQuotaInMB>1024</BufferQuotaInMB>
<ScheduledTransferPeriodInMinutes>1</ScheduledTransferPeriodInMinutes>
<ScheduledTransferLogLevelFilter>Undefined</ScheduledTransferLogLevelFilter>
</Logs>
<DiagnosticInfrastructureLogs>
<BufferQuotaInMB>0</BufferQuotaInMB>
<ScheduledTransferPeriodInMinutes>0</ScheduledTransferPeriodInMinutes>
<ScheduledTransferLogLevelFilter>Undefined</ScheduledTransferLogLevelFilter>
</DiagnosticInfrastructureLogs>
<PerformanceCounters>
<BufferQuotaInMB>0</BufferQuotaInMB>
<ScheduledTransferPeriodInMinutes>0</ScheduledTransferPeriodInMinutes>
<Subscriptions />
</PerformanceCounters>
<WindowsEventLog>
<BufferQuotaInMB>0</BufferQuotaInMB>
<ScheduledTransferPeriodInMinutes>0</ScheduledTransferPeriodInMinutes>
<Subscriptions />
<ScheduledTransferLogLevelFilter>Undefined</ScheduledTransferLogLevelFilter>
</WindowsEventLog>
<Directories>
<BufferQuotaInMB>0</BufferQuotaInMB>
<ScheduledTransferPeriodInMinutes>1</ScheduledTransferPeriodInMinutes>
<Subscriptions>
<DirectoryConfiguration>
<Path>C:\Resources\directory\8973cd09642f4dfeafe830612cc8c1fe.AllRole.DiagnosticStore\FailedReqLogFiles</Path>
<Container>wad-iis-failedreqlogfiles</Container>
<DirectoryQuotaInMB>1024</DirectoryQuotaInMB>
</DirectoryConfiguration>
<DirectoryConfiguration>
<Path>C:\Resources\directory\8973cd09642f4dfeafe830612cc8c1fe.AllRole.DiagnosticStore\LogFiles</Path>
<Container>wad-iis-logfiles</Container>
<DirectoryQuotaInMB>1024</DirectoryQuotaInMB>
</DirectoryConfiguration>
<DirectoryConfiguration>
<Path>C:\Resources\directory\8973cd09642f4dfeafe830612cc8c1fe.AllRole.DiagnosticStore\CrashDumps</Path>
<Container>wad-crash-dumps</Container>
<DirectoryQuotaInMB>1024</DirectoryQuotaInMB>
</DirectoryConfiguration>
</Subscriptions>
</Directories>
</DataSources>
<IsDefault>false</IsDefault>
</ConfigRequest>
Any ideas on why it won't seem to work?
UPDATE
We redeployed today with the following diagnostics.wadcfg and still no IISLogs. Trace logs are working. We don't have any code that calls the diagnostics because it was my understanding that the file could handle it all. Am I missing something?
<DiagnosticMonitorConfiguration xmlns="http://schemas.microsoft.com/ServiceHosting/2010/10/DiagnosticsConfiguration"
configurationChangePollInterval="PT1M"
overallQuotaInMB="4096">
<Logs bufferQuotaInMB="1024"
scheduledTransferLogLevelFilter="Verbose"
scheduledTransferPeriod="PT1M" />
<Directories bufferQuotaInMB="1024"
scheduledTransferPeriod="PT1M">
<!-- These three elements specify the special directories
that are set up for the log types -->
<CrashDumps container="wad-crash-dumps" directoryQuotaInMB="256" />
<FailedRequestLogs container="wad-frq" directoryQuotaInMB="256" />
<IISLogs container="wad-iis" directoryQuotaInMB="256" />
</Directories>
</DiagnosticMonitorConfiguration>
Could it be that the web roles are XS instances (since we're just testing right now)? Again, it did work once, but seems to be dead now.
Few suggestions:
Lower the OverallQuotaInMB to something like 4000
Increase
BufferQuotaInMB under the Directories node to a number (say 1 gig)
Lower the other individual Directory quotas so that they add up
to something slightly less than BufferQuotaInMB in #2 and so that
ALL of the quotas (including the overall Directories and and
individual Folders are under OverallQuotaInMB). IE: Logs - 1gig
(this is trace data), Directories: 1 gig, FailedRequests: 256mb,
IISLogs: 256mb, CrashDumps: 256mb
Reboot your servers (just in
case)
Good luck
Basically, I've seen diagnostics act fussy when the overall quota is set to the max space that Azure allocates for diagnostics storage (4gig). Lowering individual quotas so that they add up to something less than total quota also helps because if Azure Diagnostics ever fills up, there is breathing room before Azure removes old data.
Overall, setting up Azure Diagnostics is something of a black magic art. I've been helping AzureWatch customers do this for two years now and I still feel like I'm fumbling with the quotas. Wish they would just let users turn the thing on or off have the entire config be driven by convention vs. configuration. Almost noone ever cares to capture the data onto their VM's and not transfer it to azure storage and thus small quotas are totally fine for majority of the cases as majority of folks transfer their data to storage every few minutes.
HTH
History
In my development environment I've battled a reoccurring error with running Windows Azure Emulator. To resolve before I created a new Solution and projects based on the Azure template, along with reinstalling the October 2012 update SDK. Then I had to copy over all the files into the new projects. Before going through all that, since I know it can work, I wanted to see if there was something else I can do.
Setup
Windows 8
Visual Studio 2012 Update 1
IIS Express 8
Windows Azure Tools - October 2012
Windows Azure Caching 1.8.1.0
ASP.NET MVC 4.0 project
Web Role (2 Instances)
Cache Worker Role (2 Instances)
Caching Enabled on Cache Work Roles and set to Dedicated Role (Local Role state is: UseDevelopmentStorage=true)
Session uses Distributed Cache
Output Cache uses Distributed Cache
Web.config settings
<sessionState mode="Custom" customProvider="AFCacheSessionStateProvider" cookieName="abc_session">
<providers>
<add name="AFCacheSessionStateProvider" type="Microsoft.Web.DistributedCache.DistributedCacheSessionStateStoreProvider, Microsoft.Web.DistributedCache" cacheName="default" dataCacheClientName="default" applicationName="AFCacheSessionState" />
</providers>
</sessionState>
<outputCache defaultProvider="AFCacheOutputCacheProvider">
<providers>
<add name="AFCacheOutputCacheProvider" type="Microsoft.Web.DistributedCache.DistributedCacheOutputCacheProvider, Microsoft.Web.DistributedCache" cacheName="default" dataCacheClientName="default" applicationName="AFCacheOutputCache" />
</providers>
</outputCache>
</caching>
<dataCacheClients>
<dataCacheClient name="default">
<autoDiscover isEnabled="true" identifier="AppNameCacheWorkerRole" />
</dataCacheClient>
</dataCacheClients>
Error Details
ErrorCode<ERRCA0009>:SubStatus<ES0001>:Cache referred to does not exist
[DataCacheException: ErrorCode<ERRCA0009>:SubStatus<ES0001>:Cache referred to does not exist. Contact administrator or use the Cache administration tool to create a Cache.]
Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ErrStatus errStatus, Guid trackingId, Exception responseException, Byte[][] payload, EndpointID destination) +767
Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody, EndpointID destination) +149
Microsoft.ApplicationServer.Caching.DataCacheFactory.EstablishConnection(IEnumerable`1 servers, RequestBody request, Func`3 sendMessageDelegate, DataCacheReadyRetryPolicy retryPolicy) +967
Microsoft.ApplicationServer.Caching.<>c__DisplayClass4.<Initialize>b__1(RequestBody req) +198
Microsoft.ApplicationServer.Caching.SocketClientProtocol.SendReceive(IVelocityRequestPacket request, Func`2 delegate, EndpointID& destination) +121
Microsoft.ApplicationServer.Caching.SocketClientProtocol.Initialize(IEnumerable`1 servers) +1011
Microsoft.ApplicationServer.Caching.DataCacheFactory.GetCache(String cacheName, CreateNewCacheDelegate cacheCreationDelegate, DataCacheInitializationViaCopyDelegate initializeDelegate) +1103
Microsoft.ApplicationServer.Caching.DataCacheFactory.GetCache(String cacheName) +131
Microsoft.Web.DistributedCache.DataCacheFactoryWrapper.CreateDataCacheFromFactory(DataCacheFactory factory, String cacheName) +63
Microsoft.Web.DistributedCache.CacheHelpers.RunCacheCreationHooks(CacheConnectingEventArgs fetchingEventArgs, IDataCacheFactory dataCacheFactory, Object sender, EventHandler`1 fetchingHandler, EventHandler`1 fetchedHandler) +356
Microsoft.Web.DistributedCache.DistributedCacheOutputCacheProvider.CreateInternalProvider(IHttpRuntime httpRuntime, OutputCacheInitializationData initData, IDataCacheFactory dataCacheFactory, EventHandler`1 cacheFetching, EventHandler`1 cacheFetched) +399
Microsoft.Web.DistributedCache.DistributedCacheOutputCacheProvider.GetInternalProvider() +315
Microsoft.Web.DistributedCache.DistributedCacheOutputCacheProvider.Get(String key) +54
System.Web.Caching.OutputCache.Get(String key) +80
System.Web.Caching.OutputCacheModule.OnEnter(Object source, EventArgs eventArgs) +341
System.Web.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +80
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +165
Misc Notes:
The same configuration works fine when it is deployed to Azure, so has something to do with my local environment.
IIS Express was crashing when I had more than one instance of web roles setup, and I found a suggestion to run the Visual Studio 2010 SP1 to resolve. I had VS2010 on the system because one tools (believe it was SQL) installed it. Installing SP1 resolved the issue with IIS Express crashing, but then this error reappeared.
If I comment out the OutputCache node listed above the error goes away.
If I switch the Cache Worker Role from Dedicated Role to Co-located Role the error goes away.
Spoke with Microsoft tech support and it appears this a known issue. Most of the time it can be resolved by forcing a refresh in the browser. I've had mixed success. The fix for this issue will be part of the next Azure SDK release.
This happens to me in production a lot! I upload my package to staging first, and have to always wait until all the instances are running before hitting the staging URL. Sometimes this error still occurs and I restart all my instances for it to fix itself.
I'm creating website with custom application pool settings using WIX. After installation my new website is using Default Application Pool instead of using the one that I'm creating during installation.
I could find ways for virtual directory to point to newly created app pool but seems iis:WebSite does not have attribute for setting app pool.
Here is my code:
<iis:WebSite Id="MyWebSite" Description='MyWebsiteDesc' SiteId='*' Directory='MyWebFolder'>
<iis:WebAddress Id="AllUnassigned" Port="80" />
</iis:WebSite>
<util:User Id="MyAppPoolUser"
CreateUser="no"
Name="[APPPOOL_USER]"
Password="[APPPOOL_PASS]"
Domain="." />
<iis:WebAppPool Id="MyAppPool"
Name="[WEB_APP_NAME]"
Identity="other"
User="MyAppPoolUser"
ManagedRuntimeVersion="v4.0"
ManagedPipelineMode="Integrated" />
Use IIS:WebVirtualDir/IIS:WebApplication to point to your AppPool.
Hope, this might help you.
You have created an application pool but you have only created a web site. To use the application pool, you must create a virtual directory or application within the website. Both the WebVirtualDir and WebApplication elements have a WebAppPool attribute that you can use to configure IIS to use the application pool you are creating.
I recently faced this issue. All my web application inside website were assigned new app pool I created but base website was having DefaultAppPool.
This is the trick to assign app pool to base website
<iis:WebAppPool Id="MyAppPool" Name="YourAppPoolName" Identity="applicationPoolIdentity" />
<iis:WebSite Id='MyWebSite' Description='Dummy' Directory='APPLICATIONFOLDER'>
<iis:WebApplication Id="AnyId" Name="Dummy" WebAppPool="MyAppPool" />
</iis:WebSite>
I tried the new azure preview that came with the new sdk on my computer.
I put a worker role with cache preview and put co-located role with 30% cache size.
on my controller i put this code:
[OutputCache(Duration=int.MaxValue, VaryByParam="none")]
public ActionResult Index()
{
ViewBag.Message = "Welcome to ASP.NET MVC!";
ViewBag.Id = Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.CurrentRoleInstance.Id;
return View();
}
now i ran the worker role via the emulator with 4 instances. the result was that every time i saw a different id - which mean the output cache never work with all the 4 instances ( to be clear i configure the output cache to work with the cache preview).
Only when i put an extra cache worker role as dedicated role everything start to work like it should be.
My questions is:
Do i need the extra worker role to actually make the cache preview to work ok? - which mean the trade off of not working with azure appfabric cache is putting extra machine
Did i do something work and it should work with the web roles as co located roles?
thanks
edit:
this another section of my web.config
<dataCacheClients>
<tracing sinkType="DiagnosticSink" traceLevel="Error" />
<dataCacheClient name="default">
<autoDiscover isEnabled="true" identifier="NugetTest" />
<!--<localCache isEnabled="true" sync="TimeoutBased" objectCount="100000" ttlValue="300" />
</dataCacheClient>
if my identifier have NugetTest ( which is my web roles - which i have 4) every time i switch machine i get a different cache. if i change the identifier to my worker role i get the result
Can you add applicationName tag in the provider configuration in web.config of you app? If this is not added, instances will not share the cache across. Please note the applicationName tag.
This should be added for the web.config of webrole in both dedicated or colocated cache scenario.
Please reply if this solves your issue.
<caching>
<outputCache defaultProvider="DistributedCache">
<providers>
<add name="DistributedCache" type="Microsoft.Web.DistributedCache.DistributedCacheOutputCacheProvider, Microsoft.Web.DistributedCache" cacheName="<cacheName>" applicationName ="<anyName>" dataCacheClientName="<dataCacheClientName>" />
</providers>
</outputCache>
</caching>
I'm unable to reproduce this issue. I always see the same instance, and I'm using Ctrl+F5 in the browser (thus rule out browser cache). PLease make sure you've configured output cache provider as described on http://www.windowsazure.com/en-us/develop/net/how-to-guides/cache/.
<!-- If output cache content needs to be saved in a Windows Azure
cache, add the following to web.config inside system.web. -->
<caching>
<outputCache defaultProvider="DistributedCache">
<providers>
<add name="DistributedCache"
type="Microsoft.Web.DistributedCache.DistributedCacheOutputCacheProvider, Microsoft.Web.DistributedCache"
cacheName="default"
dataCacheClientName="default" />
</providers>
</outputCache>
</caching>
Best Regards,
Ming Xu.