Strange Azure Service bus behaviour - subscription #2 is receiving message while subscription #1 is not - azure

I am having some very strange issues with Azure Service Bus topic subscriptions. For the reference I have extracted the service bus topic configurations via service bus explorer and hoping somebody could point me into the right direction here. Really appreciate any help I can get.
Setup
I have 1 topic named topic2 and two subscription for this topic topicsub2 and topicsub3.
Issues
When publishing message to topic. Subscription topicsub3 recieves all messages that are sent while topicsub2 does not receive any messages. This behaviour is random, sometimes I would see messages in topicsub2 and sometimes the message is never delivered to the topicsub2 both subscriptions are using the same filters.
Followings are the few test scenarios that I have tried to figure out why this is happening.
I tried removing the filter from topicsub2 and sending message. But this had no impact still no messages. My assumption is that if no filters are applied all message for the topic will be forwarded to the subscription.
After removing the filter, I added default SQL filter (1=1) filter back to the subscription and tried sending message again it didn't work. I tried sending 5 messages to the topic, none of them appeared in the subscription topicsub2 and all were received by topicsub3.
I restarted the console application receiver (console application that reads from topicsub2) and tried sending messages again. This time it worked.
After about 10 mins, the subscription topicsub2 stopped receiving messages again.
Summary
This is where I am at. My investigation leads me this behaviour is random and there is some idling going on in topicsub2 that I am unaware of. Topicsub3 is working fine.
Can somebody please lead me into the right direction?
Following is the settings for topic 2 including the two subscriptions.
<?xml version="1.0" encoding="UTF-8"?>
<Entities xmlns="http://schemas.microsoft.com/servicebusexplorer" serviceBusNamespace="defaultstd">
<Topics>
<Topic>
<DefaultMessageTimeToLive>14.00:00:00</DefaultMessageTimeToLive>
<AutoDeleteOnIdle>10675199.02:48:05.4775807</AutoDeleteOnIdle>
<MaxSizeInMegabytes>1024</MaxSizeInMegabytes>
<RequiresDuplicateDetection>False</RequiresDuplicateDetection>
<DuplicateDetectionHistoryTimeWindow>00:10:00</DuplicateDetectionHistoryTimeWindow>
<Path>topic2</Path>
<EnableBatchedOperations>True</EnableBatchedOperations>
<SupportOrdering>False</SupportOrdering>
<EnableFilteringMessagesBeforePublishing>True</EnableFilteringMessagesBeforePublishing>
<IsAnonymousAccessible>False</IsAnonymousAccessible>
<Status>Active</Status>
<UserMetadata />
<EnablePartitioning>True</EnablePartitioning>
<EnableExpress>False</EnableExpress>
<IsReadOnly>False</IsReadOnly>
<Subscriptions>
<Subscription>
<LockDuration>00:00:30</LockDuration>
<RequiresSession>False</RequiresSession>
<DefaultMessageTimeToLive>14.00:00:00</DefaultMessageTimeToLive>
<AutoDeleteOnIdle>10675199.02:48:05.4775807</AutoDeleteOnIdle>
<EnableDeadLetteringOnMessageExpiration>True</EnableDeadLetteringOnMessageExpiration>
<EnableDeadLetteringOnFilterEvaluationExceptions>True</EnableDeadLetteringOnFilterEvaluationExceptions>
<TopicPath>topic2</TopicPath>
<Name>topicsub2</Name>
<MaxDeliveryCount>2</MaxDeliveryCount>
<EnableBatchedOperations>False</EnableBatchedOperations>
<Status>Active</Status>
<ForwardTo />
<ForwardDeadLetteredMessagesTo />
<UserMetadata />
<IsReadOnly>False</IsReadOnly>
<Rules>
<Rule>
<Filter>
<SqlFilter>
<SqlExpression>sys.Label NOT IN ('TestEvent')</SqlExpression>
<CompatibilityLevel>20</CompatibilityLevel>
</SqlFilter>
</Filter>
<Action>
<EmptyRuleAction />
</Action>
<Name>$Default</Name>
<IsReadOnly>True</IsReadOnly>
</Rule>
</Rules>
</Subscription>
<Subscription>
<LockDuration>00:00:30</LockDuration>
<RequiresSession>False</RequiresSession>
<DefaultMessageTimeToLive>14.00:00:00</DefaultMessageTimeToLive>
<AutoDeleteOnIdle>10675199.02:48:05.4775807</AutoDeleteOnIdle>
<EnableDeadLetteringOnMessageExpiration>True</EnableDeadLetteringOnMessageExpiration>
<EnableDeadLetteringOnFilterEvaluationExceptions>True</EnableDeadLetteringOnFilterEvaluationExceptions>
<TopicPath>topic2</TopicPath>
<Name>topicsub3</Name>
<MaxDeliveryCount>2</MaxDeliveryCount>
<EnableBatchedOperations>False</EnableBatchedOperations>
<Status>Active</Status>
<ForwardTo />
<ForwardDeadLetteredMessagesTo />
<UserMetadata />
<IsReadOnly>False</IsReadOnly>
<Rules>
<Rule>
<Filter>
<SqlFilter>
<SqlExpression>sys.Label NOT IN ('TestEvent')</SqlExpression>
<CompatibilityLevel>20</CompatibilityLevel>
</SqlFilter>
</Filter>
<Action>
<EmptyRuleAction />
</Action>
<Name>$Default</Name>
<IsReadOnly>True</IsReadOnly>
</Rule>
</Rules>
</Subscription>
</Subscriptions>
</Topic>
</Topics>
</Entities>

Related

IIS 10 application pool slow initialization times after recycle

we have an application pool that has a slower initialization time after an app pool recycle in IIS 10. (around 5-7 seconds after recycle then 30-50 ms after the first request.)
I have done some research and found that the "Application Initialization." module should do the trick.
I installed it onto the server and set the application pool to "AlwaysRunning" and the corresponding site to "PreloadEnabled == True." After making those changes we tested by recycling and the response times seemed a bit better...down to 3 to 4 seconds after recycle. I tried to then disable the "overlapped recycle" to see if that helped and again it did a bit better 1.5 to 2 seconds after recycle and then 20 to 30 ms after the first request.
Question is: Is that the best we can expect? I was hoping there would be away to fully pre-warm the app pool so that even the first request is around a few ms. The issue is that test messages we are sending to the API are small and the ones in Prod would be much larger so an initialization of 3-4 seconds could be much much longer in Prod.
Following are the steps you can perform to Auto Initialize application hosted on IIS –
• Installed Application Initialization feature - IIS 8.0 Application
https://learn.microsoft.com/en-us/iis/get-started/whats-new-in-iis-8/iis-80-application-initialization
• Make sure the warmup.dll (which should load from C:\Windows\SysWOW64\inetsrv\warmup.dll or from C:\Windows\system32\inetsrv\warmup.dll depending on the bitness of your process) present
• Configure the app pool to be always running (from the advanced properties)
eg. In the applicationHost.config (%WINDIR%\system32\inetsrv\config\applicationHost.config) file the application pool setting looks like this –
<add name="PreLoadApp" autoStart="true" managedRuntimeVersion="" startMode="AlwaysRunning">
<processModel idleTimeout="00:00:00" />
</add>
• Scroll down a little more in applicationHost.config to the configuration element. Within that section there will be an entry, modify your application as below
<site name="PreLoadApp" id="5">
<application path="/" applicationPool="PreLoadApp" preloadEnabled="true">
<virtualDirectory path="/" physicalPath="C:\inetpub\wwwroot\PreLoadApp" />
</application>
• Then selected the site from the IIS manager tree view on the left-hand side and go to the configuration editor.
this time underneath the <system.WebServer/applicationInitialization>
tag, and look at the list of requests
set the request to only target a page called (host header is optional) and also you can provide a query string
q=abhi // to identify if request is coming from preload only.
set the doAppInitAfterRestart parameter to true and apply the
settings
And you should be good by now, try recycling your application pool, it should Initialize and warmup automatically.
You can refer these MS docs to know more about Application Initialization and configuration steps –
https://learn.microsoft.com/en-us/iis/configuration/system.webServer/applicationInitialization/
https://learn.microsoft.com/en-us/iis/get-started/whats-new-in-iis-8/iis-80-application-initialization
IIS saves doAppInitAfterRestart in the web.config of the application, which might be overwritten by future deployments. Therefore I'd put the web.config under source control and make it part of the deployed artifact.
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.webServer>
<handlers>
<add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
</handlers>
<aspNetCore processPath="%LAUNCHER_PATH%" arguments="%LAUNCHER_ARGS%" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" hostingModel="inprocess" />
<!-- Add this node to web.config -->
<applicationInitialization doAppInitAfterRestart="true">
<!-- Only needed when website contains multiple child apps -->
<add initializationPage='/hangfire',hostname='' />
<applicationInitialization />
</system.webServer>
</configuration>

Implementing Gateway causing Problems in Message Flow

I have a scenario in which i am receiving a message and routing it to different message channels using Recipient-list-router.
<int:recipient-list-router input-channel="inputChannel1"
apply-sequence="true" id="router1">
<int:recipient channel="service1" />
<int:recipient channel="service2" />
</int:recipient-list-router>
This aggregator is used to aggregate the response from above 2 services and redirects the aggregated output to some other channel for further processing.
<int:aggregator input-channel="aggregator1" output-channel="channel2" >
<bean class="wsintg.service.AggregatorClass" />
</int:aggregator>
The code for individual services looks like this :
<!--service 1 code -->
<int:service-activator input-channel="service1" ref="service1Ref" />
<int:gateway id="service1Ref" default-request-channel="success_channel" error-channel="error_channel" />
<int:chain id="chainid1" input-channel="success_channel" output-channel="aggregator1">
step 1 :xsl transformation
step 2 :call to a service using ws:outbound-gateway
<int:service-activator id="sa1">
<bean class="samplePkg.returnMessage"></bean> <!--return message simply returns the message without doing anything -->
</int:service-activator>
</int:chain>
<int:chain id="chainid2" input-channel="error_channel" output-channel="aggregator1" >
<int:service-activator>
<!-- some custom exception handling -->
</int:service-activator>
</int:chain>
<!--service 2 code --> same as service 1
Problem :
If i don't use gateway in Service 1 and Service 2 and simply do the transformations, the code works perfectly.
But, as i need to do some error handling using error_channel, so i put a gateway in service1 and service 2 code. Now the flow
stucks after service activator in service 1 without throwing any
exception. and does not calls the service 2 flow.
You need to show real configuration, not just comments; we can't guess what your application looks like.
That said, you can't just send error messages to the aggregator because the aggregation information is in the failedMessage headers and needs to be promoted.
See error handling in this sample for an example.
As always, the first step to debug situations like this is to turn on DEBUG logging for org.springframework.integration and follow the messages through the flow.
You will usually be able to determine the problem that way.

Why aren't my IIS Logs being copied for my Azure web role?

This is a follow up on this question. I used Cerebrata Diagnostics Manager Remote Diagnostics to try to turn on IIS logs. I hadn't deployed with it on. It seemed to work and a few files were copied. Then it never seemed to work again. I tweaked settings again. I tried deleting the iis related blob and table storage entries to see if that would get it to start over. Here is what the config looks like in the wad-control-container that shows that it seemed to be updated based on Cerebrata tool.
<?xml version="1.0"?>
<ConfigRequest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<OnDemandTransfers />
<DataSources>
<OverallQuotaInMB>4096</OverallQuotaInMB>
<Logs>
<BufferQuotaInMB>1024</BufferQuotaInMB>
<ScheduledTransferPeriodInMinutes>1</ScheduledTransferPeriodInMinutes>
<ScheduledTransferLogLevelFilter>Undefined</ScheduledTransferLogLevelFilter>
</Logs>
<DiagnosticInfrastructureLogs>
<BufferQuotaInMB>0</BufferQuotaInMB>
<ScheduledTransferPeriodInMinutes>0</ScheduledTransferPeriodInMinutes>
<ScheduledTransferLogLevelFilter>Undefined</ScheduledTransferLogLevelFilter>
</DiagnosticInfrastructureLogs>
<PerformanceCounters>
<BufferQuotaInMB>0</BufferQuotaInMB>
<ScheduledTransferPeriodInMinutes>0</ScheduledTransferPeriodInMinutes>
<Subscriptions />
</PerformanceCounters>
<WindowsEventLog>
<BufferQuotaInMB>0</BufferQuotaInMB>
<ScheduledTransferPeriodInMinutes>0</ScheduledTransferPeriodInMinutes>
<Subscriptions />
<ScheduledTransferLogLevelFilter>Undefined</ScheduledTransferLogLevelFilter>
</WindowsEventLog>
<Directories>
<BufferQuotaInMB>0</BufferQuotaInMB>
<ScheduledTransferPeriodInMinutes>1</ScheduledTransferPeriodInMinutes>
<Subscriptions>
<DirectoryConfiguration>
<Path>C:\Resources\directory\8973cd09642f4dfeafe830612cc8c1fe.AllRole.DiagnosticStore\FailedReqLogFiles</Path>
<Container>wad-iis-failedreqlogfiles</Container>
<DirectoryQuotaInMB>1024</DirectoryQuotaInMB>
</DirectoryConfiguration>
<DirectoryConfiguration>
<Path>C:\Resources\directory\8973cd09642f4dfeafe830612cc8c1fe.AllRole.DiagnosticStore\LogFiles</Path>
<Container>wad-iis-logfiles</Container>
<DirectoryQuotaInMB>1024</DirectoryQuotaInMB>
</DirectoryConfiguration>
<DirectoryConfiguration>
<Path>C:\Resources\directory\8973cd09642f4dfeafe830612cc8c1fe.AllRole.DiagnosticStore\CrashDumps</Path>
<Container>wad-crash-dumps</Container>
<DirectoryQuotaInMB>1024</DirectoryQuotaInMB>
</DirectoryConfiguration>
</Subscriptions>
</Directories>
</DataSources>
<IsDefault>false</IsDefault>
</ConfigRequest>
Any ideas on why it won't seem to work?
UPDATE
We redeployed today with the following diagnostics.wadcfg and still no IISLogs. Trace logs are working. We don't have any code that calls the diagnostics because it was my understanding that the file could handle it all. Am I missing something?
<DiagnosticMonitorConfiguration xmlns="http://schemas.microsoft.com/ServiceHosting/2010/10/DiagnosticsConfiguration"
configurationChangePollInterval="PT1M"
overallQuotaInMB="4096">
<Logs bufferQuotaInMB="1024"
scheduledTransferLogLevelFilter="Verbose"
scheduledTransferPeriod="PT1M" />
<Directories bufferQuotaInMB="1024"
scheduledTransferPeriod="PT1M">
<!-- These three elements specify the special directories
that are set up for the log types -->
<CrashDumps container="wad-crash-dumps" directoryQuotaInMB="256" />
<FailedRequestLogs container="wad-frq" directoryQuotaInMB="256" />
<IISLogs container="wad-iis" directoryQuotaInMB="256" />
</Directories>
</DiagnosticMonitorConfiguration>
Could it be that the web roles are XS instances (since we're just testing right now)? Again, it did work once, but seems to be dead now.
Few suggestions:
Lower the OverallQuotaInMB to something like 4000
Increase
BufferQuotaInMB under the Directories node to a number (say 1 gig)
Lower the other individual Directory quotas so that they add up
to something slightly less than BufferQuotaInMB in #2 and so that
ALL of the quotas (including the overall Directories and and
individual Folders are under OverallQuotaInMB). IE: Logs - 1gig
(this is trace data), Directories: 1 gig, FailedRequests: 256mb,
IISLogs: 256mb, CrashDumps: 256mb
Reboot your servers (just in
case)
Good luck
Basically, I've seen diagnostics act fussy when the overall quota is set to the max space that Azure allocates for diagnostics storage (4gig). Lowering individual quotas so that they add up to something less than total quota also helps because if Azure Diagnostics ever fills up, there is breathing room before Azure removes old data.
Overall, setting up Azure Diagnostics is something of a black magic art. I've been helping AzureWatch customers do this for two years now and I still feel like I'm fumbling with the quotas. Wish they would just let users turn the thing on or off have the entire config be driven by convention vs. configuration. Almost noone ever cares to capture the data onto their VM's and not transfer it to azure storage and thus small quotas are totally fine for majority of the cases as majority of folks transfer their data to storage every few minutes.
HTH

Exception Handling for Service Bus Queue Host Service

I have a WCF service that is connected to a service bus Queue ready to receive messages. This is working great but i would like to be able to mark the message as a DeadLetter if i have an issue processing the message. Currently if my code throws an exception the message still gets removed from the queue, but i want to be able in configuration to specify to not deleted from the queue but mark it as a DeadLetter. I've done some search and I can't figure out on how to do that. I am currently running the service as a windows service
Uri baseAddress = ServiceBusEnvironment.CreateServiceUri("sb",
"namespace", "servicequeue");
_serviceHost = new ServiceHost(typeof(PaperlessImportServiceOneWay), baseAddress);
_serviceHost.Open();
config:
<services>
<service name="Enrollment.ServiceOneWay">
<endpoint name="ServiceOneWay"
address="sb://namespace.servicebus.windows.net/servicequeue"
binding="netMessagingBinding"
bindingConfiguration="messagingBinding"
contract="IServiceOneWaySoap"
behaviorConfiguration="sbTokenProvider" />
</service>
</services>
<netMessagingBinding>
<binding name="messagingBinding" closeTimeout="00:03:00" openTimeout="00:03:00"
receiveTimeout="00:03:00" sendTimeout="00:03:00" sessionIdleTimeout="00:01:00"
prefetchCount="-1">
<transportSettings batchFlushInterval="00:00:01" />
</binding>
</netMessagingBinding>
<behavior name="sbTokenProvider">
<transportClientEndpointBehavior>
<tokenProvider>
<sharedSecret issuerName="owner" issuerSecret="XXXXXXXXXXXXXXXXXXXXXXXX" />
</tokenProvider>
</transportClientEndpointBehavior>
</behavior>
In your interface for the opertion Contract add this
[ReceiveContextEnabled(ManualControl = true)]
then you can manage to commit or abandon the message
Found it in this link:
http://msdn.microsoft.com/en-us/library/windowsazure/hh532034.aspx

nservicebus using generic host with azure queues causes msmq error

I have an on premise service bus that is configured to handle messages from an azure queue. The problem i am having is that the host is reporting an msmq error saying that it could not create the error queue. Aside from the fact that it should not be using msmq, it also handles the messages with no problems despite the error so it does not seem to be critical.
My Host is running as a class library configured to start with the nservicebus.host.exe process.
Here is my host code and config:
internal class EndpointConfig : IConfigureThisEndpoint, AsA_Server, IWantCustomInitialization
{
#region IWantCustomInitialization Members
public void Init()
{
Configure.With()
.DefaultBuilder()
.AzureMessageQueue()
.JsonSerializer()
.UnicastBus()
.IsTransactional(true)
.InMemorySubscriptionStorage();
}
#endregion
}
Config:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<configSections>
<section name="UnicastBusConfig" type="NServiceBus.Config.UnicastBusConfig, NServiceBus.Core" />
<section name="AzureQueueConfig" type="NServiceBus.Config.AzureQueueConfig, NServiceBus.Azure"/>
<section name="MessageForwardingInCaseOfFaultConfig" type="NServiceBus.Config.MessageForwardingInCaseOfFaultConfig, NServiceBus.Core" />
</configSections>
<MessageForwardingInCaseOfFaultConfig ErrorQueue="error" />
<AzureQueueConfig QueueName="sender" ConnectionString="UseDevelopmentStorage=true" PeekInterval="5000" MaximumWaitTimeWhenIdle="60000" />
<startup useLegacyV2RuntimeActivationPolicy="true">
<supportedruntime version="v4.0" />
<requiredruntime version="v4.0.20506" />
</startup>
</configuration>
And Here is the actual Error Message:
2012-04-24 07:57:10,973 [1] ERROR NServiceBus.Utils.MsmqUtilities [(null)] <(nul
l)> - Could not create queue error#UseDevelopmentStorage=true or check its exist
ence. Processing will still continue.
System.Messaging.MessageQueueException (0x80004005): Message Queue service is no
t available.
at System.Messaging.MessageQueue.Create(String path, Boolean transactional)
at NServiceBus.Utils.MsmqUtilities.CreateQueue(String queueName, String accou
nt)
at NServiceBus.Utils.MsmqUtilities.CreateQueueIfNecessary(Address address, St
ring account)
EDIT: Adding .MessageForwardingInCaseOfFault() to the initialization corrected the issue.
Looks like AsA_Server assumes msmq, guess you'll have to configure the process manually
Adding .MessageForwardingInCaseOfFault() to the init method resolved the issue. Still feels like there is an underlying bug, but it is working.
I suspect that below described the next hurdle (not handling errors correctly) but i will have to try to force a failed message to verify.
As described in:
NServiceBus error queues in Azure

Resources