I've recently tried to upgrade my WebRole from Azure SDK v1.6 to v1.7. This appears to have worked OK. I can build and run the role in my devfabric just fine. When I try to deploy the upgraded project to the real cloud, the instances never start. They just sit in the "busy" state. Interestingly, they don't do the typical "recycle loop", they just sit at "busy" forever.
When I log into the instances with RDP, I see the following error in the event logs:
The application '/' belonging to site '1' has an invalid AppPoolId 'DefaultAppPool' set. Therefore, the application will be ignored.
Followed by:
Site 1 was disabled because the root application defined for the site is invalid. See the previous event log message for information about the root application is invalid.
Looking in IIS manager confirms that there is no AppPool called "DefaultAppPool". There also are none of the typical AppPools with GUIDs for names that Azure creates. Unsurprisingly, none of my sites exist either.
So how do I resolve this?
I had the same issue after upgrading to v1.7, but upon looking at the Windows Azure logs in the Azure VM I noticed the following exception:
An unhandled exception occurred. Type: System.ArgumentException Process ID: 2340
Process Name: DiagnosticsAgent
Thread ID: 1
AppDomain Unhandled Exception for role Backend_IN_0
Exception: Endpoint http://xxxx.blob.core.windows.net/ is not a secure connection.
So I changed the Diagnostics connection string to use https instead of http and voilá, that solved my problem.
Hope that works for you, I've been pulling my hair off for two days.
Related
My IdentityServer being hosted in IIS seems to be recycled due to app_offline.htm according to the Windows application log. It seems to happen sometimes when the Discovery endpoint is being requested, but I can't deduce any obvious pattern to when it does or doesn't happen.
The event description is:
Application 'C:\Users\IP21Admin\Desktop\sts\' was recycled after detecting app_offline.htm.
The last few log entries in the IdentityServer log are:
[2021-01-14 17:01:32 DBG] IdentityServer4.Hosting.EndpointRouter
Request path /.well-known/openid-configuration/jwks matched to endpoint type Discovery
[2021-01-14 17:01:32 DBG] IdentityServer4.Hosting.EndpointRouter
Endpoint enabled: Discovery, successfully created handler: IdentityServer4.Endpoints.DiscoveryKeyEndpoint
[2021-01-14 17:01:32 INF] IdentityServer4.Hosting.IdentityServerMiddleware
Invoking IdentityServer endpoint: IdentityServer4.Endpoints.DiscoveryKeyEndpoint for /.well-known/openid-configuration/jwks
[2021-01-14 17:01:32 DBG] IdentityServer4.Endpoints.DiscoveryKeyEndpoint
Start key discovery request
I found a similar problem someone posted as an issue int he ASPNET core repository here:
https://github.com/dotnet/aspnetcore/issues/21528
I've tried using process monitor to find where app_offline.htm comes from, but every single entry is w2wp.exe operation 'CreateFile' with result 'NAME NOT FOUND'.
Process Monitor Output
I've tried manually putting in my own app_offline.htm and the process monitor will give completely different operations, which leads me to suspect that there wasn't actually an app_offline.htm being created previously. Perhaps the Windows Application event is misleading and there's some sort of crash happening?
Anyone have any suggestions as to what could be wrong?
Edit:
So having looked into this further, the IdentityServer4 instance in IIS seems to get recycled after successfully issuing an Authorisation Code Token from token endpoint. I hooked up remote debugging and saw that when the app is getting recycled, there is 100% CPU usage by the IIS worker, and the process memory of app stays constant; don't know if that's useful information or not.
The symptom of the problem is that when OidcClient of my client GUI application proceeds to get UserInfo from the UserInfo endpoint after getting the Authorisation Code Token; before UserInfo, its attempt to load the discovery document fails due to "Server has been shutdown".
Perhaps the TokenEndpoint causes the middleware to get stuck or something causing a recycle? I assume the server can't respond to the DiscoveryEndpoint request because something in(or after) the TokenEndpoint request caused the server app to get recycled before it could respond to the DiscoveryEndpoint request.
With today's attempts, the log now ends at:
[2021-01-15 13:01:23 DBG] IdentityServer4.Endpoints.TokenEndpoint
Token request success.
[2021-01-15 13:01:23 VRB] IdentityServer4.Hosting.IdentityServerMiddleware
Invoking result: IdentityServer4.Endpoints.Results.TokenResult
Turns out the issue was that IIS automatically recycles the app if any file changes in the directory. I had some log files and SQLite files in the folder that was being written to during requests.
I have a .NET Core app running which is listening to an Azure Service Bus topic.
When I run the app I get this error:
A sessionful message receiver cannot be created on an entity that does
not require sessions. Ensure RequiresSession is set to true when
creating a Queue or Subscription to enable sessionful behavior.
I think the code is running on a separate thread and I can't seem to find the point where the Exception is thrown.
I also don't have a D:\ drive which is where the SessionClient.cs file is located.
Is this D:\ drive in Azure, and if so does this mean there is code running in Azure?
How can I debug this?
Erase your queue and create it again wihtout the 'Require session' checked. Also check your code because maybe you have set the bool RequireSession to true. The fact is that you have enable or disable the sessions once you have configured the queue in the opposite configuration
You're getting the stacktrace that includes the reference of the sources compiled and deployed for that version of the package. The D:\ drive is irrelevant to the issue. What this is is a mismatch how you receive messages and how the entity is configured. You appear to have a sessionless entity, but in the code, you use a receiver that is configured to work with a session.
I have a service fabric cluster which hosts numerous applications. One of the applications has a service type where the service is created, runs for a bit, and then is deleted. Everything works great, but the cluster virtually always has its state set to error because there will be a few of these in the "Unhealthy evaluations" section.
Error event: SourceId='System.Hosting', Property='CodePackageActivation:Code:EntryPoint'.
There was an error during CodePackage activation.The service host terminated with exit code:7148
I've wrapped both the program's main and RunAsync in exception handlers, but never see anything in analytics. Is there any way to look up what exit code 7148 means? Thanks.
7148 is a general error code that indicates that something failed in SF in the process of setting up or activating your service's host process. So that's the reason that you're not seeing any errors or exceptions - your code is never getting a chance to run.
Examples of things I've seen that led to 7148:
The exe was not actually a windows exe due to corruption
The service's manifest had a reference to a cert or some other pre-req like an endpoint that was incorrectly configured (like a port that was already in use or the wrong thumbprint for a cert)
Something blew up inside Windows that cause the process creation to fail, like a failure to correctly configure host networking for a container
Most of the times when I see this I have to look at the windows error logs to see what's really happening. The SF folks are also trying to capture more common causes of failures and reporting them as better health errors rather than relying on 7148.
having an issue with WF4 service running in IIS.
Error in event viewer is "Failed to invoke service menagement endpoint at 'net.pipe://[servername]//servicemenagement.svc' to activate service '/A_WorkflowService.xamlx' \rException; 'The message with To 'net.pipe://[servername]//ServiceManagement.svc' cannot be processed at the receiver, due to an AddressFilter mismatch at the EndpointDispatcher. Check that the sender and receiver's EnpointAddresses agree."
people having the exact same issue, but no solutions
https://social.msdn.microsoft.com/Forums/en-US/88f97778-8bd7-4952-9c3b-943114ddb2d8/failing-to-invoke-appfabric-service-management-service-endpoint?forum=wfprerelease
http://developers.de/blogs/damir_dobric/archive/2010/08/28/appfabric-error-not-able-to-activate-any-of-the-hosts.aspx
https://social.msdn.microsoft.com/Forums/en-US/d2db4da0-b7a3-43d1-98e7-084960cec98e/wms-was-not-able-to-activate-any-of-the-hosts-backing-off?forum=dublin
net.pipe protocol is enabled on the web site and binding is also added in IIS. Basically everything that I found on the web is followed, but I still can't figure out what exactly is that error message trying to say.
According to https://msdn.microsoft.com/en-us/library/ff383422%28v=azure.10%29.aspx - the address of a service supposed to be {scheme}://hostname:port//ServiceManagement.svc and in my error application bit is missing, but for what reason, no idea.
Thanks,
Pav
My Azure web role is constantly recycling.
The WaAppAgent.log file on my Azure web role contains a whole stream of these errors:
[00000008] [05/15/2012 00:10:20.90] GetMachineGoalState() failed with exception: Microsoft.ServiceModel.Web.WebProtocolException: Server Error: Gone (Gone) ---> System.Net.WebException: The remote server returned an error: (410) Gone.
at System.Net.HttpWebRequest.GetResponse()
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
--- End of inner exception stack trace ---
at Microsoft.ServiceModel.Web.WebHttpChannelProxy`1.Invoke(IMessage msg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.WindowsAzure.RoleContainer.Protocol.IControlSystem.GetMachineGoalState()
at Microsoft.WindowsAzure.GuestAgent.ContainerStateMachine.ControlSystem.GetGoalState().
[00000008] [05/15/2012 00:10:20.90] Caught exception in pre-initialization heartbeat thread, will continue heartbeats: System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.WindowsAzure.GuestAgent.ContainerStateMachine.ContainerStateManager.InitialHeartbeatThread(Object waitEvent)
[00000005] [05/15/2012 00:10:23.24] Agent runtime initialized.
The Azure service status dashboard says everything is green on their side, but it always says that unless the entire platform is down in a smoldering heap so I don't really trust it at all.
Is this my problem, or theirs?
I take your statement that you are using Full IIS based Windows Azure Web Role. I have seen such error with VM Role but with Web Role it seems very strange. I wouldn't weight much on your logs either at this time because it may mislead to wrong direction.
If your Web role status shows green (Ready) state but your site is not available then issue could be related to application itself because green or ready state means the role host process (in this case waiishost.exe) is healthy. It would be rare occurrence when role status is green and your role host process is unhealthy. Based on what you have provided either there is an issue within the VM startup sequence means your role not even started however the portal must not show ready state.
Please log into your Azure VM using RDP and..
Check first that WaIISHost.exe process is running or not.. keep an eye in this process for 2 minutes to check if this process is crashing and restarting again
Please check application event log because it should have some pattern of exception to find the root cause
Please check recent Azurebootstrapper and iisconfigurator logs located in C:\logs folder for any specific issue during VM boot and IIS start respectively.
At last please backup the logs (drive C:\logs and C:\Resources), please try rebooting your instance. Still have problem contact Windows Azure support team:
https://www.windowsazure.com/en-us/support/contact/