My Azure Functions were running fine and all of a sudden I am getting several "Assembly changes detected. Restarting host..." messages that is preventing my functions from completing.
I am not deploying new code so not sure what is triggering the Assembly Change event to fire. I was running on the latest version of the runtime and have since reverted to version 1.0.10947 thinking that maybe the underlying runtime was updated, but I'm still getting that line showing up in the logs.
Update
Now that #Alexey has helped me track down what is causing the Assembly changes to be detected. I would like to ask if anyone can tell me WHY an assembly change is being detected even-tough I have not changed/redeployed my application.
After looking in your logs we opened an issue https://github.com/Azure/azure-webjobs-sdk-script/issues/1533#issuecomment-303595960.
Your functions had multiple restores but now issue is gone. Restores could be initiated by changing project.json.
If you are stuck with the multiple
Assembly changes detected. Restarting host
I fixed my issue by deleted the log file in the Kudu services:
https://[FunctionAppName].scm.azurewebsites.net/
and follow on the top menu:
Debug Console >> powerShell
And the file log is :
LogFiles >> Application >> Functions >> function >> [Function name]
You can remove the log file.
my 2c.
I was struggling with this issue for ages and not sure what was causing it. I believe I may have the answer.
Our solution has been toying with consumption plans, but pulled back to full App Service Plans because the initiation times were too long for our rather unique usage patterns.
But 2 of the appsetting params were still in place: WEBSITE_CONTENTSHARE And WEBSITE_CONTENTAZUREFILECONNECTIONSTRING.
per:
https://learn.microsoft.com/en-us/azure/azure-functions/functions-app-settings#websitecontentazurefileconnectionstring
these are ONLY for consumption plans.
I removed them and... touch wood, the issue seems to be resolved.
Related
I have a Azure App Service app that I'm trying to get deployed.
Today I ran into an issue where .NET informed me (via the yellow screen of death when I browse to the URL of my app) that I had a missing DLL (for the purposes of this question I don't think it really matters).
I used FileZilla to publish my changes in an attempt to do a manual deployment first and then work my way to automate it.
After so many attempts to fix it I later realized that the error message never changed. I did something more severe and renamed my bin folder into something completely different and the exact same error message would appear.
I've stopped the service, restarted it, and as mentioned, renamed folders, etc. and still the exact same error message persisted.
I also decided to open up the Azure Portal Console for my App Service app to browse a bit and to my amazement, nothing seemed to have reflected at all. The FTP shows one thing and the Console shows another.
Would anyone have any idea as to why this is happening?
I eventually got it to work and I will share what I tried.
I deleted the web app and created it again (I found this to be important the first time around). This was quite time consuming and did help but it wasn't long before the same problem happened again.
Then I finally found a solution that seems to give me consistent results:
I kept on editing the Web.config which seems to force a recompile and clear some sort of cache. So each time the web app stopped updating, I would make a slight change in the Web.config, upload it via FTP and the app finally updates.
If anyone has any more details on this, it would be greatly appreciated.
Right now I again faced the issue that old code is used on an Azure Function App even after the zip deployment through KUDU returns success.
Of course, that is after some 30 mins that I expect the new code to get loaded, not immediately.
The issue is marked as closed.
What is considered to be the best practice in this case:
Programmatically force the Function App to restart, say, through Azure CLI or Powershell Az modules?
Or there is another way to mitigate the issue?
While restarting should fix it, my suggestion would be to enable "Run from package": https://learn.microsoft.com/en-us/azure/azure-functions/run-functions-from-deployment-package. That removes the chance of having old files running as the deployment is atomic.
You'd set an app setting of WEBSITE_RUN_FROM_PACKAGE to 1 and continue deploying the same way you are today. The site will be run directly from that package (wwwroot will appear as read-only in kudu) so there's no unzipping and copying, which may be causing the issue you're having.
Note: it looks like we're still tracking the issue here: https://github.com/Azure/azure-functions-host/issues/2636.
In my case the issue lay in the CI-CD pipelines, where an out-of-date artifact was being deployed - thus the successful deploy, but the old code.
I have a webAPI deployed on window azure, till last week I was able to run the WebAPI and debug but all of a sudden when I run the web application I get the below message.
Role instances are taking longer than expected to start. Do you want
to continue waiting?
And I tried to enable disable caching(default caching is disable) and also cleared the local storage but still the same issue persist, its happening in not only my machine but also on other dev machine as well. Could it be because of automatic software updates related to emulator or other SW or any other issue?
Happened withh me today. I ran into an issue of the Azure emulator not working. The roles were going in a continuous loop of Running, Busy and Suspended.
Initially I thought it might be due to some code I had changed ď. However, it turned out that a recent Windows update had caused this issue.
After spending a frustrating amount of time un-installing the recent updates one by one, I found the miscreant.
For Fix please un-install the Security Update KB3126593 (https://support.microsoft.com/en-us/kb/3126593).
While searching for the particular update, make sure you type âKB3126593â in the search bar. Typing 3126593 will not work.
I created a cloud service and tested it successfully locally. I added service configurations for stage and production. Here is a snippet of my staging-configuration:
and here my configuration-settings:
Then when I publish I set up the deployment as follows:
All this worked like 2 weeks ago. But now he deploys in VS and when I look into Azure Service Configure area it looks like this:
I played a little bit with the "Update development ..."-checkbox on the second screen but the result is the same.
So it ignores all the settings I made and just won't tranistion my configuration to the ine I named "CloudStage". My current Web PI tells me that I use Windows Azure SDK for .NET (VS 2013) 2.3. I don't get the point.
Edit
Some more things I observed:
No WADLogsTable and WADWindowsEventLogsTable is generated automatically in the staging storage.
I deactivated Remote Desktop because it was one of the changes I made to monitor the event log (which wasn't useful here)
I manually changed the connection strings in Azure Portal but it seems as if the worker is totally unaware of the storage (rebooted it with no success).
Edit
I recognized another thing. Here you can see a running deployment of my service:
See the warning-mark on the left? If I go to my Error list this is shown:
This warning is senseless since it tells me that I did everything the right way. My *.Local.csfg-files are pointing to the local storage. So?!?
This seems weird. Please check the in your ServiceConfiguration.CloudStage.cscfg to verify the expected values.
Have you tried updating any other property like Enabling Remote desktop? Does that get updated on your deployment? You should select the "Deployment Update" check box in the publish dialog. Now, when deploying to an existing Cloud Service, it should ask you if you want to replace it.
If you get the Object reference error every time you right click on project, there might be some issue with the Azure SDK set up.
I'm a little bit further now. What I did was:
Deleted all Services in Azure.
Deleted all Storage Accounts in Azure
Removed my Service-Project completely from solution (not the library containing the worker-logic).
Re-added storage-accounts in Azure.
Re-added services in Azure.
Re-added a project in the solution and added the worker-logic inside it.
Builded up all the publishing-stuff again.
Published it.
The first publish ended like the one described in my question. After I checked the "Update development..."-option in properties of my worker it finally took my transitions into the stage!
Now I recognized, that WADLogsTable was still empty. I hit the instance right in server-explorer and choosse "Update diagnostic settings...". There was an option "Transfer period" suddenly set to "None". This explained to me, why my table was empty and after I set it back to "1" my table is filling again!
Another funny thing beside: When I right-click my Cloud-project in the solution I get "Object reference not set to an instance...". When I just click it left and choose Build->Publish it works.
I just hope that I can help somebody with this. Lets see if it's stable now.
Edit: Yesterday it worked - today is still the same issue :-(.
When you get "Object reference not set to an instance.." for a CloudService project you usually have some kind of mismatch. It could be that a setting in the ServiceConfiguration is not defined in the ServiceDefinition. It could also be that there is a publish profile defined in the .ccproj file for the CloudService that doesn't exist. This might also be what is causing your problems with the different configurations.
So it turns out that the problem is completely on client-side. My Visual Studio (now with SDK 2.4) is doing something wrong. I set up a fresh installation with all the stuff needed :-( and there it works perfect. I'll try to determine if one of my extensions is causing the strange "Object reference not set..."-bug.
Repair-Installation of VS does not solve the problem btw.
I'm currently working on an Azure project that works 100% locally with emulator resources. I'm now trying to deploy a worker role, but I'm running into an issue that I'm not sure how to troubleshoot.
Upon deploying the worker role in my Azure portal, the two instances continually loop through "recycling".
I can try to RDP into the role, but I only have about a minute to look around before the connection closes, I'm assuming due to the recycling.
After some searching it doesn't seem like this is a super common problem. Is there something trivial I'm overlooking that could be causing this issue? How would you go about troubleshooting this? Thank you for your time :)
In case of missing Reference you can troubleshoot this issue by:
Unzip your CSPKG file and then again unzip .CSSX file (just rename CSSX to zip) and match that everything references and static content is all there.. This way you can match what is on VM. Also in 2 minute windows when you RDP, try to look for Application event log for exception and get it because that would be the key to find the root cause.
IF you could see the exception in event log and look for the exception, you sure can find where it was generated. You can also use Intellitrace which might require you to redeploy the app.
Also there are ways were copying WinDBG and locking to the specific process you can debug it. I am not sure how much you would want to try but just copy the WinDBG to VM and use it would be enough (not sure how much experience you have with WinDBG though and how much time you would want to spent.)
Also been pestered by this role recycle issue numerous times. Here is the sequence of steps to debug persistent role recycles:
Debugging Azure Role Recycles
Enable Remote Access to your role - RDP login
Check eventvwr.msc (Windows Logs -> Application, App & Service Logs->Windows Azure)
Review the Azure text file logs across both C:\logs and c:\resources
Review custom logs in the Volume E: or F: for any custom role startup logging
Run AzureTools and attach to startup processes (download WinDBG, use Utils->Attach Debugger, select process - WaWorkerHost/WaIISHost, etc), use G to continue and watch debugger output for assemblies failing to load.
Installing Azure Debugging Tools via Powershell
PS> md c:\tools; Import-Module bitstransfer; Start-BitsTransfer http://dsazure.blob.core.windows.net/azuretools/AzureTools.exe c:\tools\AzureTools.exe; c:\tools\AzureTools.exe
If all items above fail - try using other tools in the AzureTools treasure trove - such as fusion logging, etc, this approach above will work!
WinDBG Sample Output - Failing to Locate Assembly (WaIISHost)
The most likely cause is that you have a missing assembly. One tactic to catch this is to wrap any startup processing in a master try/catch that manual logs the error to Azure storage.
If you added any referrences, check to make sure they're set to copylocal=true and that any external assets that were included in your service package were also set to be included.
From Avkash above:
Yes. this mean some issue in your Worker Role code is causing your Worker Role Host Process to crash.. If you look your fault stack you must see the function or the link from your code which generate this fault. IF you need help open a free Azure Support incident to Windows Azure Support team and they will help you.
Just a suggestion: Also Check the installable(if any)and any other references you use are 64bit.Azure VMs have 64bit OS. Once i was stuck up with this kind of problem due to 32/64 bit issues.
Are your worker roles exiting their work loop? A local recycle is very fast and you might not notice it, but spin-up time in the cloud can be long.
If the issue is caused by a startup batch file, I have stopped the loop by editing the batch file on the instance to include "exit /b 0" at the beginning. This will tell Azure that the startup was successful and you then have all the time you need to diagnose issues without the VM getting killed.