Since yesterday I'm suddenly having consistent deployment failures for my existing WebJobs. The error I'm getting is
Start Web Deploy Publish the Application/package to https://octotrip-api-test-dev.scm.azurewebsites.net/msdeploy.axd?site=octotrip-api-test__dev ...
Adding sitemanifest (sitemanifest).
Adding directory (XXX\app_data).
Adding directory (XXX\app_data\jobs).
Adding directory (XXX\app_data\jobs\continuous).
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Microsoft\VisualStudio\v15.0\Web\Microsoft.Web.Publishing.targets(4295,5): Error : Web deployment task failed. ((3/16/2017 1:32:23 PM) An error occurred when the request was processed on the remote computer.)
(3/16/2017 1:32:23 PM) An error occurred when the request was processed on the remote computer.
An error was encountered when processing operation 'Create Directory' on 'D:\home\site\wwwroot\app_data\jobs\continuous'.
The error code was 0x800700B7.
Cannot create "D:\home\site\wwwroot\app_data\jobs\continuous" because a file or directory with the same name already exists.
at Microsoft.Web.Deployment.NativeMethods.RaiseIOExceptionFromErrorCode(Win32ErrorCode errorCode, String maybeFullPath)
at Microsoft.Web.Deployment.DirectoryEx.CreateDirectory(String path)
at Microsoft.Web.Deployment.DirPathProviderBase.CreateDirectory(String fullPath, DeploymentObject source)
at Microsoft.Web.Deployment.DirPathProviderBase.Add(DeploymentObject source, Boolean whatIf)
Publish failed to deploy.
Important to note that at the time of the deployment; there is no app_data folder (I removed it manually using FTP). You can see it is creating the folder structure step by step; app_data, then the jobs folder, then the continuous folder. I confirmed using FTP that he succesfully creates the app_data/jobs folder; but creating the continuous folder consistently crashes.
This happens when deploying using a build definition in VSTS as well as deploying as a WebJob from within Visual Studio; with all the WebJob NuGet up to date.
Thanks
It is very odd that can't create folder for WebJob. I can't repro it in my side. Do you have a try to deploy it to another website to check whether it can be repro?
Based on my experience, we could troubleshoot with following ways:
Restart the Website
If it is possible try to redeploy the Website and remove additional files at destination
Scale up and scale down back Service plan
Note:change will be applied to all WebApps in the ServicePlan
Please contact Azure Support team for help if it is not resolved .
Worked around the issue by simply creating a new Azure Web App. Something messed up going on with the file system. Take a look.
app_data or subdirectories of app_data don't exist on the server. (confirmed using CMD and FTP).
This only happens for
app_data\jobs\triggered and
app_data\jobs\continuous
(app_data\jobs\triggereddddd works for example)
These folders have existed on this server before (they contain the data for the WebJobs that we're running)
open Azure Portal and go to your web app
Under "development tools" select "K Advanced Tools" this will open the Kudu control panel in a new browser window.
Select "Debug Console" -> "CMD"
in the Items table, locate the wwwroot entry and delete it.
continue working ...
Related
I'm using Right-Click publish in Visual Studio to publish directly to my staging slot in Azure. I've added Application Insights that added a WebJob to my app.
I have enabled Remove additional files at destination and Precompile during Publishing turned on.
But when a Web Job is running and my staging site is not 'stopped', because it's still warmed up - I get the following error:
Web deployment task failed. ((18-Jan-18 22:47:23) An error occurred
when the request was processed on the remote computer.)
(18-Jan-18 22:47:23) An error occurred when the request was processed
on the remote computer. An error was encountered when processing
operation 'Delete Directory' on
'D:\home\site\wwwroot\App_Data\jobs\continuous'. The error code was
0x80070091. The directory is not empty.
at
Microsoft.Web.Deployment.NativeMethods.RaiseIOExceptionFromErrorCode(Win32ErrorCode
errorCode, String maybeFullPath) at
Microsoft.Web.Deployment.DirectoryEx.Delete(String path) at
Microsoft.Web.Deployment.DirPathProviderBase.Delete(Boolean
whatIf) ProjectX.Web 7 0
I can fix this by manually stopping the staging slot, upload it, and then start it again. This is quite cumbersome though.
Is there a fix?
I read somewhere that you can automatically take the (staging) app offline during publish but I can't find the setting.
UPDATE 1
<PropertyGroup>
<EnableMSDeployAppOffline>true</EnableMSDeployAppOffline>
</PropertyGroup>
...is not working for me.
UPDATE 2
Adding the WEBJOBS_STOPPED setting on the Staging slot is not working for me.
As you have enabled Application Insights on your Azure App Service, it has internally installed an extension which started a continuous web job and content corresponding to web job got copied to App_Data folder. The actual problem is because extension data get copied to App_Data folder where only user's data should reside instead of infrastructure data.
Now when you are selecting the option "Remove Additional Files From Destination" in visual studio, it tries to remove all contents including Application Insights data from App_Data folder. Since Application Insights web job is already running in background hence corresponding dll files are locked.
Even if you go with option of "App Offline" or "Stop web job" and successfully able to delete all contents then your deployment will succeed but this way your Application Insights will get corrupted and hence you may have to configure Application Insights again.
Suggestion:
Select skip "App_Data" folder along with option "Remove Additional Files From Destination".
It will delete all content except App_Data folder content and hence dll files corresponding to Application Insights continuous web job will be skipped.
Reference :
VSTS - Azure App Service Task Documentation
Limitation
Though this solution have a limitation - If you want to remove/modify existing user defined web jobs then can't skip App_Data folder. In such case either have to perform two step deployment or have to reinstall application insight extension again.
I found this in another SO question:
<PropertyGroup>
<EnableMSDeployAppOffline>true</EnableMSDeployAppOffline>
...
</PropertyGroup>
... adding to your publishing profile (.pubxml).
The full answer is here: https://stackoverflow.com/a/20888597/647845
UPDATE
This is not working for me. Maybe for others. Seems to ignore the setting.
When trying to publish my .Net Core app to a remote IIS server I am unable to publish with the "Delete Existing Files" option selected. When I try I get hundreds of errors for each file saying: Unable to remove ... The process cannot access the file ... because it is being used by another process.
If I shut down the kestrel server or put app_offline.htm in the web folder I still get an error for the root folder of the web app: Unable to remove directory "\remoteserver\webapp\". The process cannot access the file "\remoteserver\wenapp\' because it is being used by another process.
I literally have to shut down IIS to publish which is not ideal. Any suggestions? I have made sure the permissions allow read an write to users.
I've created a web job using the webjobs sdk that is linked to my web application. Well it isn't anymore (I removed the webjobs-list.json file from the web app project), because when it was linked, it just copied the source code files into the websites' app-data folder, not the bin folder/binaries/exe. But now that my web job is deployed from visual studio online to azure with my website, and tries to run, it's not reading the connection strings (AzureWebJobsStorage and AzureWebJobsDashboard) from the connection strings in my web application's settings in the azure portal. It looks for them in the config file that is deployed with the exe that is generated from compiling the webjob. If I manually put those strings in that config and check in, and redploy it will use them from there, so it's not checking the portal at all (even though my web app is running fine using what is defined in the portal). I thought that all web jobs were supposed to look for connection strings in the azure portal first? That isn't happening at all for me. Has anyone else experienced this? Could it have something to do with how it's deployed by visual studio online?
Part of the error getting logged for the webjob process when trying to start:
Microsoft.Azure.WebJobs.Host.Indexers.FunctionIndexingException: Error indexing method 'Functions.ProcessTestQueueMessage' ---> System.InvalidOperationException: Microsoft Azure WebJobs SDK 'Storage' connection string is missing or empty. The Microsoft Azure Storage account connection string can be set in the following ways:
[04/01/2017 02:27:40 > ceef22: INFO] 1. Set the connection string named 'AzureWebJobsStorage' in the connectionStrings section of the .config file in the following format , or
[04/01/2017 02:27:40 > ceef22: INFO] 2. Set the environment variable named 'AzureWebJobsStorage', or
[04/01/2017 02:27:40 > ceef22: INFO] 3. Set corresponding property of JobHostConfiguration.
[04/01/2017 02:27:40 > ceef22: INFO] at Microsoft.Azure.WebJobs.Host.Executors.StorageAccountParser.ParseAccount(String connectionString, String connectionStringName, IServiceProvider services)
Also, the only way I could get this to really deploy was to remove the webjobs-list.json file from the properties folder in the web project. I know I'm not supposed to do that, that everything should just work and deploy with that in there, but when I left it in there, it wouldn't copy the binaries/bin/executable files into the App_Data\Jobs\continuous[webjob] folder on the website...it would literally copy all of the source code\files for the web job project into there (which of course wouldn't work/run, since it's just code, not an exe). After removing that webjobs-list.json file, and then adding a second task to my release to find the [webjob].zip file from the build, then it would copy just the binaries and executable into the App_Data\Jobs\continuous[webjob] folder. Maybe this is just an issue with trying to build and release from visual studio online, and it doesn't handle web job deployment well...not sure, I haven't been able to find a lot of information on setting that up to deploy a web job with a web app.
Normally, this should just work. The flow is:
The WebJobs exe doesn't run in-place, but first gets copied to a temp folder
As part of that copying process, if any of the AppSettings & Conn Strings in the foo.exe.config file are set as Azure settings, they get replaced (only in the temp copy, not under App_Data).
Can you make sure that you are deploying both your foo.exe and foo.exe.config?
Update: as we found on chat, the problem was that you had a typo in the name of the connection string in Azure.
I am getting the following error:
Error: The process cannot access the file 'C:\DWASFiles\Sites\mywebsitename\VirtualDirectory0\site\wwwroot\newrelic\NewRelic.Agent.Core.dll' because it is being used by another process.
In the Running deployment command... log file when attempting to deploy an Azure website from Github.
Would appreciate any pointers as to what could be causing this.
UPDATE: Turns out this is also failing when publishing directly from VS.NET with the following:
1>C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\Web\Microsoft.Web.Publishing.targets(4196,5): Warning : An error was encountered when processing operation 'Create File' on 'NewRelic.Agent.Core.dll'.
1>Retrying operation 'Update' on object filePath (mywebsitename\newrelic\NewRelic.Agent.Core.dll). Attempt 1 of 2.
1>C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\Web\Microsoft.Web.Publishing.targets(4196,5): Error : Web deployment task failed. ((06/07/2013 23:54:58) An error occurred when the request was processed on the remote computer.)
This was working before and I am not sure why it would have stopped.
NewRelic recommend stopping the website to unload the file and allow the deployment to go through.
As an alternative, you can set COR_ENABLE_PROFILING to 0 in your app settings on the configure tab to temporarily disable the profiling, which should then allow you to continue with the deployment while leaving the website operational throughout.
Instead of stopping the website you can temporarily turn off New Relic monitoring via the Configure tab on manage.windowsazure.com:
Configure > developer analytics > select "OFF" > Save
Deploy
Configure > developer analytics > select "ADD-ON" > Choose Add-on from dropdown > Save
Worked for me, both with a regular deployment from VS and an automatic build from VSO.
This is a known issue with the New Relic .NET agent for Azure Websites when performing an upgrade of the agent. The workaround is to stop the website to release the dll, finish the deployment and then restart the instance.
https://newrelic.com/docs/dotnet/azure-web-sites#h2-1
Not really a solution but more of a work-around, in the publish dialog view a preview of the changes and uncheck the NewRelic.Agent.Core.dll file so that it doesn't get published.
None of these answers work for me anymore. I have an Azure Basic tier website plan, which hosts multiple actual websites.
If I don't stop the website, I get the error mentioned above (newrelic.agent.core.dll is in use)...
If I do stop the website (or all of them), I get an error saying that the publishing endpoint isn't available.
If I go to the configure tab and disable the AddOn, we still get the error mentioned above (newrelic.agent.core.dll is in use)...
Pretty much we just republish over and over again with different permutations of the above until if works. It took me hours the other day, took me 10 minutes today.
If you are using webdeploy, then you can configure your webdeploy settings so that it ignores the file. However, if you do that, you will manually have to deploy any updates to the new relic agent.
I had a similar issue with the new relic log file being locked, and solved it by:
Moving the new relic log file to a subdirectory of the web root (e.g. \newreliclogs)
Adding 2 lines to my powershell script that configured the skip directive to ignore that whole directory. e.g. (where destBaseOptions is of type Microsoft.Web.Deployment.DeploymentBaseOptions
$skipDirective = new-object Microsoft.Web.Deployment.DeploymentSkipDirective("NewRelicLog","objectName=dirPath,absolutePath=.*\newreliclogs$")
$destBaseOptions.SkipDirectives.Add($skipDirective)
Depending on how you are using webdeploy, the configuration is achieved slightly differently, I used the following links to help me piece it together:
https://technet.microsoft.com/en-us/library/dd569089(WS.10).aspx
https://msdn.microsoft.com/en-us/library/microsoft.web.deployment.deploymentskipdirective(v=vs.90).aspx
https://msdn.microsoft.com/en-us/library/dd543313(v=vs.90).aspx
http://blogs.iis.net/jamescoo/archive/2009/11/03/msdeploy-api-scenarios.aspx
http://forums.iis.net/p/1192163/2031814.aspx#2031813
And I used the powershell script from the Octopus Deploy Library at https://library.octopusdeploy.com/#!/step-template/actiontemplate-web-deploy-publish-website-(msdeploy).
I have an Orchard CMS website deployed to Windows Azure using Git Deploy from BitBucket.
Just with my latest changes I'm getting an error when I try to deploy:
The process cannot access the file
'C:\DWASFiles\Sites[MyWebsite]\VirtualDirectory0\site\wwwroot\App_Data\Dependencies\Joel.Net.Akismet.dll'
because it is being used by another process.
Given its Azure Websites, I can't "stop the AppPool" or anything like that. I've tried stopping the site, but then I can't do a deployment with the site stopped. I'm out of ideas as to how to troubleshoot further.
Trying to update via ftp
There is no option to "restart" but I can stop and start the site, this does not work (502 web server error).
Deleting the files the the \App_Data\Dependencies folder via ftp does not work, and gets the same error.
Stopping the site and then uploading the files via ftp does work. But trying to redeploy via Git, I get a similar error:
Command: deploy.cmd Handling Basic Web Site deployment. KuduSync.NET
from:
'C:\DWASFiles\Sites[MyWebsite]\VirtualDirectory0\site\repository' to:
'C:\DWASFiles\Sites[MyWebsite]\VirtualDirectory0\site\wwwroot' Error:
The process cannot access the file
'C:\DWASFiles\Sites[MyWebsite]\VirtualDirectory0\site\wwwroot\App_Data\Dependencies\Markdown.dll'
because it is being used by another process. Copying file:
'App_Data\Dependencies\dependencies.compiled.xml' Copying file:
'App_Data\Dependencies\dependencies.xml' Copying file:
'App_Data\Dependencies\Ionic.Zip.dll' Copying file:
'App_Data\Dependencies\Joel.Net.Akismet.dll' Copying file:
'App_Data\Dependencies\Markdown.dll' An error has occurred during web
site deployment. Handling Basic Web Site deployment.
I'm sure its not just the one dll, but rather, all the dlls in the \App_Data\Dependencies folder don't get "stopped" during a Git deployment.
Trying to set up another site on Azure Websites to do the Git Deploy
I have set up another Azure site for Git to deploy the same branch of code to and this works fine.
But when I switch my domain name to the new Azure Website, my new site no longer works, but my old one can accept git deploy.
So the website with my domain name directed at it, must be getting activity causing the update to stop working. Is there a way to stop the site AND do a Git deploy?
So after removing the Dependencies folder from my repo as per #David's comments, I'm getting a new error:
Command: deploy.cmd Handling Basic Web Site deployment. KuduSync.NET
from:
'C:\DWASFiles\Sites[MyWebsite]\VirtualDirectory0\site\repository'
to: 'C:\DWASFiles\Sites[MyWebsite]\VirtualDirectory0\site\wwwroot'
Error: Access to the path 'Joel.Net.Akismet.dll' is denied. Copying
file: '.gitignore' Deleting file:
'App_Data\Dependencies\Joel.Net.Akismet.dll' An error has occurred
during web site deployment. Handling Basic Web Site deployment.
Seems like the Kudu sync is seeing that there are no files in the Dependencies folder, and trying to delete them! The same happens when I have no Dependencies folder at all - it tries to delete the Dependencies folder.
#David is there a kudu ignore file?
SOLVED!
OK I kept at it, and I have finally resolved it.
I deleted the entire app_data folder (backed it up first) then checked this into Git.
This was pushed into Azure, giving me a fresh Orchard site
I then re-added my app_data but chose to ignore the app_data/dependencies folder
pushed to Git and updated to Azure perfectly
Hope my next check in still works (fingers crossed)
I'm guessing that the root of the problem is that you are committing the App_Data\Dependencies folder into your repo. This folder is something that Orchard creates dynamically, and should not be in your repo.
I suspect that if you don't do this (by using .gitignore), everything will work fine and you won't get this error.
You can restart the website just before deployment, which is akin to recycling the app pool. Just hit the restart button from the footer on the dashboard.