Azure DevOps release deployment failing Disk Space Error - azure

I have an app I am trying to deploy through azure's DevOps pipeline. I have successfully deployed apps before, and the first build of this app deployed fine. However all subsequent builds have been failing at the deployment step. They seem to be specifically failing at the "Deploy Azure App Service" step, generating a ERROR_NOT_ENOUGH_DISK_SPACE error code.
The deployment terminal output shows the following errors:
Got connection details for Azure App Service:'MMOMarketTracker'
[command]"C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -verb:sync -source:package='D:\a\r1\a\_MMMarketTracker\build\build.zip' -dest:contentPath='MMOMarketTracker',ComputerName='https://mmomarkettracker.scm.azurewebsites.net:443/msdeploy.axd?site=MMOMarketTracker',UserName='$MMOMarketTracker',Password='***',AuthType='Basic' -enableRule:AppOffline -enableRule:DoNotDeleteRule -userAgent:VSTS_ba2ca09f-578d-4893-99ac-412c3f26fde5_release_1_9_9_1
2018-11-22T21:02:35.2302461Z Info: Using ID 'e7a243b1-97df-4ec2-b8d9-c679c2e35512' for connections to the remote server.
Info: Adding directory (MMOMarketTracker\s\.git\objects\15).
Error: C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe failed with return code: 4294967295
at ChildProcess.<anonymous> (D:\a\_tasks\AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1\3.4.16\node_modules\vsts-task-lib\toolrunner.js:568:30)
at emitTwo (events.js:106:13)
at ChildProcess.emit (events.js:191:7)
at maybeClose (internal/child_process.js:886:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:226:5)
[command]"C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -verb:sync -source:package='D:\a\r1\a\_MMMarketTracker\build\build.zip' -dest:contentPath='MMOMarketTracker',ComputerName='https://mmomarkettracker.scm.azurewebsites.net:443/msdeploy.axd?site=MMOMarketTracker',UserName='$MMOMarketTracker',Password='***',AuthType='Basic' -enableRule:AppOffline -enableRule:DoNotDeleteRule -userAgent:VSTS_ba2ca09f-578d-4893-99ac-412c3f26fde5_release_1_9_9_1
Info: Using ID '201db2e8-ee9f-4e39-9a29-2187cb7056bd' for connections to the remote server.
Info: Adding directory (MMOMarketTracker\s\.git\objects\15).
##[error]Failed to deploy web package to App Service.
##[error]Error Code: ERROR_NOT_ENOUGH_DISK_SPACE
More Information: Web Deploy detected insufficient space on disk. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_NOT_ENOUGH_DISK_SPACE.
Error: The error code was 0x80070070.
Error: There is not enough space on the disk.
at Microsoft.Web.Deployment.NativeMethods.RaiseIOExceptionFromErrorCode(Win32ErrorCode errorCode, String maybeFullPath)
at Microsoft.Web.Deployment.DirectoryEx.CreateDirectory(String path)
at Microsoft.Web.Deployment.DirPathProviderBase.CreateDirectory(String fullPath, DeploymentObject source)
at Microsoft.Web.Deployment.DirPathProviderBase.Add(DeploymentObject source, Boolean whatIf)
Error count: 1.
##[error]Error: C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe failed with return code: 4294967295
The first error seems to be caused by the remote server failing to add a git object. This error will usually occur 2-3 times each deployment, and will result in the "Retrying to to deploy the package" message shown.
The second error will happen once and will end the deployment. I have been to the link the error recommends and there is no help section for this type of error.
I downloaded the build file and it is only 49.3 MB large, and the Microsoft servers have a 10GB size for deployments. I am running on a hosted build agent, not a private one.
Other questions I found online similar to this are mostly unanswered. Some have been updated later to say the deployment started working for seemingly no reason.
Any help would be appreciated.

For anyone else who gets the same problem, I basically had too many other apps taking up space on my azure account. The article Brendan Green linked explains this.
(https://blogs.msdn.microsoft.com/puneetgupta/2017/06/06/azure-app-service-what-is-taking-my-disk-space-for-my-azure-web-app/)
By going to the Quotas page on any of your azure apps (in the azure portal) you can see how much space you have left.

This happens when you're out of staging slots in your pricing tier. So, a workout solution is to upgrade your pricing tier to acquire more staging slots.
The staging slots are used for testing and deployments before swapping them into production and it does not associate with your file system disk space.

You may be able to reclaim disk space by triggering the build agent cleanup process.

Related

Azure Function App - Publish failed

I'm unable to publish (WebDeploy) my Azure Function App (It's a Durable Function, I know it deosn't matter, but, just in case if matters). This started happening today.
I'm getting a "Publish Failed" popup and the below error message in the output window of my Visual Studio.
Error Message
<ProjectName> -> C:\search\source\<Solution Folder>\obj\Release\netstandard2.0\PubTmp\Out\
C:\Program Files\dotnet\sdk\2.1.202\Sdks\Microsoft.NET.Sdk.Publish\build\netstandard1.0\PublishTargets\Microsoft.NET.Sdk.Publish.MSDeploy.targets(139,5): error : Web deployment task failed. (Could not connect to the remote computer ("<functionappname>.scm.azurewebsites.net"). On the remote computer, make sure that Web Deploy is installed and that the required process ("Web Management Service") is started. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_DESTINATION_NOT_REACHABLE.) [<projectname>.csproj]
C:\Program Files\dotnet\sdk\2.1.202\Sdks\Microsoft.NET.Sdk.Publish\build\netstandard1.0\PublishTargets\Microsoft.NET.Sdk.Publish.MSDeploy.targets(139,5): error : The requested resource does not exist, or the requested URL is incorrect. [<projectname>.csproj]
C:\Program Files\dotnet\sdk\2.1.202\Sdks\Microsoft.NET.Sdk.Publish\build\netstandard1.0\PublishTargets\Microsoft.NET.Sdk.Publish.MSDeploy.targets(139,5): error : Error details: [<projectname>.csproj]
C:\Program Files\dotnet\sdk\2.1.202\Sdks\Microsoft.NET.Sdk.Publish\build\netstandard1.0\PublishTargets\Microsoft.NET.Sdk.Publish.MSDeploy.targets(139,5): error : Could not connect to the remote computer ("<functionappname>.scm.azurewebsites.net"). On the remote computer, make sure that Web Deploy is installed and that the required process ("Web Management Service") is started. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_DESTINATION_NOT_REACHABLE. [<projectname>.csproj]
C:\Program Files\dotnet\sdk\2.1.202\Sdks\Microsoft.NET.Sdk.Publish\build\netstandard1.0\PublishTargets\Microsoft.NET.Sdk.Publish.MSDeploy.targets(139,5): error : The remote server returned an error: (404) Not Found. [<projectname>.csproj]
Publish failed to deploy.
Function Runtime - beta
When I access the Function App Url in a browser I get "Your Function App 2.0 preview is up and running" message". And, I don't see anything wrong in the Kudu portal as well.
I have deleted the Function and re-created it with the same name with no luck.
Any help is much appreciated.
Update : Here is another error that I'm getting as well.
C:\Program Files\dotnet\sdk\2.1.202\Sdks\Microsoft.NET.Sdk.Publish\build\netstandard1.0\PublishTargets\Microsoft.NET.Sdk.Publish.MSDeploy.targets(139,5): error : Web deployment task failed. ((7/25/2018 1:16:27 PM) An error occurred when the request was processed on the remote computer.) [<Project Name>.csproj]
C:\Program Files\dotnet\sdk\2.1.202\Sdks\Microsoft.NET.Sdk.Publish\build\netstandard1.0\PublishTargets\Microsoft.NET.Sdk.Publish.MSDeploy.targets(139,5): error : [<Project Name>.csproj]
C:\Program Files\dotnet\sdk\2.1.202\Sdks\Microsoft.NET.Sdk.Publish\build\netstandard1.0\PublishTargets\Microsoft.NET.Sdk.Publish.MSDeploy.targets(139,5): error : (7/25/2018 1:16:27 PM) An error occurred when the request was processed on the remote computer. [<Project Name>.csproj]
C:\Program Files\dotnet\sdk\2.1.202\Sdks\Microsoft.NET.Sdk.Publish\build\netstandard1.0\PublishTargets\Microsoft.NET.Sdk.Publish.MSDeploy.targets(139,5): error : An error was encountered when processing operation 'Create Directory' on 'D:\home\site\wwwroot\bin'. [<Project Name>.csproj]
C:\Program Files\dotnet\sdk\2.1.202\Sdks\Microsoft.NET.Sdk.Publish\build\netstandard1.0\PublishTargets\Microsoft.NET.Sdk.Publish.MSDeploy.targets(139,5): error : The error code was 0x800703E6. [<Project Name>.csproj]
C:\Program Files\dotnet\sdk\2.1.202\Sdks\Microsoft.NET.Sdk.Publish\build\netstandard1.0\PublishTargets\Microsoft.NET.Sdk.Publish.MSDeploy.targets(139,5): error : Invalid access to memory location. [<Project Name>.csproj]
C:\Program Files\dotnet\sdk\2.1.202\Sdks\Microsoft.NET.Sdk.Publish\build\netstandard1.0\PublishTargets\Microsoft.NET.Sdk.Publish.MSDeploy.targets(139,5): error : [<Project Name>.csproj]
C:\Program Files\dotnet\sdk\2.1.202\Sdks\Microsoft.NET.Sdk.Publish\build\netstandard1.0\PublishTargets\Microsoft.NET.Sdk.Publish.MSDeploy.targets(139,5): error : at Microsoft.Web.Deployment.NativeMethods.RaiseIOExceptionFromErrorCode(Win32ErrorCode errorCode, String maybeFullPath) [<Project Name>.csproj]
C:\Program Files\dotnet\sdk\2.1.202\Sdks\Microsoft.NET.Sdk.Publish\build\netstandard1.0\PublishTargets\Microsoft.NET.Sdk.Publish.MSDeploy.targets(139,5): error : at Microsoft.Web.Deployment.FileSystemInfoEx.set_Attributes(FileAttributes value) [<Project Name>.csproj]
C:\Program Files\dotnet\sdk\2.1.202\Sdks\Microsoft.NET.Sdk.Publish\build\netstandard1.0\PublishTargets\Microsoft.NET.Sdk.Publish.MSDeploy.targets(139,5): error : at Microsoft.Web.Deployment.DirPathProviderBase.Add(DeploymentObject source, Boolean whatIf) [<Project Name>.csproj]
Publish failed to deploy.
Thanks,
Praveen
I fixed this issue by setting WEBSITE_RUN_FROM_PACKAGE to 0 on the application settings. I was uploading the function files by syncing a WebDeploy package from Azure DevOps to the App Service file system and not running it from the .zip package.
On https://learn.microsoft.com/bs-cyrl-ba/azure/azure-functions/run-functions-from-deployment-package explains about running Functions from a deployment package and all the benefits of doing it this way, but then remember to:
Just upload the .zip deployment package to d:\home\data\sitepackages (don't use the msdeploy verb "sync" to sync the file system at wwwroot)
Add a packagename.txt containing only the name of the package file in folder, without any whitespace
Go to azure portal, stop the function app, publish from vs, then start again.
Don't know if this is closed but I had same issue while deploying new SDK version to an older azure function app. Here is what I did:
Instead of downloading deployment profile from Azure portal, created a new one with visual studio automated deployment.
This is important- While deploying from Visual studio it should ask you to upgrade version of deployed app, click yes and your app will get deployed.
Can't reproduce your error when using WebDeploy with SCM. When I set WEBSITE_WEBDEPLOY_USE_SCM to false(in Application settings) to avoid using SCM, I saw An error was encountered when processing operation 'Create Directory' on 'wwwroot' like the second error you have met.
Several options to try:
Visit https://functionappname.scm.azurewebsites.net in browser to check whether your network environment is fine and has no firewall restriction.
Check whether you have proxy set like Fiddler when publishing.
Go to portal, Reset publish credentials then Download publish profile. Delete old publish profiles and import the new one to publish again.
If all of those suggestions fail, delete all old resources relate to the function. Then create a brand new function app with new app service plan and storage. Try to publish to this new app.
This seems to happen to me when I have made changes that for some reason necessitate the following.
Rebuild the project (make sure you can run it locally).
Click to publish, but prior to that click on "Manage Application Settings" do
a quick check and click OK.
Attempt to publish.
If that fails close visual studio and azure portal (stop running or disable all relevant objects) and repeat.
Worst case create a new profile and deploy to that.
It was easy to miss for me...but the Function on Azure was blocking my publish from Visual Studio 2017.
What was needed was a restart of the Function on Azure.
1) Go to Azure Portal and click into your Function app.
2) Click on the name of your Function, and then in the Overview tab - in the second-row menu you should see Restart. (You'll see it in the row with: Stop - Swap - Restart - Get publish profile...etc.)
3) Click Restart
4) Go back to Visual Studio 2017 and Republish.
My error message which I initially missed:
C:\Program Files\dotnet\sdk\2.2.103\Sdks\Microsoft.NET.Sdk.Publish\build\netstandard1.0\PublishTargets\Microsoft.NET.Sdk.Publish.MSDeploy.targets(140,5): error : Web deployment task failed. (Web Deploy cannot modify the file 'MyApp.Functions.dll' on the destination because it is locked by an external process. In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.) [C:\Github\2019AzureBlobLocalOnly\MyApp.Functions\MyApp.Functions.csproj]
A similar issue to #Andrew Chung where there were locks on the files in Azure and it couldn't delete old function dll files. I was trying to upgrade to .NET core and publish from a different project. Unfortunately, i didn't see his response until it was too late, and had followed #Jerry Liu's advice (#4) to remove the function app and add in a new one. This didn't seem to work for me as until I recreated the app with a different name. I suspect there may be a DNS conflict where xxxx.scm.azurewebsites.net is still pointing to the IP address of your old decommissioned app.

cloudfoundry/staticfile-buildpack static web page failing to deploy on BlueMix Devops

I have a simple static web site project on GitHub. I've created a pipeline that does a "Simple" build and deploy. The deploy log file shows the following "less than informative" error:
Preparing to start the job...
Cloning the 'master' branch from repo 'https://github.com/FlatBallFlyer/ssgpp.git'
Repository successfully cloned
Target: https://api.ng.bluemix.net
Using manifest file /home/pipeline/424009fc-100d-4068-9a23-3dde2271cba4/manifest.yml
Using stack cflinuxfs2...
OK
Creating app ssgpp in org Mike.Storey#us.ibm.com / space ssgpp as Mike.Storey#us.ibm.com...
OK
Using route ssgpp.mybluemix.net
Binding ssgpp.mybluemix.net to ssgpp...
OK
Uploading ssgpp...
FAILED
Error processing app files: Error uploading application.
Server error, status code: 500, error code: 0, message:
Finished: FAILED
The manifest.yml from the git project is simply:
applications:
- name: ssgpp
memory: 64M
buildpack: https://github.com/cloudfoundry/staticfile-buildpack.git
host: ssgpp
stack: cflinuxfs2
This is a giga-pixel panoramic photo web app that contains A LOT of files - about 1.5gb of images. I'm not sure if I'm blowing the top off of a BlueMix limit or not.
I guess you reached the limit of your plan in BlueMix.
BTW, 1.5GB isn't a reasonable size - for me - to deploy an app. You'll always send the files when doing a simple bugfix? I'd suggest you to move your images to another service, dedicated to store files, such Cloudinary.

Team Services Azure App Service Deploy failure - Virtual Application characters?

I have 2 Azure App Service Deploy steps in a Team Services build, they have the same Subscription details. The only difference is that they target different virtual applications - the one that fails contains an é character, could that be the cause? Here the step output:
2017-01-17T16:04:22.8323742Z 8570d8e4-b732-4c45-87c5-619fe9343427 exists true
2017-01-17T16:04:24.5778681Z Got connection details for Azure App Service:'foo'
2017-01-17T16:04:24.6088690Z Running command: "C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -verb:getParameters -source:package="C:\a\1\a\Foo.zip" > "C:\a\1\s\parameter.xml"
2017-01-17T16:04:25.4027522Z Running command: #echo off
2017-01-17T16:04:25.4027522Z "C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" -verb:sync -source:package="C:\a\1\a\Foo.zip" -dest:auto,ComputerName='https://foo.scm.azurewebsites.net:443/msdeploy.axd?site=foo',UserName='********',Password='********',AuthType='Basic' -setParam:name='IIS Web Application Name',value='foo/bar-é' -userAgent:VSTS_foo_build_154_744 2>error.txt
2017-01-17T16:04:25.4027522Z if %errorlevel% neq 0 exit /b %errorlevel%
2017-01-17T16:04:25.4067519Z [command]C:\Windows\system32\cmd.exe /C C:\a\1\s\msDeployCommand.bat
2017-01-17T16:04:25.4157662Z 'C:\Program' is not recognized as an internal or external command,
2017-01-17T16:04:25.4167533Z operable program or batch file.
2017-01-17T16:04:25.4207534Z ##[error]Failed to deploy App Service.
2017-01-17T16:04:29.2496923Z ##[error]Error: Error: C:\Windows\system32\cmd.exe failed with return code: 9009
2017-01-17T16:04:29.2576899Z ##[section]Finishing: Azure App Service Deploy: foo-dev-global
It's almost as if the Powershell command itself is broken, I'm guessing by the special character.
It maybe not the special character causes that issue. I try to deploy the virtual Application with the special characters é , following is my detail steps. Hope it is useful. We also can get moreinfo about WebDeploy error code please refer to document.
1.Create a new build definition for the project
2.Select build solution corresponding VS version.
Before config Azure App Service Deployment, we need to create a WebApp in the Azure portal and the edit the appsetting to add the
Virtual applications info
Config the Azure App Service Deployment with service name and Virtual application created in the step3
5.New Queue build for the build definition
Then we can check that it deployed successfully from deployment log or from kudu.
This is now working as expected, without me making any changes. Having informed MS of the issue, I can only conclude they've released a fix to the build task.

All but first deployment of Orchard CMS to Azure Web Apps Fails

In Visual Studio Online I have a scripted build that builds Orchard 1.9.1 from source then, using the Azure Web App Deployment task, deploys the compiled Web Deploy Package to Azure.
The first deployment to a fresh Web App instance works great and the system functions as expected. However, any subsequent deployment to the same Web App fails with:
Unable to perform the operation ("Delete File") for the specified
directory ("HtmlAgilityPack.dll"). This can occur if the server
administrator has not authorized this operation for the user
credentials you are using. Learn more at:
http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_INSUFFICIENT_ACCESS_TO_SITE_FOLDER.
The file specified in the error can be different, but the deployment fails consistently. The link in the error isn't of much help as the implementation details covered (IIS, root folder etc) are abstracted away in an Azure Web App.
Where should I look to fix this permissioning issue?
I went with the default build template for Azure Website when creating the build, details of which are below for info:
Build Step:
MSBuild Arguments:
/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true
/p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.stagingDirectory)"
Azure Web App Deployment step:
Pertinent information for the build log:
2015-09-28T05:36:22.7928424Z Package= C:\a\d8c9303e\staging\Orchard.Web.zip
2015-09-28T05:36:22.7938416Z Slot=
2015-09-28T05:36:22.7938416Z AdditionalArguments=
2015-09-28T05:36:22.8028415Z packageFile= Find-Files -SearchPattern C:\a\d8c9303e\staging\Orchard.Web.zip
2015-09-28T05:36:22.8198428Z packageFile= C:\a\d8c9303e\staging\Orchard.Web.zip
2015-09-28T05:36:22.8258420Z Get-AzureWebSite -Name **** -ErrorAction SilentlyContinue
2015-09-28T05:36:28.4361621Z Publish-AzureWebsiteProject -Name "****" -Package "C:\a\d8c9303e\staging\Orchard.Web.zip"
2015-09-28T05:36:33.8798112Z ##[error]Unable to perform the operation ("Delete File") for the specified directory ("HtmlAgilityPack.dll"). This can occur if the server administrator has not authorized this operation for the user credentials you are using.
2015-09-28T05:36:33.8818107Z ##[error] Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_INSUFFICIENT_ACCESS_TO_SITE_FOLDER.
Use the provided Orchard.proj msbuild script using the Precompiled target.

Why is Web Deployment Agent Service (MsDepSvc) crashing with a FileNotFoundException?

I'm attempting to deploy a .NET 4 web application to a Windows 2008R2 server using Web Deploy V2. As part of this deployment we copy some batch files to the target machine. When I execute the command below, Web Deployment Agent Service (MsDepSvc) stops and a FileNotFoundException is logged to the system Application event log.
The web deploy command:
"C:\Program Files (x86)\IIS\Microsoft Web Deploy V2\msdeploy.exe" -verb:sync -source:contentPath="C:\work\27f84e3eca3a33d5\Output\BatchFiles" -dest:contentPath="c:_DeployCommands",computerName="192.168.1.14"
Full console output:
C:\Windows\system32>"C:\Program Files (x86)\IIS\Microsoft Web Deploy
V2\msdepl y.exe" -verb:sync
-source:contentPath="C:\work\27f84e3eca3a 3d5\Output\BatchFiles"
-dest:contentPath="c:_DeployCommands",computer ame="192.168.1.14" Info: Using ID '326b55b4-602c-4cdf-ae5c-052e9d44b377' for connections
to the remote server. Warning: Retrying the sync because a socket
error (10054) occurred. Retrying operation 'Serialization' on object
MSDeploy.contentPath (sourcePath). Attempt 1 of 25. Info: Using ID
'a6e468ca-53b9-474b-8568-42dba196bd72' for connections to the remote
server. Error Code: ERROR_COULD_NOT_CONNECT_TO_REMOTESVC More
Information: Could not connect to the destination computer
("192.168.1.14" using the specified process ("Web Deployment Agent
Service") because the serve did not respond. Make sure that the
process ("Web Deployment Agent Service") i started on the destination
computer. Error: The remote server returned an error: (503) Server
Unavailable. Error count: 1.
Event Log Entry:
Application: MsDepSvc.exe Framework Version: v4.0.30319 Description:
The process was terminated due to an unhandled exception. Exception
Info: System.IO.FileNotFoundException Stack: at
Microsoft.Web.Deployment.Tracer.TraceError(Microsoft.Web.Deployment.DeploymentTraceSource,
System.String, System.Object[]) at
MSDepSvc.WDAgent.BeginProcessRequest(System.Net.HttpListenerContext)
at System.Net.LazyAsyncResult.Complete(IntPtr) at
System.Net.ListenerAsyncResult.IOCompleted(System.Net.ListenerAsyncResult,
UInt32, UInt32) at
System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32,
UInt32, System.Threading.NativeOverlapped*)
How can I diagnose this?
We ran into a similar issue. You're best bet is to decompile using ILSpy and add extra logging in every catch block and replace the exe on the client to see the exact error. In our scenario, we used on older version 7.1.1631.0 that works. There's a newer version 7.1.1762.0 that's caused the problem. The exe is copied from the client and not the server.
I had the exact same issue. It turned out to be a bug in Web Deploy 3.5 as described here: Could not find file or assembly Microsoft.Web.Deployment.Tracing with Web Deploy V3 using tempagent

Resources