I have a large number of Azure WebJobs that all deploy to a single Azure App Service, along with a Website on the same Azure App Service. Each WebJob all use the WebJobs SDK and the Microsoft.Web.WebJobs.Publish nuget package (we are up-to-date with version 1.0.13) to package them up for deployment. The following are the MSBuild arguments we use in the CI build (VSTS) to produce the deploy package:
/p:DeployOnBuild=true /p:PublishProfile=Release /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation=$(Build.StagingDirectory)
This produces a package that works fine (in that the WebJobs run correctly as WebJobs) with the "Azure Web Service Deploy" VSTS task, that isn't the problem.
The problem is that the .zip file package duplicates all the WebJob assemblies. These duplicates end up making it out to the Azure App Service.
The folder structure in each WebJob package's .zip file is:
- Content/[build agent full path]/
- app_data/jobs/continuous/[web job name]/[assembly files]
- bin/[assembly files]
This causes a problem for 3 reasons:
We've started investigating partnering with a security vendor who will perform static analysis our our deployment packages. This duplication causes reporting issues.
Because the bin/[assembly files] makes it out to the App Service, they get inter-mixed with the assemblies for the Website that is also deployed to the same App Service.
Our build / release time is slowed down by transferring all this extra bloat in these deployment packages. My team practices Continuous Delivery and expects the pipeline to be fast.
So why does the Microsoft.Web.WebJobs.Publish package add in the bin/[assembly files] in addition to the required app_data/jobs/continuous/[WebJobName]/[assembly files]? And more importantly, how can I prevent the packaging process from including bin/[assembly files]?
I'd really hate to have to add build steps to piece apart the zipped packages and put it back together without the extra junk, or have to figure out a way to hand-craft the publish package. You have 1 job, Microsoft.Web.WebJobs.Publish! :)
My update to https://stackoverflow.com/a/44611520/8654143
I had to add more than one override for existing targets to make it work:
<Target Name="CollectFilesFromContent" />
<Target Name="CollectFilesFromReference" />
<Target Name="CollectFilesFromIntermediateAssembly" />
<Target Name="CollectFilesFrom_SourceItemsToCopyToOutputDirectory" />
How to find it?
1) Run msbuild with /verbosity:diag
2) Inspect the log and look for DestinationRelativePath=bin; there will be related entry FromTarget=SomeTargetName
3) Add <Target Name="SomeTargetName" /> to your .csproj
You should end with entries marked with FromTarget=PublishWebJob.
I think it is possible to filter contents of FilesForPackagingFromProject variable somehow (probably override CustomCollectFiles target and modify FilesForPackagingFromProject there) but I don't know msbuild good enough.
Sorry to hear that you are running into this issue. As a temporary work-around you can override this target(CollectFilesFromIntermediateAssembly) in the csproj. This will stop the files from getting published to the bin folder.
Sample here:
https://github.com/vijayrkn/ASPNetPublishSamples/blob/master/WebJobFullFramework/WebJobFullFramework.csproj#L56-L57
We will fix this issue in the WebJobs.Publish NuGet package soon.
Related
Has anyone used Azure DevOps to deploy an Umbraco v8 application to Azure?
I can successfully deploy via Visual Studio, but Azure DevOps only seems to copy over all the dlls to the target, no views or anything, and a zip does not get generated.
If anyone has managed this could they share the steps they took please?
The steps I have in the pipeline are:
Build
Test
Deploy
Publish Symbols
Copy File to
Publish Artifact
What I did notice is that I need a publish profile to successfully publish from Visual Studio, however I cant see where a profile should be included in Devops, unless it should be in the build arguments?
My current build arguments:
/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactstagingdirectory)\"
I'm using the Visual Studio Build task.
To straight up answer your question, yes MSBuild arguments make or break the deployment here.You can see mine in my screenshot. I took it from a production deployment that's successful for Umbraco.
So I can't see your actual pipeline, but from what I see you don't actually 'build' the application.
As in you have to use Nuget first, then restore the packages, then build the application. You don't need to copy anything (so you don't need a copy step), with the correct MS Build arguments it will build and copy to build.artifactstagingdirectory on its own. Then you need to have the appropriate release pipeline, but one thing at a time. The build steps I use are underneath. After the npm custom step is the actual building of the application.
P.S. Make sure that the appropriate files to run the application are included in the csproj, because that's where azure devops reads what to pick and build.
I'll assume that you're simply copying files onto a VM and that you have dealt with the deployment of your database / content.
If you want to copy all the files from your Visual Studio project you need to ensure that your MSBuild will copy them into the build/artifact directory.
In order to copy the files that you're missing you need to ensure that you have set the “Build Action” to “Copy to output directory” for example your views or pretty much anything that you'd code in App_Data. This is how you'd set the build action in VS: https://learn.microsoft.com/en-us/visualstudio/ide/build-actions?view=vs-2019
We have 2 Azure Webjobs connected to our ASP WebApi application. None of them is using any dependencies from the other one. And yet, after publishing, one of them has executables from the other one. It's worth mantioning that this only happens on publish. Everything is normal on VS build.
This is how files hierachy looks like on Azure FTP
The first one, Deployment, is being published as expected. Those are executables in it's folder:
The second one, EmailSender, has executables from the Deployment:
What's curious, there is also app.publish folder in both of them, containing only one and the same file WebJob.Deployment.exe:
Deployment job works fine. Unfortunately Azure don't recognize EmailSender job, instead it executes Deployment. The only solution that works right now is to manualy delete Deployment's executables directly from FTP, on every publish.
Right now we tried couple of things from SO and blogs, but with no success.
Microsoft.Web.WebJobs.Publish producing duplicate assemblies in deploy package
Azure Webjobs not getting updated after new publish
GitHub solution
Azure Web Job Run Command Incorrectly Set
Edit:
I did accomplished something. It did not resolve the problem, but we don't have app.publish folder anymore. Here is the link to solution on SO. I don't know why we had 'ClickOnce security' options checked for WebJob.Deployment application.
Update:
I run few tests with MSBuild and found something curious. As I said before, Visual Studio Publish works just fine - no additional executables are deployed. But when I run MSBuildprogram (with the same publish profile and project configuration) I got additional Deployment.exe inside EmailsSender folder. This is the command I run:
MSBuild RestAPI.Host.csproj /t:Build /p:Configuration="Develop" /p:Platform="AnyCPU" /p:DocumentationFile="RestAPI.Host.XML" /p:DeployOnBuild="true" /p:PublishProfile="fakebuild_develop.pubxml" /p:OutputPath="backend\build\app\\" /p:SolutionDir="backend\\"
Can someone tell me what's the difference between MSBuild and VS Publish? I cannot find anything usefull on the internet.
I am trying to setup the CI for Azure Web Jobs using local TFS 2013. The build server itself is fully patched and has Build Tools 2017 installed. I have tried a standalone Web Jobs project in a solution as well as a solution with a Web project and Web Jobs project. From VS2017 the publish (in case of Web project and Web Jobs project) and Publish as WebJobs (in case of standalone Web Jobs project) work just fine and everything gets deployed. NuGet package Microsoft.Web.WebJobs.Publish is included. In both scenarios, the local build/publish publish properly website as well as jobs in app_data/... folder but TFS builds don't even create app_data folders. It seems the final collection of files for publishing is trying to pick the files from WebJobs/bin/** folder which isn't even created in TFS build. The TFS builds add additional argument /p:OutDir="C:\Builds\2\iPayment.Core\Dev-iPayment.ServiceBus.Agents\Binaries\\ and the assembly files for WebJobs are not collected and thus not dropped into app_data/... The build parameters are passed as
/p:PublishProfile=ReleaseWebDeploy
/p:_DestinationType=AzureWebSite
/p:DeployOnBuild=True
/p:WebPublishMethod=Package
/p:PackageAsSingleFile=true
/p:SkipInvalidConfigurations=true
The symptoms are similar to what are described in http://chamindac.blogspot.com/2015/08/package-azure-web-job-with-tfs-build-to.html but my version of XAML build is slightly different than in the blog post and there is nothing to override the OutDir without probably modifying the build template itself.
In a standalone WebJobs project, if I use the MSBuild command line without the OutDir, the bin/** are created and packaged properly and get deployed to Azure but that option doesn't work for the solution where both WebProject and WebJobs are in a single solution and need to be deployed together. Also, I could not find any way to empty the OutDir outside of XAML definition.
I have confirmed the presence of webjob-publish-settings.json (in both cases) and webjobs-list.json (in case of where solution has WebProject as well as WebJobs), which is actually needed and works fine from VS2017 deployment. The publish profile is also correct and it is used by the VS2017 deployment. The TFS build is configured to build the solution in both cases. Microsoft.Web.WebJobs.Publish.1.1.0 is used which is the latest version.
It turns out that the OutputPath property was not being set in our template. There is no "Output Location" to be set as "As Configured". I ended up passing MSBuild Arguments and modifying the template to do the replacement for $(OutDir) in two places
MSBuildArguments.Replace("$(OutDir)", [BinariesDirectory]))] and MSBuildArguments.Replace("$(OutDir)", [outputDirectory]))] respectively for <mtbwa:MSBuild ...> under "Try to Compile the Project" and "For Each Configuration in BuildSettings.PlatformConfigurations"
This article shows one step closer to an efficient and effective continuous integration workflow for an Azure function. (Thanks to Donna M from Microsoft).
https://blogs.msdn.microsoft.com/appserviceteam/2017/03/16/publishing-a-net-class-library-as-a-function-app/
Rather than re-create our azure functions as web applications, we would like to know exactly what needs to be included in a repository for Kudu or the Azure Function Runtime to automatically perform Nuget Restore and then build the project. Ideally, we could just add whatever is needed to our projects to satisfy the build system.
Previously we had a deploy.cmd script in our repositories according to the Azure Web App convention to orchestrate the restore and build ourselves. It worked in January because Azure Functions are built on web apps, however that has stopped working recently and we found that the folder structure has been rearranged a bit (probably due to the fixes they implemented for locking of .dll files). We understand that deploy.cmd was never officially supported, so we'd just like to know how to modify our existing projects to work in the current version of Azure Functions.
Here's a really good answer on all things nuget & functions deployment:
How can I use NuGet packages in my Azure Functions?
Nuget restore and project build should happen automatically for .csx function apps.
All you need is to follow the folder structure conventions. Here's an example:
Function1/
function.json
run.csx
other.csx
project.json
Function2/
function.json
run.csx
project.json
Shared/
shared.csx
host.json
To make sure that Shared folder code is watched for changes, add Shared to watchDirectories in host.json.
I have an ASP.NET MVC4 app and I'm using the hosted Team Foundation Service to automate builds and deploy to an Azure WebSite.
This works great except for deploying the database changes.
I'm using Fluent Migrator to specify the changes to the database and currently I have a custom implementation of IDatabaseInitializer so that when my Entity Framework context spins up, it does any migrations as necessary. Obviously this leaves the database migration until 'first run' of the application rather than the build and deploy.
What I'd like to be able to do is get the build to attempt the migration before it deploys to azure (and therefore fail the build if there's a problem with the migration).
Is this possible with the hosted build service? Can I add post-build steps as I can with a self hosted inhouse TFS?
Add the following to your .csproj file:
<UsingTask TaskName="FluentMigrator.MSBuild.Migrate" AssemblyFile="$(MSBuildProjectDirectory)\FluentMigrator.MSBuild.dll" />
<Target Name="Migrate">
<Message Text="Migrating database" />
<Migrate Database="sqlserver" Connection="YOUR CONNECTION STRING" Target="$(IntermediateOutputPath)YOUR DLL NAME.dll">
</Migrate>
</Target>
Also make sure the target is added after the Build target like so:
<Project ToolsVersion="4.0" DefaultTargets="Build;Migrate;Test" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
You may have noticed that I load the msbuild task from the project directory. I copied the FluentMigrator.MSBuild.dll AND the FluentMigrator.Runner.dll assemblies into my project folder. When I tried to load the assemblies from the packages folder I got an error saying the assemblies were not in the correct format...