How to only build modified projects when building multiple configurations with MsBuild - visual-c++

I have a solution which builds a number of projects. Some of these projects produce DLLs for redistribution amongst other development teams (for integration). These same DLLs are utilized by our own application and they are primarily modified and updated for the maintenance of this application.
We have run into the problem that unwanted dependencies are sometimes added to the DLLs we provide to third parties. As such, we had created a new configuration within our visual studio solution which only builds the DLLs we provide to third parties. By building this configuration we are able to ensure that our gated-checkin will fail if any new dependencies are added for these projects.
<!-- Build only the Database Access provider DLLs, if a new, unwanted dependency is added, this MsBuild task will fail -->
<Message Text="Build Database Access Providers"/>
<MSBuild Condition="'$(Configuration)'=='Release'" Properties="Configuration=Database Providers Release; Platform=Mixed Platforms" Projects="#(MySolution)" Targets="Build" BuildInParallel="$(BuildInParallel)" />
<!-- ... -->
<!-- Build all the projects in the solution -->
<MSBuild Properties="Configuration=$(Configuration); Platform=Mixed Platforms" Projects="#(MySolution)" Targets="Build" BuildInParallel="$(BuildInParallel)" />
The “Database Providers Release” configuration builds each of the required projects in the project’s “Release” configuration.
Because the first call to MsBuild builds the individual projects in the Release configuration and the full build also builds the individual projects in the Release configuration, we thought that the “Build” target would detect that the build outputs are up to date and would skip rebuilding these projects a 2nd time when we execute the second call to MsBuild. However, the MsBuild script will always rebuild the projects in the second call to the MsBuild task. This is increasing our build time and impacts our build’s code metrics (for instance the number of build warnings increases because the same warnings are reported twice).
If I simply invoke the solution as a whole twice (using the identical build configuration at the solution level), MsBuild will correctly skip all the solution items (I.e. it performs incremental building of the projects). E.g:
<!-- Build all the projects in the solution -->
<MSBuild Properties="Configuration=$(Configuration); Platform=Mixed Platforms" Projects="#(MySolution)" Targets="Build" BuildInParallel="$(BuildInParallel)" />
<!— This 2nd build will skip every project as MsBuild detects the projects have been built. -->
<MSBuild Properties="Configuration=$(Configuration); Platform=Mixed Platforms" Projects="#(MySolution)" Targets="Build" BuildInParallel="$(BuildInParallel)" />
If I run the 2 different build configurations within of Visual Studio directly, it will correctly detect that the various projects have already been built and will skip rebuilding them a 2nd time (I.e. the setup for incremental building of the vcxprojs is correct.)
Is there a way to build 2 solution configurations with MsBuild (where each solution configuration is using the same project configurations) and have MsBuild skip projects which were already built in the first build when they are encountered in the 2nd build?

Related

Microsoft.Web.WebJobs.Publish producing duplicate assemblies in deploy package

I have a large number of Azure WebJobs that all deploy to a single Azure App Service, along with a Website on the same Azure App Service. Each WebJob all use the WebJobs SDK and the Microsoft.Web.WebJobs.Publish nuget package (we are up-to-date with version 1.0.13) to package them up for deployment. The following are the MSBuild arguments we use in the CI build (VSTS) to produce the deploy package:
/p:DeployOnBuild=true /p:PublishProfile=Release /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation=$(Build.StagingDirectory)
This produces a package that works fine (in that the WebJobs run correctly as WebJobs) with the "Azure Web Service Deploy" VSTS task, that isn't the problem.
The problem is that the .zip file package duplicates all the WebJob assemblies. These duplicates end up making it out to the Azure App Service.
The folder structure in each WebJob package's .zip file is:
- Content/[build agent full path]/
- app_data/jobs/continuous/[web job name]/[assembly files]
- bin/[assembly files]
This causes a problem for 3 reasons:
We've started investigating partnering with a security vendor who will perform static analysis our our deployment packages. This duplication causes reporting issues.
Because the bin/[assembly files] makes it out to the App Service, they get inter-mixed with the assemblies for the Website that is also deployed to the same App Service.
Our build / release time is slowed down by transferring all this extra bloat in these deployment packages. My team practices Continuous Delivery and expects the pipeline to be fast.
So why does the Microsoft.Web.WebJobs.Publish package add in the bin/[assembly files] in addition to the required app_data/jobs/continuous/[WebJobName]/[assembly files]? And more importantly, how can I prevent the packaging process from including bin/[assembly files]?
I'd really hate to have to add build steps to piece apart the zipped packages and put it back together without the extra junk, or have to figure out a way to hand-craft the publish package. You have 1 job, Microsoft.Web.WebJobs.Publish! :)
My update to https://stackoverflow.com/a/44611520/8654143
I had to add more than one override for existing targets to make it work:
<Target Name="CollectFilesFromContent" />
<Target Name="CollectFilesFromReference" />
<Target Name="CollectFilesFromIntermediateAssembly" />
<Target Name="CollectFilesFrom_SourceItemsToCopyToOutputDirectory" />
How to find it?
1) Run msbuild with /verbosity:diag
2) Inspect the log and look for DestinationRelativePath=bin; there will be related entry FromTarget=SomeTargetName
3) Add <Target Name="SomeTargetName" /> to your .csproj
You should end with entries marked with FromTarget=PublishWebJob.
I think it is possible to filter contents of FilesForPackagingFromProject variable somehow (probably override CustomCollectFiles target and modify FilesForPackagingFromProject there) but I don't know msbuild good enough.
Sorry to hear that you are running into this issue. As a temporary work-around you can override this target(CollectFilesFromIntermediateAssembly) in the csproj. This will stop the files from getting published to the bin folder.
Sample here:
https://github.com/vijayrkn/ASPNetPublishSamples/blob/master/WebJobFullFramework/WebJobFullFramework.csproj#L56-L57
We will fix this issue in the WebJobs.Publish NuGet package soon.

How to build unit test project when performing an Azure Cloud service CI build via Visual Studio Online?

I have a Visual Studio solution housing an Azure Cloud Service with the following projects:
CloudService
CloudServiceRole
Tests
Where the Tests project is a standard MSTest project that contains unit tests for the business logic in the CloudServiceRole project.
The code is stored in Visual Studio Online and I have hooked up the automated CI build deployment that Azure offers. When I check in code, my staging deployment of the cloud service is automatically updated. However, the Tests project is never even built during the CI builds! This, of course, means that no unit tests run during the build as the "run unit tests" part of the build process finds no assemblies with tests.
My goal is to change this so the tests project is built and all the unit tests executed.
Looking at the MSBuild arguments that the CI deployment process uses, it appears that only the CloudService:Publish target is executed. The CloudService project has no dependency on the Tests project so MSBuild never even builds the latter.
What I have tried
I cannot manually add a CloudService->Tests dependency because when I add dependencies on projects that are not Cloud Service Role projects, I get an error during build (The item "C:\a\src\MyProject\Tests\Tests.csproj" in item list "ProjectReferenceWithConfiguration" does not define a value for metadata "Name".) and I cannot add a CloudServiceRole->Tests dependency because that would make a circular dependency.
Instructing MSBuild to build the full solution by manually adding a /t:Build parameter resulted in yet another error: C:\a\bin\ServiceDefinition.csdef: Need to specify the physical directory for the virtual path 'Web/' of role Web.
Adding the Tests project as a separate build target, alongside the solution, results in the tests getting built! However, at the same time, it disables the Continuous Deployment functionality: More than one solution found. Continuous Deployment skipped.
Trying to make a fake Cloud Service Role project that references the Tests project but has zero instances configured results in a build error: WAT100: The following roles 'Tests.FakeRole' have an instance count of 0. Instance count of 0 is not supported in deployments to Microsoft Azure. Attempting to disable this validation results in a build error due to a defect in the Azure SDK.
You need to run a Build and a Publish separately. I ran into the same problem on my VSO (now VSTS) project and this fixed it. This happens because your cloud service doesn't depend on your unit test project.
1) Visual Studio Build (or MSBuild) action with arguments /t:Build (clean here)
2) Visual Studio Build (or MSBuild) action with arguments /t:Publish (do not clean here)
Note: I had to run these actions separately (not /Build;Publish) otherwise I got an error about the cloud service entry point.
Pieced this together from this question and from here and here.
One workaround that appears to bring results is to add a pre-MSBuild script to the build definition and explicitly build the Tests project in that script.
:: This is used as a pre-build script in Continuous Deployment builds because on their own, they do not build the Tests project.
"c:\Windows\Microsoft.NET\Framework64\v4.0.30319\MSBuild.exe" %~dp0\..\..\Tests\Tests.csproj /t:Build /p:Configuration=Debug;Platform=AnyCPU;OutDir="%~dp0\..\..\..\..\bin\\"
It appears to do the job, although I am not sure what side-effects I should be aware of. My main concern is that the binaries from this build go into the same directory as the binaries from the Cloud Service build - is there perhaps some possibility of conflict here?

Separate TFS Build excluding a single project

I have a solution that currently builds nicely in TFS. I have just added a Sandcastle help File Builder project to it so that the help files can be generated. Now the solutions takes ages to build locally (it has also doubled the build time on the TFS Server).
In practice I'd be happy generating the help files once per day, or only when specifically requested as part of a build.
I tried creating a separate solution (for a new build) which included the Sandcastle project but I got a warning saying that the projects were already under source control.
What is the best way to configure TFS/the solution for this sort of situation? Can I have two builds that run on separate schedules and include/exclude certain projects in the solution?
You can create a new solution configuration that excludes building your help project. Then in your TFS build you can specify which configuration to build. You could have a CI build that doesn't include your help project, and then a nightly build that includes everything.

Team City with Visual Studio solution build steps

I have a Visual Studio 2012 solution containing a Windows service project and a web application project.
I want Team City (version 8.0.3) to create two zip files (one for the service and one for the web app) that I will deploy manually.
Should I create a build step to build the entire solution, followed by a build step to publish the Windows service and a build step to publish the web site (via publish profiles). Then use Artifact paths in General Settings to zip up these two published folders?
Or should I have just one build step to build the solution and then use the Artifact paths to create the two zip files?
Or is there a better way than either of the above?
You have to ask yourself if these two projects are linked and in which manner they should be built together.
My feeling is : if your projects are in the same solution, they are linked in some ways and have to be built together.
Then, you should build your solution (sln) and not projects (*proj).
Application organization
Generally, your build server should not redefine -too much- the way your applications are organized. You should always use your plateform application descriptor to build your applications.
In case of .NET and Visual Studio, the application descriptor is your solution (sln). It defines the needs and how your application have to be built.
If your project have to be built separately, they should be in differents solution unless you prefer to create specific solution configuration (in addition to Release & Debug).
Anyway, the solution is still the build entry point.
TeamCity
Speaking about TeamCity, different and standalone applications should be in separate build configurations.
The pure build (code compilation) should be in one build step and you should not use too much code compilation runners in one build configuration.
Your build configurations should reflect your applications farm logic.
If you need to link them in some ways which are related to packaging for example, you can link your build configurations through snapshot or artifact dependency.

CruiseControl.NET issue when multiple projects use shared projects

I have CruiseControl.NET configured to monitor 10+ .NET projects. All of these projects (web apps, windows services, wcf services, etc...) use shared class library projects so in the ccnet.config I had to set the svn path for each project to the root of the svn repo. If I didn't do it this way then a change to a shared assembly wouldn't trigger projects that depend on it to rebuild.
The issue is that because I've set the svn path for each project to the root of the repo then that means that any change at all triggers a rebuild of every single project, which takes a really long time. How do you get around this issue with using shared assemblies in multiple projects without having every single project rebuild every time a little change is made??
Here's another way to see the issue:
CC Project #1 = svn://repo/WebApps/WebsiteA (references svn://repo/Shared/ClassLibraryA)
CC Project #2 = svn://repo/WebApps/WebsiteB (references svn://repo/Shared/ClassLibraryB)
For CC Project #1, you can't set the svn path to svn://repo/WebApps/WebSiteA, as if you did and ClassLibraryA changed then it wouldn't trigger a build. However if you set the path to svn://repo, then it'll pick up the changes to the ClassLibraryA, but then it'd also trigger CC Project #2.
Any suggestions would be greatly appreciated...
You may want to use the Project Trigger to start a build :
http://confluence.public.thoughtworks.org/display/CCNET/Project+Trigger
If your ClassLibraryA has a project on CruiseControl (whose svnpath is svn://repo/Shared/ClassLibraryA) then your WebsiteA would look like :
<project name="WebSiteA">
<triggers>
<projectTrigger project="ClassLibraryA">
<triggerStatus>Success</triggerStatus>
<innerTrigger name="ClassLibraryA" type="intervalTrigger" seconds="60" buildCondition="ForceBuild" />
</projectTrigger>
<intervalTrigger seconds="300"/>
</triggers>
<cb:svn-block svnpath="repo/WebApps/WebsiteA" />
<tasks>
<...>
</tasks>
<publishers>
<...>
</publishers>
</project>
The answer was to use the sourcecontrol multi block in CruiseControl.NET, which allows you to specify multiple svn paths for each project:
http://ccnet.sourceforge.net/CCNET/Multi%20Source%20Control%20Block.html

Resources