I'm working on a project in Azure DevOps and, as recommended in the doc, I created an Artifacts Feed with maven central as upstream source to store all my dependencies (I don't really need to publish artifacts for now).
So I configured my local maven to fetch all the dependencies from my feed instead of maven central and it all works fine, except that it's very slow compared to maven central.
When I start from an empty .m2 on my local machine, it takes 1 min 15 secs to build my project when downloading the dependencies from maven central, but it takes over 8 minutes to do the same when downloading the dependencies from the Feed (which contains already all the dependencies).
I could live with that, since the download of everything happens only on the first build.
But the issue is that it's also slower when building my project from Azure Pipelines, which I really didn't expect since it's a connection from Azure to Azure and within the same organization. In this case, it takes at least twice the time when using the feed rather than maven central. And this will be true every time since Azure Pipelines gives you a fresh VM each time you build (I'm using a hosted agent), so there's no dependencies caching in this case.
It's really annoying since my project is just a HelloWorld so far, so it will only get worse over time.
Using a repository manager/feed is the best practice according to both Maven and Azure, but at this point I'm really thinking of going for the bad practice of getting everything from maven central instead of my feed, at least in my pipeline, to improve the performance.
Am I the only one having this issue ? What are your thoughts about this ?
Finally, after diving into the documentation for Azure Pipelines recently, I found out there is a way to cache the maven repository between runs so it partially solves my issue since the full download of the dependencies will happen only once.
Here is the doc in question for those who are interested.
Related
We have a self-hosted build agent on an on-prem server.
We typically have a large codebase, and in the past followed this mechanism with TFS2013 build agents:
Daily check-ins were built to c:\work\tfs\ (taking about 5 minutes)
Each night a batch file would run that did the same build to those folders, using the same sources (they were already 'latest' from the CI build), and build the installers. Copy files to a network location, and send an email to the team detailing the build success/failures. (Taking about 40 minutes)
The key thing there is that for the nightly build there would be no need to get the latest sources, and the disk space required wouldn't grow much. Just by the installer sizes.
To replicate this with Azure Devops, I created two pipelines.
One pipeline that did the CI using MSBuild tasks in the classic editor- works great
Another pipeline in the classic editor that runs our existing powershell script, scheduled at 9pm - works great
However, even though my agent doesn't support parallel builds what's happening is that:
The CI pipeline's folder is c:\work\1\
The Nightly build folder is c:\work\2\
This doubles the amount of disk space we need (10gb to 20gb)
They are the same code files, just built differently.
I have struggled to find a way to say to the agent "please use the same sources folder for all pipelines"
What setting is this, as we have to pay our service provider for extra GB storage otherwise.
Or do I need to change my classic pipelines into Yaml and somehow conditionally branch the build so it knows it's being scheduled and do something different?
Or maybe, stop using a Pipeline for the scheduled build, and use task scheduler in Windows as before?
(I did try looking for the same question - I'm sure I can't be the only one).
There is "workingDirectory" directive available for running scripts in pipeline. This link has details of this - https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/command-line?view=azure-devops&tabs=yaml
The number '1' '2'...'6' of work folder c:\work\1\, c:\work\2\... c:\work\6\ in your build agent which stands for a particular pipeline.
Agent.BuildDirectory
The local path on the agent where all folders for a given build
pipeline are created. This variable has the same value as
Pipeline.Workspace. For example: /home/vsts/work/1
If you have two pipelines, there will also be two corresponding work folders. It's an except behavior. We could not configure pipelines to share the same build folde. This is by designed.
If you need to use less disk space to save cost, afraid stop using a Pipeline for the scheduled build, and use task scheduler in Windows as before is a better way.
I am using Team Services for my application build/deploy but I am finding that the build step is extremely slow. It ranges in time from 6 minutes to sometimes 15 minutes just for the solution build. A large proportion of this time is taken up by the nuget package restore, which can be up to 5 minutes.
The way I see it there are 2 potential ways in which I could speed up the build time, but I am unsure if it is possible to do these things:
Configure Team Services to clone the repository to the same disk location each time it does a build so that it only needs to restore new / remove old nuget packages
Upgrade the power of the build agent
Does anyone know if either of these things are possible, or does anyone have any other tips on how to speed up the build step?
If you're using the hosted queue, it has to clone the repo and restore the packages every time -- you don't get a dedicated agent, so every build is from scratch.
You can set up an on-premise build agent if you need capabilities that exceed those of the hosted agent.
We have currently released our code to Production, and as a result have cut and branched to ensure we can support our current release, whilst still supporting hot-fixes without breaking the current release from any on-going development.
Here is our current structure:
Project-
/Development
/RC1
Until recently using Octopus we have had the following process:
Dev->Staging/Dev Test->UAT
Which works fine as we didn't have an actual release.
My question is how can Octopus support our new way of working?
Do we create a new/clone project in Octopus named RC1 and have CI from our RC1 branch into that? Then add/remove as appropriate as this RC's are no longer required?
Or is there another method that we've clearly missed out on?
It seems that most organisations that are striving for continuous something end up with a CI server and continuous deployment up to some manual sign off environment and then require continuous delivery to production. This generally leads to a branching strategy in order to isolate the release candidate to allow hot fixing.
I think a question like this raises more points for discussion, before trying to provide a one size fits all answer IMHO.
The kind of things that spring to mind are:
Do you have "source code" dependencies or binary ones for any shared components.
What level of integration / automated regression testing do you have.
Is your deployment orchestrated by TFS, or driven by a user in Octopus.
Is there a database as part of the application that needs consideration.
How is your application version numbering controlled.
What is your release cycle.
In the past where I've encountered this scenario, I would look towards a code promotion branching strategy which provides you with one branch to maintain in production - This has worked well where continuous deployment to production is not an option. You can find more branching strategies discussed on the ALM Rangers page on CodePlex
Developers / Testers can continually push code / features / bug fixes through staging / uat. At the point of release the Dev branch is merged to Release branch, which causes a release build and creates a nuget package. This should still be released to Octopus in exactly the same way, only it's a brand new release and not a promotion of a previous release. This would need to ensure that there is no clash on version numbering and so a strategy might be to have a difference in the major number - This would depend on your current setup. This does however, take an opinionated view that the deployment is orchestrated by the build server rather than Octopus Deploy. Primarily TeamCity / TFS calls out to the Ocotpus API, rather than a user choosing the build number in Octopus (we have been known to make mistakes)
ocoto.exe create-release --version GENERATED_BY_BUILD_SERVER
To me, the biggest question I ask clients is "What's the constraint that means you can't continuously deploy to production". Address that constraint (see theory of constraints) and you remove the need to work round an issue that needn't be there in the first place (not always that straight forward I know)
I would strongly advise that you don't clone projects in Octopus for different environments as it's counter intuitive. At the end of the day you're just telling Octopus to go and get this nuget package version for this app, and deploy it to this environment please. If you want to get the package from a different NuGet feed for release, then you could always make use of the custom binding on the NuGet field in Octopus and drive that by a scoped variable depending on the environment you're deploying to.
Step 1 - Setup two feeds
Step 2 - Scope some variables for those feeds
Step 3 - Consume the feed using a custom expression
I hope this helps
This is unfortunately something Octopus doesn't directly have - true support for branching (yet). It's on their roadmap for 3.1 under better branching support. They have been talking about this problem for some time now.
One idea that you mentioned would be to clone your project for each branch. You can do this under the "Settings" tab (on the right-hand side) in your project that you want to clone. This will allow you to duplicate your project and simply rename it to one of your branches - so one PreRelease or Release Candidate project and other is your mainline Dev (I would keep the same name of the project). I'm assuming you have everything in the same project group.
Alternatively you could just change your NuSpec files in your projects in different branches so that you could clearly see what's being deployed at the overview project page or on the dashboard. So for your RC branch, you could just add the suffix -release within the NuSpec in your RC branch which is legal (rules on Semantic Versioning talk about prereleases at rule #9). This way, you can use the same project but have different packages to deploy. If your targeted servers are the same, then this may be the "lighter" or simpler approach compared to cloning.
I blogged about how we do this here:
http://www.alexjamesbrown.com/blog/development/working-branch-deployments-tfs-octopus/
It's a bit of a hack, but in summary:
Create branch in TFS Create branch specific build definition
Create branch specific drop location for Octopack
Create branch specific Octopus Deployment Project (by cloning your ‘main’ deployment
Edit the newly cloned deployment, re-point the nuget feed location to your
branch specific output location, created in step 3
I have a Go package running on Windows and is working fine but now I'm at a stage where I would like to test this on production CentOS 6.5 server.
What is the best practice to deploy this from Windows to CentOS?
Would I have to use my Git repo to distribute to Linux operating system, compile then deploy the binary to the server?
Also I have multiple files, so I would imagine go build *.go would suffice or are there better options for doing compilation?
What is the best practice to deploy this from Windows to CentOS?
As far as best practices go I would recommend using continuous integration. You can setup jenkins, or there are some cloud options out there: codeship.io, travis-ci.org, drone.io, wercker.com, ... Some of them have free plans available.
Basically you'd commit your code to git and push that out to Github (or Bitbucket if you want free private repos). The continuous integration server will be notified whenever you push out changes, and will build, test and create a release tar archive of your project. You can then take this resulting tar and download it to your CentOS box. In 6.5 you'll need to create an init.d script to keep your program up and running. You can see an example here (the system v script).
CentoOS 7 uses systemd now which would be slightly easier to setup.
Taking this one step further it's also possible to setup continuos deployment, in which the download, extraction and installation can also be automated. Depending on your project it may or may not make sense to set up continuous deployment. (Auto-pushing to production might be a little too automatic) You can find an example in wercker here.
Although there is an an up-front cost to setting up continuous integration if this is a project that other people will contribute too, or one that you intend to work on long-term, the cost will definitely be worth it. (Future you will be greatful when you come back to this project 6 months from now, change 1 line of code, and don't have to remember all the manual steps it took to deploy)
I have an Azure Website configured to deploy from a Bitbucket repository. This works fine.
Since the application is still in active development, I update the nuget packages it uses quite frequently. This causes the packages folder to keep growing indefinitely, unless I go and manually delete the packages.
Now, in my local machine this is not a big issue. Space is cheap. But in Azure, this makes us go over the quota really fast, as old packages accumulate.
How can I customize the Azure deploy process so that it deletes all the packages after a successful deployment?
(I am open to other solutions as well)
You can utilize the custom deployment script feature where you add a step that cleans up the packages directory.
You can read about it here:
http://blog.amitapple.com/post/38418009331/azurewebsitecustomdeploymentpart2/
Another option is to add a post deployment action, by adding a script file (.cmd/.bat) that has the clenup logic to the following directory in your site: d:\home\site\deployments\tools\PostDeploymentActions\, this script will run after the deployment completes successfully.
Read more about it here:
https://github.com/projectkudu/kudu/wiki/Post-Deployment-Action-Hooks