Deployment
I currently work for a company that deploys through github. However, we have to log in to all 3 servers to update them manually with a shell script. When talking to the CTO he made it very clear that auto-deployment is like voodoo to him. Which is understandable. We have developers in 4 different countries working remotely. If someone where to accidentally push to the wrong branch we could experience downtime, and with our service we cannot be down for more than 10 minutes. And with all of our developers in different timezones, our CTO wouldn't know till the next morning and we'd have trouble meeting with the developers who had the issue because of vast time differences.
My Problem: Why I want auto-deploy
While working on my personal project I decided that it may be in my best interest to use auto-deployment, but still my project is mission critical and I'd like to mitigate downtime and human error as much as possible. The problem with manual deployment is that I simply cannot manually deploy on up to 20 servers via SSH in a reasonable amount of time. The problem perpetuates when I consider auto-scaling. I'd need to spin up a new server from an image and deploy to it.
My Stack
My service is developed on the Node.js Express framework. These environments are very rich in deployment and bootstraping utilities. My project uses npm's package.json to uglify my scripts on deploy, and also runs my service as a daemon using forever-monitor. I'm also considering grunt.js to further bootstrap my environments for both production and testing environments.
Deployment Methods
I've considered so far:
Auto-deploy with git, using webhooks
Deploying manually with git via shell
Deploying with npm via shell
Docker
I'm not well versed in technologies like Docker, but I'm interested and I'd definitely give points to whoever gave me a good description as to why I should or shouldn't use Docker, because I'm very interested in its use. Other methods are welcome.
My Problem: Why I fear auto-deploy
In a mission critical environment downtime can put your business on hold, and to make matters worse there's a fleet of end users hitting the refresh button. If someone pushes something that's not build passing to the production branch and that's auto-deployed, then I'm looking at a very messy situation.
I love the elegance of auto-deployment, but the risks make me skeptical. I'm very much in favor of making myself as productive as possible. So I'm looking for a way to deploy to many servers with ease, and in very efficient manner.
The Answer I'm Looking For
Explain to me how to mitigate the risks of auto-deployment, or explain to me an alternative which is better suited to my project. Feel free to ask for any missing details in the comments.
No simple answer here. I offer a set of slides published by Mike Brittain from Etsy, a company that practices continuous deployment:
http://www.slideshare.net/mikebrittain/mbrittain-continuous-deploymentalm3public
Selected highlights:
Deploy frequently and in small batches
Use config/feature flags to control system behaviour and "dark release" major features
Code review all changes to the production branch
Invest in monitoring and improve the feedback loop
Manage "services" separately to the "application" and be mindful of run-time version and backwardly compatible changes.
Hope this helps
Related
First of all, I'm not really sure if this question goes here in stackoverflow or if I should ask it on another place. Please if that's the case, indicate me in the right way :)
So, for context, this is an app that I was asked to develop for my job. At first I thought in doing a webapp and host it inside the company servers and domain (intranet), but it isn't possible due to external issues that I can't control.
Is there another way to achieve this? The app must have a database and should be accessible for a bunch of users at the same time.
Of course we want to spend the least amount of money possible to make this happen. Also, using a workstation of our own to host everything is not possible either.
Edit: I didn't finish developing, but for now I'm developing it in Python Flask.
The number of users is small really, just up to five people.
OK - I guess a lot of what you'll get in response to this is your definition is too vague. Things such as scale, number of users, programming languages used to create the web app etc are important when talking about hosting.
However, for me, there are three very good options out there for free hosting, up to a certain amount of traffic.
1.) Heroku - Heroku.com
A world known web hosting platform. You can publish code through GitHub, and it has some extensive coverage for different types of web apps. Definitely worth a look.
2.) Netlify - netlify.com
Similar to Heroku, but used by some major companies. Allows you to host for free to a point, and is relatively simple to get started with.
3.) Vercel - vercel.com
A bit more technical in my opinion - but again, very similar to the above two and has a free tier.
All three are great options, and I'd recommend looking into them in more detail to see what option is best for you. Can't go wrong with any of them.
I had a similar problem: A Python-Flask-SQLite app for me and my office pals to use together.
The solution was creating one .exe file with pyinstaller, hosting this and the database files in a network drive (one that everyone that will use the app has access). As everybody (~10 people) sees the same db, things works fine!
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 2 years ago.
Improve this question
We have around 12 services and other applications such as presto.
We are thinking about building Docker containers for each service and application. Is it right to dockerize all of them?
When would a Docker container not be the ideal solution?
PROS:
Quick local environment set up for your team - if you have all your services containerized. It will be a quick environment set up for your development team.
Helps Avoid the "It works on mine, but doesn't work on yours problem" - a lot of our development issue usually stems from development environment setup. If you have your services containerized, a big chunk of this gets offloaded somewhere else.
Easier deployments - while we all have different processes for deploying code, it goes to tell that having them containerized makes thing a hell lot easier.
Better Version Control - as you already know, can be tagged, which helps in VERSION CONTROL.
Easier Rollbacks - since you have things version controlled, it goes to say that it is easier to rollback your code. Sometimes, by just simply pointing to your previously working version.
Easy Multi-environment Setup - as most development teams do, we set up a local, integration, staging and production environment. This is done easier when services are containerized, and, most of the times, with just a switch of ENVIRONMENT VARIABLES.
Community Support - we have a strong community of software engineers who continuously contribute great images that can be reused for developing great software. You can leverage that support. Why re-invent the wheel, right?
Many more.. but there's a lot of great blogs out there you can read that from. =)
CONS:
I don't really see much cons with it but here's one I can think of.
Learning Curve - yes, it does have some learning curve. But from what I have seen from my junior engineers, it doesn't take too much time to learn how to set it up. It usually takes you longer when you are figuring out how to containerized.
SOME CONCERNS:
Data Persistence - some engineers are having concerns with data persistence. You can simply fix this by mounting a volume to your container. If you want to use your own database installation, you can simply switch your HOST, DB_NAME, USERNAME and PASSWORD with the one you have in your localhost:5432 and all should be fine.
I hope this helps!
You should containerize all Linux-based services that are stateless and require frequent upgrades/changes/patches. These include all types of front-end and application servers.
Databases/datastores, on the other hand, are a more complex case, since there are issues of performance and data persistence/integrity. Also, databases are not upgraded/patched as frequently as front-end applications.
*Windows containers will only run in Windows.
Docker is a recipe for consistency and reproducibility.
To make a nice cup of tea, you need boiling water, put some tea bag in it and let it brew for three minutes. How you achieve boiling water is absolutely irrelevant.
Now let's imagine that you need to serve up 12 cups of tea. Does your staff know how to make a proper brew? Does your staff know how to use a kettle or a pan? What guarantee do you have that each cup of tea will be the same?
You could spend a lot of time training people and make you sure you have all the appliances you need. Or you can invest in a machine that will produce the same cup of tea over and over again.
The analogy may seem stupid but my point is that relatively common problems already have well-known solutions.
Unless it's a one-off scenario or you have additional constraints we don't know about, what reasons do you have to not consider Docker?
There is no issue with dockerizing multiple services. I think you need to consider about following things too.
You have to think about how to save your data you have used inside the container. By default the data inside the container will destroy the the container shuts down. You may have to mount a volume in order to keep the data permanently.
You may not be able to get the bare-metal performance when running in docker.
IMO It's not a good choice if you are going to run all the applications in docker unless you need to take the advantage of containerization. But it is easy to run stateless applications and services with docker.
Currently we are running a C# (built on Sharepoint) project and have implemented a series of automated process to help delivery, here are the details.
Continuous Integration. Typical CI system for frequent compilation and deployment in DEV environment.
Partial Package. Every week, a list of defects accompanied fixes is identified and corresponding assemblies are fetched from the full package to form a partial package. The partial package is deployed and tested in subsequent environments.
In this pipeline, there are two packages going through are being verified. Extra effort is used to build up a new system (web site, scripts, process, etc) for partial packages. However, some factors hinder its improvement.
Build and deploy time is too long. On developers' machines, every single modification on assemblies triggers around 5 to 10 minute redeployment in IIS. In addition, it takes 15 minutes (or even more) to rebuild the whole solution. (The most painful part of this project)
Geographical difference. Every final package is delivered to another office, so manual operation is inevitable and package size is preferred to be small.
I will be really grateful to have your opinions to push the Continuous Delivery practices forward. Thanks!
I imagine the reason that this question has no answers is because its scope is too large. There are far too many variables that need to be eliminated, but I'll try to help. I'm not sure of your skill level either so my apologies in advance for the basics, but I think they'll help improve and better focus your question.
Scope your problem to as narrow as possible
"Too long" is a very subjective term. I know of some larger projects that would love to see 15 minute build times. Given your question there's no way to know if you are experiencing a configuration problem or an infrastructure problem. An example of a configuration issue would be, are your projects taking full advantage of multiple cores by being built parallel /m switch? An example of an infrastructure issue would be if you're trying to move large amounts of data over a slow line using ineffective or defective hardware. It sounds like you are seeing the same times across different machines so you may want to focus on configuration.
Break down your build into "tasks" and each task into the most concise steps possible
This will do the most to help you tune your configuration and understand what you need to better orchestrate. If you are building a solution using a CI server you are probably running using a command like msbuild.exe OurProduct.sln which is the right way to get something up and running fast so there IS some feedback. But in order to optimize, this solution will need to be broken down into independent projects. If you find one project that's causing the bulk of your time sink it may indicate other issues or may just be the core project that everything else depends on. How you handle your build job dependencies is dependent up your CI server and solution. Doing it this way will create more orchestration on your end, but give faster feedback if that's what's required since you're only building the project that had the change, not the complete solution.
I'm not sure what you mean about the "geographical difference" thing. Is this a "push" to the office or a "pull" from the offices? This is a whole other question. HOW are you getting the files there? And why would that require a manual step?
Narrow your scope and do multiple questions and you will probably get better (not to mention shorter and more concise) answers.
Best!
I'm not a C# developer, but the principles remain the same.
To speed up your builds, it will be necessary to break your application up in smaller chunks if possible. If that's not possible, then you've got bigger problems to attack right now. Remember the principles of API's, components and separation of concerns. If you're not familiar with these principles, it's definitely worth the time to learn about them.
In terms of deployment - great that you've automated it, but it sounds exactly the same as you are building a big-bang deployment. Can you think of a way to deploy only deltas to the server(s), are do you deploy a single compressed file? Break it up if possible.
I have a number of Windows servers at work that are used for staging web sites for clients while they are being created.
I wanted to start using versioning on them so that when we work with outside vendors on a project, if/when they overwrite my work, I'd like to be able to go back and get the version before.
My question is that I think I'm not looking for the correct terms in searching for information, but what kind of resources are there to learn how to install the software for versioning or a site to help me get started.
Any and all suggestions would be appreciated.
Steph
Since your development workflow can be decentralized (as in "there isn't always one central repository), DVCS tools, with their common tasks described here) can be more adapted.
Git-Scm
Mercurial (see HgInit.com for a very good tutorial like the kind you are after)
Plastic SCM (which has a DVCS nature)
Weird question, perhaps. We have a number of simple utilities written in-house that need to be run on an automated basis. These are not build jobs. Just things like running SendOutHourlyEmailAlarms.exe, KeepFoldersInSynch.exe and such. I would normally set these things up as simple scheduled tasks/AT commands (or a Windows Service if more granular control is needed over the scheduling), but a co-worker has set up a number of these tasks as build projects on the CruiseControl.NET server. I asked him why he set these up this way and his response was that the executions (and their logs, return values, thrown exceptions) were all tracked and logged and that this information was accessible through an organized interface on the build server website. I couldn't argue with this.
But this just has a smell that I can't quite identify. Is this a proper use of CruiseControl.NET? If not, what are the dangers? Even if it may fit the bill, aren't there other products better suited for this type of thing?
We have all sorts of non-build related tasks for the exact same reason as your coworker had, I want one spot to look up any and all jobs I need run.
Some Examples of our CC.NET projects:
FTP installers to Remote QA
Creating Source Code Documentation
Create VM's with the installers
installed for QA in the morning
Archiving Installers
Pretty much anything I have to do by hand more than once, becomes a project. IMHO it is much better than a scheduled task for one other reason as well. Our config files are in source control, so we have 1 place to make adjustments. We do not have to log into multiple servers and make adjustments or wonder which server did that.
I think your coworker has made a good argument. If these tasks are related to the development process, then placing them in CruesControl.Net as a project seems acceptable. I would draw the line at utilizing a development server to run production processes though. Although it is true that "If the only tool you have is a hammer, you tend to see every problem as a nail," it doesn't mean that the hammer isn't capable of solving a lot of problems!
Just because a tool is designed to solve a particular problem does not mean that it will not have equal facility at solving similar problems outside the scope originally concieved by the tool creator. If CruiseControl.NET solves these problems well, then it is absolutely the appropriate tool to use.