My requirement is to design a framework to enforce custom coding standards (per Gitlab project) pre-push or while raising an MR. We are looking at Gitlab CI/ Webhooks / Server-side githooks.
We faced resistance from Gitlab admins that Server-side githooks are not the right way as it may result in a server overload and lead to Gitlab performance issues.
Also, we want to give the developer the choice to commit with or without fixing the issues that crop up after the check (in case of emergency situations). The check should happen on the server side and not at the client-side (developer). The applications may be backend/front-end with technologies like Java/ReactJS/NodeJS etc.
What would be the best way to implement this?
The best way to do this is to have a build stage that runs a linter for your language, and have it fail if the code doesn't meet the standards.
Then enforce rules so that all merges to master must go through a merge request, and all build stages must pass before a request can be approved.
Related
I am building a product using hasura that will be used in production. Is it bad practice security wise to publish the repo with all the configuration and structure? Basically all the database migrations and config.
Of course I won't include any keys or sensitive information.
Is this risky?
The question is 100% about security best practices. I don't mind anyone copying or using the code.
If it is not necessary to publish the configuration and structure, I would not do it. If you have a configuration error, it would be very easy for someone to check your configuration and exploit your rules. Configuration can get hairy, and if you make a mistake and document it in a repository, that could open the database up to attack. Even if you are a database expert and have protected against all threats, there could be a new exploit discovered in the future that can take advantage of certain configurations.
Bottom line. A database's first priority is to be secure and prevent loss. This would not improve security and could lead to problems in the future. In my humble opinion, I would not publish your configuration settings.
Check out this article from berkeley.edu for further research:
https://security.berkeley.edu/education-awareness/best-practices-how-tos/system-application-security/database-hardening-best
I'm trying to setup a git server and I want to allow only a specific user to push commits to master branch.
I have tried to use the Linux group permission setting to meet the requirement above but it seems not a correct way.
And I even don't know what are the key words for searching the answer for this.
Any help would be appreciated.
Git does not allow you to have private branches, but you can achieve this functionality by implementing your own server-side pre-receive hook. Github enterprise specific pre-receive hook example is here, as a reference.
However, if you are using Git hosting services (like Github) they might have an option for this. Github, in particular, has an option called branch restrictions, but it requires you to have a paid subscription, unless your project is public.
You have two options:
By far the easiest solution is to use hosting software that already provides this functionality. You might want to look at GitLab, which has free options for both SaaS (hosted at gitlab.com) and self-managed instances (running your own gitlab instance). Or github. Or bitbucket. Or I'm sure there are others I'm not thinking of.
If you really don't want to use any of those, you can implement access control on a simple git server, but it's not so simple. The short (or rather, glib) answer is "hooks" - but a hook is just a script that runs when something happens - like in this case you'd use the prereceive hook, which runs when someone's trying to push and decides whether to accept the push. Now, how does your hook know who is pushing? (The commit metadata does not indicate who's pushing. What you need is authentication around the actual connection, and visibility of the authentication in your script so that the script may implement your authorization rules. That very quickly breaks down into "it depends on your environment".)
Since it's not really possible to exhaustively cover every scenario for doing this manually, hopefully either you'll find a pre-packaged solution you like, or you'll find the above to be enough to get you pointed in the right direction to do it manually.
Deployment
I currently work for a company that deploys through github. However, we have to log in to all 3 servers to update them manually with a shell script. When talking to the CTO he made it very clear that auto-deployment is like voodoo to him. Which is understandable. We have developers in 4 different countries working remotely. If someone where to accidentally push to the wrong branch we could experience downtime, and with our service we cannot be down for more than 10 minutes. And with all of our developers in different timezones, our CTO wouldn't know till the next morning and we'd have trouble meeting with the developers who had the issue because of vast time differences.
My Problem: Why I want auto-deploy
While working on my personal project I decided that it may be in my best interest to use auto-deployment, but still my project is mission critical and I'd like to mitigate downtime and human error as much as possible. The problem with manual deployment is that I simply cannot manually deploy on up to 20 servers via SSH in a reasonable amount of time. The problem perpetuates when I consider auto-scaling. I'd need to spin up a new server from an image and deploy to it.
My Stack
My service is developed on the Node.js Express framework. These environments are very rich in deployment and bootstraping utilities. My project uses npm's package.json to uglify my scripts on deploy, and also runs my service as a daemon using forever-monitor. I'm also considering grunt.js to further bootstrap my environments for both production and testing environments.
Deployment Methods
I've considered so far:
Auto-deploy with git, using webhooks
Deploying manually with git via shell
Deploying with npm via shell
Docker
I'm not well versed in technologies like Docker, but I'm interested and I'd definitely give points to whoever gave me a good description as to why I should or shouldn't use Docker, because I'm very interested in its use. Other methods are welcome.
My Problem: Why I fear auto-deploy
In a mission critical environment downtime can put your business on hold, and to make matters worse there's a fleet of end users hitting the refresh button. If someone pushes something that's not build passing to the production branch and that's auto-deployed, then I'm looking at a very messy situation.
I love the elegance of auto-deployment, but the risks make me skeptical. I'm very much in favor of making myself as productive as possible. So I'm looking for a way to deploy to many servers with ease, and in very efficient manner.
The Answer I'm Looking For
Explain to me how to mitigate the risks of auto-deployment, or explain to me an alternative which is better suited to my project. Feel free to ask for any missing details in the comments.
No simple answer here. I offer a set of slides published by Mike Brittain from Etsy, a company that practices continuous deployment:
http://www.slideshare.net/mikebrittain/mbrittain-continuous-deploymentalm3public
Selected highlights:
Deploy frequently and in small batches
Use config/feature flags to control system behaviour and "dark release" major features
Code review all changes to the production branch
Invest in monitoring and improve the feedback loop
Manage "services" separately to the "application" and be mindful of run-time version and backwardly compatible changes.
Hope this helps
My company is a custom development shop for a number of projects, some larger and some smaller. Currently we handle all of our client communication through email. So we email a design doc, they mark it up and send it back. Then we roll out a beta version of their product and they email us with any bugs, new features, etc. And so on....
As I am working on implementing a new bug tracking system (it looks like it will be Mantis right now), I got to wondering how we could best allow our customers an interface with our development process that would provide better tracking of feature requests and client submitted bugs as well as communicate our responses back to the client.
If anyone is aware of a a bug tracking system that does this exceptionally well I'd be interested to hear of that. Otherwise I'm just looking for some general guidelines or good business practices that have allowed your companies to interface effectively and efficiently with your clients.
UPDATE: My company uses a LAMPP stack and as we are a small shop with a limited budget we tend to stick to tools that are open-source and free.
Do most people either use Team Foundation Server to handle this or emails back and forth?
I think the key is to have the dedicated tracking system there for bugs/requests, and to establish a set process for communication. With that at minimum you will start getting consistent feedback. From there you can tweak it to get your specific needs.
As an aside, rather than just using e-mail for your communication, I strongly recommend going to smething like BaseCamp for a project management tool. I find that it helps greatly with keeping messages, documentation, and timelines communicated to the client.
If you are using Team Foundation Server, I recommend you to install TeamPlain Web Access. They allow you to expose a web interface to your TFS project. The only things left to do, is give rights to your client and a username and a password.
Otherwise, there is some paying tools like FogBugz. Of course, the principal is having to bug reporting tools directly linked to your Source Control so that the developers can easily fix bugs.
Although I know of no specific tools (at least no open source ones), I suggest that you setup a system which will cover your overall requirements gathering and implementation process. Requirements could be tracked in the system, which would also contain the design documents (which could be "checked out from" and "committed to" the system). This way, you would tackle the problem of having multiple revisions of design documents around. Addionally, the design documents and the requirements could be tracked easily. If this system were linked to your source code management system, you would additionally ease your development process/requirements tracking.
Another possibility is to use two products in concert, here's our current setup with a team of 12:
osTicket for incoming requests from clients
Allows for issues to be handled by support staff and bugs to be verified
Status can be checked with just an email address and ticket ID
Typically users don't submit detailed enough bug reports so is a good first step
redmine for development tickets
Ticket created by QA or a developer if issue is a real bug
Provides solid enough project and release management
Is a solid step up from trac and mantis (and provides migration tools)
I'm looking to re-organize the way we release our internal software. All of the code (PHP webapps, some Java apps and Perl scripts) is checked into Subversion repositories but there are no branches or tags, everything is checked into trunk (only around 1-3 devs per app). On the production linux servers, the software is just directly run from a working svn copy (actually most of the changes happen there as well).
Since we have a lot of small apps and release very often small changes to the running system, I'm looking for a very lean or transparent way to do some release engineering and to clean up this mess abit.
Are there any tools out there that may help me to do so in a heterogenous environment (language-wise) like that?
Or has anyone an idea how to do this in a proper way?
Otherwise I'd thought of writing some release (shell) scripts that automatically create subversion tags from trunk and then do a checkout of the corresponding tag to the production servers. But that sounds kinda hack'ish as well to me.
Thanks,
Haes.
Continuous Integration is definitely the way to go - any CI (even minimalist batch files) is better than none - but it'll only be as good as the policies you have in place. Since your files don't really end up as a 'binary' or 'distributable', marking a release might merely require only that you tag the repository, or even just stash the Subversion revision number somewhere. The important policy that you need is that any release can be reconstructed whenever you need it - so you can compare current and previous releases, or go back to an older release if something goes wrong. Don't worry about the 'overhead' of creating tags in svn - that's very efficient.
A release script that does the subversion tag sounds fine. A CI implementation (I'd recommend CruiseControl since it's ideal for heterogeneous work, although heterogeneity requires a bit more configuration overhead) is great, since you can automatically kick the process off on a subversion checkin, and run automated tests that determine whether it's good enough to tag or not.
I'd definitely not auto-deploy to a release server. A 'staging area' (call it 'nightly build', 'beta test', whatever) would be better. Let your users bang away on that before you decide it's good enough to roll out onto the production servers. And, as long as you've got the policy in place of being able to rollback to an earlier version, you've mitigated the possibility of a bad roll-out.
The auto-checkout onto production servers is the only 'hackish' part - an automated checkout, test, tag, beta deploy is slick enough. Rolling-out to production shouldn't have an easy button, though.
Use tags and branches; make it a part of the development cycle. When you update that "stable-1.0" branch, have tested the change(s) and tagged it "release-1.0.5", you simply do "svn switch" on the server to the new tag. Didn't work, despite having tested it? Switch back, and figure out what's wrong.
But beware, branching in subversion can be a pain, at least pre version 1.5. If you or your developers are not experienced with branches, expect a bit of hassle and/or mistakes in the beginning. But as long as you've committed no code should be lost (at worst simply difficult to merge).
Your developers really should learn how to use branching; it can be very useful for a variety of purposes (not just for release engineering).
Do not automatically switch over code on your production servers; somebody might accidentally hit the wrong button. Production updates should always be done with care. Scripts for adding new tags is, imho, unnecessary due to the simplicity of it, but your mileage may vary.
One last thing, don't allow anyone to have changes on your production server. It might cause conflicts, and those tend to take time to resolve. Not to mention, it destroys your ability to reproduce a given release on different workstations (works fine here! why not on the server? hmm).
Some Continuous Integration Servers do this sort of thing, Hudson, for example, has subversion integration. It can tag, run test, and deploy for you.
i'd use Hudson. in addition to fetching from and tagging in svn (ref sblundy), it can be useful in release management with the proper plugins. f.ex., you could try a plugin to "promote" the builds you deploy to production, and keep a list of both the promoted builds themselves and a change/commit log for the various versions.