There are many websites providing cloud coding sush as Cloud9, repl.it. They must use server virtualisation technologies. For example, Clould9's workspaces are powered by Docker Ubuntu containers. Every workspace is a fully self-contained VM (see details).
I would like to know if there are other technologies to make sandboxed environment. For example, RunKit seems to have a light solution:
It runs a completely standard copy of Node.js on a virtual server
created just for you. Every one of npm's 300,000+ packages are
pre-installed, so try it out
Does anyone know how RunKit acheives this?
You can see more in "Tonic is now RunKit - A Part of Stripe! " (see discussion)
we attacked the problem of time traveling debugging not at the application level, but directly on the OS by using the bleeding edge virtualization tools of CRIU on top of Docker.
The details are in "Time Traveling in Node.js Notebooks"
we were able to take a different approach thanks to an ambitious open source project called CRIU (which stands for checkpoint and restore in user space).
The name says it all. CRIU aims to give you the same checkpointing capability for a process tree that virtual machines give you for an entire computer.
This is no small task: CRIU incorporates a lot of lessons learned from earlier attempts at similar functionality, and years of discussion and work with the Linux kernel team. The most common use case of CRIU is to allow migrating containers from one computer to another
The next step was to get CRIU working well with Docker
Part of that setup is being opened-source, as mentioned in this HackerNews feed.
It uses linux containers, currently powered by Docker.
Related
I read over internet "containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server".
I also read that linux containers cannot run on windows.
The benifits of containers stated "Containers run as an isolated process in userspace on the host operating system."
I don't understand if the containers are not platform independent what we are actually achieving out it?
1) Anyhow all the applications on a linux box should run as an isolated process in their userspace.
2) If containers only contain app code + runtimes + tools + libraries. They can be shipped together. What conatiners are getting here?
Posting the comment as answer::
If containers only contain app code + runtimes + tools + libraries.
They can be shipped together. What conatiners are getting here?
Suppose there is an enterprise with thousands of employees and all of them work on Visual Studio C++. Now, the administrator can create a container with the installed (only C++ components) and configured VS, and deploy that container to all employees. The employees can instantly start working without bothering about installation and configuration of the application. Again, if the employee somehow corrupts the application, they only need to download the container again and they are good to go.
Sandboxing
Security
Maintenance
Mobility
Backup
Many more to go.
Are container platform independent?
IMHO, I don't think so, as they rely on the system calls. Though, I am open to other concepts if anybody knows better on this topic.
Even only considering one platform, containers have their advantages; just not perhaps the ones you need right now. :-) Containers help in administration/maintenance of complex IT systems. With containers you can easily isolate applications, their configuration, and their users, to achieve:
Better security (if someone breaks in, damage is usually limited to one container)
Better safety (if something breaks, or e.g. you make an error, only applications in a given container will be victim to this)
Easier management (containers can be started/stopped separately, can be transferred to another hosts (granted: host with the same OS; in case of Linux containers the host must also be Linux))
Easier testing (you can create and dispose-off containers at will, anytime)
Lighter backup (you can backup just the container; not the whole host)
Some form of increasing availaibility (by proper pre-configuration and automatic switching a container over to another host you can be up and running quicker in case of the primary host failure)
...just to name the first advantages coming to mind.
Can anyone understand and explain the fundamental differences of Docker and Rocket?
I don't seem to get it.
Maybe it's just too new of a direction.
Hope someone can explain the fundamental pros and cons of Docker vs Rocket.
Thanks
https://coreos.com/blog/rocket/
https://github.com/coreos/rocket
Rocket is an "early stage" container manager, just like was Docker a few month ago. The new "container runtime" was lanched by CoreOS few days ago but the intresting thing is WHY?
Essentially CoreOS, as well as others Open Source developers and startups, says that Docker broke the idea of a “standard container”, simplicity and composability, they where started from, proved by the fact that they removed original shipping containers 'manifesto'.
By my side, I already saw many signals of that, starting from the "legal empowerment" of Docker brand, to the drop off LXC containers, the Linux open source tecnology at the base of Docker, which permitted their own climb since the very beginning. I shoot my first hot reaction here, in responce to the "counterattack" of Solomon Hykes (founder & creator of the Docker project) to CoreOS announcement.
That why I found quite curious (hilariously) the recent declaration of Solomon Hykes : "We're standing on the shoulders of giants"
They also have raised doubts about security and composability perspective:
From a security and composability perspective, the Docker process
model - where everything runs through a central daemon - is
fundamentally flawed. To “fix” Docker would essentially mean a rewrite
of the project, while inheriting all the baggage of the existing
implementation.
So what is Rocket at the end ?
Rocket is an alternative to the Docker runtime, designed for server
environments with the most rigorous security and production
requirements. Rocket is oriented around the App Container
specification, a new set of simple and open specifications for a
portable container format.
What's the difference with Docker ?
The promise foundation of freedom and industrial open standards like in DNS, HTTP, IMAP, SMTP, TCP/IP, ISO/OSI stack ... Internet? Or more a concrete security and composability perspective.
Have a look at the eclectic speaker and amazing developer Kelsey Hightower Rocket Tutorial & Demo.
ongoing UPDATE (SPECs - OPENSOURCE - VISION):
[MUST READ] Amazing nitty-gritty details about the matter of Docker flawed
universal toolkit for emulating Heroku, regardless of stack or container engine
Rocket & App Container Spec Overview
Not clear now, they just forked ;)
But Rocket want stick to pure unix philosophy
Unix philosophy: tools should be independently useful
Which implies that Docker is willing to pay less attention to this topic. To my opinion it was not the case till that moment, but yes Docker announced orchestration tools in the future...
CoreOS is building their own orchestration stack so they don't really need one of Docker.
Summing up: for now use Docker. And ask this question again in a year.
Rocket - uses systemd-nspawn ( they can also do exec of kvm )
There is also an intention to make rocket a generic framework to manage any virtualized environment that is shipped with coreOS
Docker - uses lxc ( inturn does clone ( namespace ) & pivot_root ) It starts with a base image that is read only and adds more images to it. It uses union mount to add more read-only filesystems on to the base root fs. It also implements copy on write. It starts with an empty read-write layer and if you write something to a file the file is first copied to the read-write layer. checkout aufs.
The net effect is very similar ( if both are configured to use containers) but the way apps are packaged and deployed are different.
rocket claims to provide better flexibility by providing app spec.
Docker provides easy/quick portable packaging and deployment.
Now (2020) Rocket is officially dead: https://github.com/rkt/rkt/issues/4024
After acquisition by Red Hot new owner concentrates efforts on https://podman.io/
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
There are a bunch of managed cloud based hosting services for nodejs out there which seem relatively new and some still in Beta.
Yet another path to host a nodejs app is setting up a stack on a VPS like Linode.
I'm wondering what's the basic difference here between these two kinds of deployment.
Which factors should one consider in choosing one over another?
Which one is more suitable for production considering how young these services are.
To be clear I'm not asking on choosing a provider but to decide whether to host on a managed nodejs specific hosting or on an old fashioned self setup VPS.
Using one of the services is for the most part hands off - you write your code and let them worry about managing the box, keep your process up, creating the publishing channel, patching the OS, etc...
In contrast having your own VM gives you more control but with more up front and ongoing time investment.
Another consideration is some hosters and cloud providers offer proprietary or distinct variations on technologies. They have reasons for them and they offer value but it does mean that if you want to switch cloud providers, it might mean you have to rewrite code, deployment scripts etc... On the other hand using VMs with standard OS as the baseline is pretty generic. If you automate/script/document the configuration of your VMs and your code stays generic, then your options stay open. If you do take a dependency on a proprietary cloud technology then it would be good to abstract it away behind an interface so it's a decoupled component and not sprinkled throughout your code.
I've done both. I did the VM path recently mostly because I wanted the learning experience. I had to:
get the VM from the cloud provider
I had to update and patch the OS
I had to install and configure git as a publishing channel
I had to write some scripts and use things like forever to keep it running
I had to configure the reverse http-proxy to get it to run multiple sites.
I had to configure DNS with the cloud provider, open ports for git etc...
The list goes on. In the end, it cost me more up front time not coding but I learned about a lot more things. If those are important to you, then give it a shot. If you want to focus on writing your code, then a node hosting provider may be for you.
At the end of it, I had also had more options - I wanted to add a second site. I added an entry to my reverse proxy, append my script to start up another app with forever, voila, another site. More control. After that, I wanted to try out MongoDB - simple - installed it.
Cost wise they're about the same but if you start hosting multiple sites with many other packages like databases etc..., then the VM can start getting cheaper.
Nodejitsu open sourced their tools which also makes it easier if you do your own.
If you do it yourself, here's some links that may help you:
Keeping the server up:
https://github.com/nodejitsu/forever/
http://blog.nodejitsu.com/keep-a-nodejs-server-up-with-forever
https://github.com/bryanmacfarlane/svchost
Upstart and Monit
generic auto start and restart through monitoring
http://howtonode.org/deploying-node-upstart-monit
Cluster Node
Runs one process per core
http://nodejs.org/docs/latest/api/cluster.html
Reverse Proxy
https://github.com/nodejitsu/node-http-proxy
https://github.com/nodejitsu/node-http-proxy/issues/232
http://blog.nodejitsu.com/http-proxy-middlewares
https://github.com/nodejitsu/node-http-proxy/issues/168#issuecomment-3289492
http://blog.argteam.com/coding/hardening-node-js-for-production-part-2-using-nginx-to-avoid-node-js-load/
Script the install
https://github.com/bryanmacfarlane/svcinstall
Exit Shell Script Based on Process Exit Code
Publish Site
Using git to publish to a website
IMHO the biggest drawback of setting up your own stack is that you need to manage things like making Node.js run forever, start it as a daemon, bring it behind a reverse-proxy such as Nginx, and so on ... the great thing about Node.js - making firing up a web server a one-liner - is one of its biggest drawbacks when it comes to production-ready systems.
Plus, you've got all the issues of managing and updating and securing your server yourself.
This is so much easier with the hosters: Usually it's a git push and that's it. Scaling? Easy. Replication? Easy. ...? Easy. All within a few clicks.
The drawback with the hosters is that you can not adjust the environment. Okay, you can probably choose which version of Node.js and / or npm to run, but that's it. You have no control over what 3rd party software is installed. You've got no control over the OS. You've got no control over where the servers are located. And so on ...
Of course, some hosters allow you access to some of these things, but there is rarely a hoster that supports all.
So, basically the question regarding Node.js is the same as with each other technology: It's a pro vs con of individualism, pricing, scalabilty, reliability, knowledge, ...
I personally chose to go with a hoster: The time and effort I save easily outperform the disadvantages. Mind you: For me, personally.
This question needs to be answered individually.
Using Docker is another way to simplify the setup on single Linux VPS. With Docker both development and production setups are faster, more robust, and more secure.
The setup is faster and more robust because you will be deploying ready Node.js image at once, without running any installation scripts. And it would be more secure because internal dependencies, such as database, can be hidden from outside world completely and accessible only from Docker internal network. On top of it, Docker significantly simplifies the upgrade process for underlying OS and Node.js runtime.
There are two ways to setup Node.js Docker environment. The first one – follow the instruction published here how to dockerize your application and deploy it with Docker, alongside with databases when needed. The guide gives the instructions for the development setup, the production setup will be similar.
Another way would be deploying official Node.js docker image and mounting application code as a volume or a folder to Node.js image. That would allow to update Node.js image going forward without re-building and re-deploying the application. Such approach solves long-standing problem with security patching of Docker images.
To help out with the setup of Docker on single machine - you can use Abberit Admin Panel. It will set up Node.js environment for you with a click of a button, including databases if you need them. The tool is free, and you can turn it off after you have completed initial setup. On the other hand, if later you decide to reduce maintenance tax of the production - you can migrate into managed service without any changes in the app.
Disclaimer: I am one of the founders of Abberit.
For the first time, I am writing a web service that will call upon external programs to process requests in batch. The front-end will accept file uploads and then place them in a queue. The workers on the backend will take that file, run it through ffmpeg and the rest of my pipeline, and send an email when the process is complete.
I have my backend process working on my computer (Ubuntu 10.04). The question is: should I try to re-create that pipeline using binaries that I've compiled from scratch? Or is it okay to use apt when configuring in The Real World?
Not all hosting services uses Ubuntu, and not all give me root access. (I haven't chosen a host yet.) However, they will let me upload binaries to execute, and many give me shell access with gcc.
Usually this would be a no-brainier and I'd compile it all from scratch. But doing so - not to mention trying to figure out how to create a platform-independent .tar.gz binary - will be quite a task which ultimately doesn't really help me ship my product.
Do you have any thoughts on the best way to set up my stack so that I'm not tied to a specific hosting provider? Should I try creating my own .deb, which contains Ubuntu's version of ffmpeg (and other tools) with the configurations I need?
Short of a setup where I manage my own servers/VMs (which may very well be what I have to do), how might I accomplish this?
The question is: should I try to re-create that pipeline using binaries that I've compiled from scratch? Or is it okay to use apt when configuring in The Real World?
It is in reverse: it is not okay to deploy unpackaged in The Real World IMHO
and not all give me root access
How would you be deploying a .deb without root access. Chroot jails?
But doing so - not to mention trying to figure out how to create a platform-independent .tar.gz binary - will be quite a task which ultimately doesn't really help me ship my product.
+1 You answer you own question. Don't meddle unless you have to.
Do you have any thoughts on the best way to set up my stack so that I'm not tied to a specific hosting provider?
Only depend on wellpackaged standard libs (such as ffmpeg). Otherwise include them in your own deployment. This problem isn't too hard too solve for 10s of thousand Linux applications over decades now, so it would probably be feasible for you too.
Out of the box:
Look at rightscale and other cloud providers/agents that have specialized images/tool chains especially for video encoding.
A 'regular' VPS provider (with Xen or Virtuozzo) will not normally be happy with these kinds of workload, but EC2, Rackspace and their lot will be absolutely fine with that.
In general, I wouldn't believe that a cloud infrastructure provider that doesn't grant root access will allow for computationally intensive workloads. $0.02
Approach (A)
From my experience I saw that for a small team there's a dedicated server with all development tools (e.g. compiler, debugger, editor etc.) installed on it. Testing is done on dedicated per developer machine.
Approach (B)
On my new place there's team utilizing a different approach. Each developer has a dedicated PC which is used both as development and testing server. For testing an in-house platform is installed on the PC to run application over it. The platform executes several modules on kernel space and several processes on user space.
Problem
Now there are additional 2 small teams (~ 6 developers at all) joining to work on the exactly same OS and development environment. The teams don't use the mentioned platform and can execute application over plain Linux, so no need in dedicated machine for testing. We'd like to adopt approach (A) for all 3 teams, but the server must be stable and installing on it in-house platform, described above, is highly not desirable.
What would you advise?
What is practice for development environmentin your place - one server per team(s) or dedicated PC/server per developer?
Thanks
Dima
We've started developing on VMs that run on the individual developers' computers, with a common subversion repository.
Benefits:
Developers work on multiple projects simultaneously; one VM per project.
It's easy to create a snapshot (or simply to copy the VM) at any time, particularly before those "what happens if I try something clever" moments. A few clicks will restore the VM to its previous (working) state. For you, this means you needn't worry about kernel-space bugs "blowing up" a machine.
Similarly, it's trivial to duplicate one developer's environment so, for example, a temporary consultant can help troubleshoot. Best-practices warning: It's tempting to simply copy the VM each time you need a new development machine. Be sure you can reproduce the environment from your repository!
It doesn't really matter where the VMs run, so you can host them either locally or on a common server; the developers can still either collaborate or work independently.
Good luck — and enjoy the luxury of 6 additional developers!