I have been recently working on a project that involves the integration of Kafka, Spark and Cassandra. One of the key things I noticed when trying to get the whole thing setup is that there are a lot of version conflicts that needs to be very carefully matched in order to get these technologies to work together.
In addition, it was important to take note of the Scala version used with Spark when writing your own Spark-Jobs.
A slight change in the version of one of the above technologies breaks the complete flow and requires a proper redo of matching them together.
The task was not very straight forward (at least for me and I guess it's the same for all) and I am wondering how do companies which have these technologies working in sync actually manage this?
As I see it, it is an important problem with new releases and bug fixes being rolled out, to keep these tools working together without a break.
Can someone who has experience with regards to this enlighten me as to how companies actually manage/maintain these conflicts?
Or is it an overstatement to say it's an actual problem?
Thanks in advance
Murphy kicked my a$$ about an hour ago.
Context:
I recently joined a new employer and the product was quite outdated in terms of dependencies, Angular 1.2.x, Angular-UI 0.12.0, etc...
This is the first employer I've worked at that does daily builds to prod etc. (previosuly I've only worked in what can be called large corporate, with much slower turn around) Part of my initial task was to upgrade dependencies where I can. Thus earlier this morning we had a watercooler talk with some of the devs about why all of our bower dependencies are hardcoded to specific versions.
The 2 schools of thought are:
Hardcoding versions obviously gives 100% security as versions can't dynamically jump, but has the drawback that if someone doesnt actively update we'll fall behind again.
I'm of the opinion that semantic versioning gives us some form of security (coupled with having multiple staging environments), and that it should be good enough to, say, have Angular set to say ^1.5.9.
Quoted from the Semantic Version Docs:
Minor version Y (x.Y.z | x > 0) MUST be incremented if new, backwards
compatible functionality is introduced to the public API. It MUST be
incremented if any public API functionality is marked as deprecated.
It MAY be incremented if substantial new functionality or improvements
are introduced within the private code. It MAY include patch level
changes. Patch version MUST be reset to 0 when minor version is
incremented.
Problem:
Earlier this morning we deployed to staging, and everything seemed good to go, then we deployed to production an hour or so ago and ... BOOM
The issue was the AngularJs change from 1.5.9, to 1.6.0. I've seen in the migration docs (migrate 1.5 -> 1.6) that this has been noted:
You may also notice that this release comes with a longer-than-usual
list of breaking changes. Don't let this dishearten you though, since
most of them are pretty minor - often not expected to affect real
applications. These breaking changes were necessary in order to:
Question:
Where is my disconnect? ...or is the semantic version docs just a false sense of security I've had all along?
How do people out there in the handle these situations? Do people make use of auto dependency upgrading in any real world solutions (excuse me if this is super obvious to some), as to me, the fact that the build passed staging, and broke in production is actually more concerning.
(The reason I'm asking is because the fear of small incremental updates are now back and stronger than ever, and I'm not sure if I agree with the sentiment of it all...)
Seems pretty simple, if they make breaking changes, they should have bumped it up to 2.0.0. They are not doing semantic versioning. Not all projects using X.Y.Z. style versions are doing semantic versioning.
Try to catch how this went "boom" in an automated way in your testing and staging environments. Can't fear moving forward, it has to be done sometime, and I'd rather move step-by-step more frequently, than to suddendly move up many versions as would be done with an entirely manual process.
I have a website I'm developing. I try to use SEMVER with any changes I make. Sometimes I'm not sure when I need to bump up the minor version. For example,
I have my website/app at v0.9.0. I have this deployed on a production site. One day, I decide to add some new functionality to the website. I add my new functionality, test it and deploy it and it works great. Since this is not a bug fix, does that mean I need to go from v0.9.0 to v1.0.0? So essentially, if I'm at vX.0.0, it only takes 10 new features to be independently added (no matter the size) in order to go to version X+1 ?
It seems like there should be some way to incrementally increase the minor version. I see a lot of people do v0.10.0 as the next version.
Is that the recommended way to do it?
No Jake, it does not mean you need to go to version 1.0.0. The next minor version after version 0.9.0 would be 0.10.0.
See https://docs.npmjs.com/getting-started/semantic-versioning for a nice intro.
I used node.js for a development project a few years ago, and this app is somewhat "mothballed" for the time being — it needs to stay online, it needs to stay secure, but it shouldn't need much attention. It is currently running on node.js v0.10.32, but I would now like invest in a "final" migration to a Long Term Support (LTS) release so it will be easier to maintain for the foreseeable future.
At first glance, the node.js homepage makes it look like v4.4.5 is obviously the only available LTS release:
However if I click that LTS schedule link, it tells a different story. As far as I can tell, version 6 of node.js is also slated to be an LTS release, with that support ending a full year later than version 4 will be.
Given that:
v6.2.0 is a versioned release
v6 is purported to receive LTS maintenance until 2019-04-01
theoretically no changes in v6.x should break backwards compatibility
Why would I bother upgrading to v4 instead of v6? Seems like v4 buys me one less year of security patches, but no additional compatibility guarantees?
With gratitude to jasnell and TheAlphaNerd who patiently answered my acerbic questions over on GitHub, here's my understanding of how node.js "long term support" releases are handled:
All releases based on an even-numbered major branch are what other projects might call a "long term support" version. They are promised at least 30 months of support from the first "cut" made available (e.g. a packaged v6.0.0 releases).
However, the node.js maintainers see "LTS" as more of a release phase than a version type. While they intend for minor/patch releases made when a major branch is in its active improvement (see "CURRENT" below) phase to be stable and backwards compatible, in the real world they might make mistakes.
So they divide development into three distinct phases:
CURRENT: new features (and bug fixes and security patches)
ACTIVE LTS: bug fixes (and security patches)
MAINTENANCE: only security patches
Odd-numbered major versions get only the first phase before being left behind. Even-numbered major versions — the ones we're mostly concerned about here — go through all three phases.
The CURRENT phase starts with the first public release, and starts the clock ticking on the 30-month support window. They may add significant new features, which should in theory be backwards-compatible but in practice may turn up some issues (new bug gets added, change made to poorly-defined behavior, they fix an old bug you had worked around poorly, etc.)
Then at some point the team decides to move the main development effort to a short-lived odd-numbered major branch (presumably when they need to intentionally break backwards compatibility). At that point the even-numbered branch moves to ACTIVE LTS and they get much more careful with the changes they make: primarily just bug fixes. So if you really want stability, this is the time to get "on board" with a particular release.
Eventually it moves even further to the MAINTENANCE phase, where the code is touched only in response to the most critical bugs (think: security patches). But by then there's probably a new release in LTS "phase" already.
So the choice on the homepage right now is between two even-numbered branches, "v4.4.5 LTS" and "v6.2.0 current". If the newer branch were odd-numbered then it would not be a good candidate for a production deploy where long-term support is desired.
My actual options are a bit more complex even:
I could simply stay on v0.10 which will get critical fixes until October. Or bump up to v0.12 to get those until the end of the year. But neither one of those gains me much, and so I'll rule them out.
I can deploy v4.4.5 which will is still getting general bug fixes right now, and will get security fixes for quite a while yet. This should give me the most stable install. The support cycle is getting to be halfway over already, though — and when it runs out I'll have missed this opportunity now to catch up with some of the major changes that have already happened in v5 and v6.
I am leaning towards deploying v6.2.0, assuming all my dependencies support it right now. This not only buys me a year longer term of "lifecyle remaining", but also gets me fully caught up with any breaking changes that were introduced between v0.10 and now. (It also gets me access to any useful new features — but in this case I don't have a chance to take advantage of it.) The risk I take is that when I update to any hypothetical v6.2.1 or v6.3.0 or beyond that comes along, it might accidentally yield some bad surprises.
In my case I'd rather deal now with the major intentional changes that v5 and v6 have already introduced, and then hopefully be all set (or at least only minor pain) from now all the way until April 2019.
I have been looking into MeeGo, maemo, Android architecture.
They all have Linux Kernel, build some libraries on it, then build middle layer libraries [e.g telephony, media etc...].
Suppose i wana build my own system, say Linux Kernel, with some binariers like glibc, Dbus,.... UI toolkit like GTK+ and its binaries.
I want to compile every project from source to customize my own linux system for desktop, netbook and handheld devices. [starting from netbook first :)]
How can i build my own customize system from kernel to UI.
I apologize in advance for a very long winded answer to what you thought would be a very simple question. Unfortunately, piecing together an entire operating system from many different bits in a coherent and unified manner is not exactly a trivial task. I'm currently working on my own Xen based distribution, I'll share my experience thus far (beyond Linux From Scratch):
1 - Decide on a scope and stick to it
If you have any hope of actually completing this project, you need write an explanation of what your new OS will be and do once its completed in a single paragraph. Print that out and tape it to your wall, directly in front of you. Read it, chant it, practice saying it backwards and whatever else may help you to keep it directly in front of any urge to succumb to feature creep.
2 - Decide on a package manager
This may be the single most important decision that you will make. You need to decide how you will maintain your operating system in regards to updates and new releases, even if you are the only subscriber. Anyone, including you who uses the new OS will surely find a need to install something that was not included in the base distribution. Even if you are pushing out an OS to power a kiosk, its critical for all deployments to keep themselves up to date in a sane and consistent manner.
I ended up going with apt-rpm because it offered the flexibility of the popular .rpm package format while leveraging apt's known sanity when it comes to dependencies. You may prefer using yum, apt with .deb packages, slackware style .tgz packages or your own format.
Decide on this quickly, because its going to dictate how you structure your build. Keep track of dependencies in each component so that its easy to roll packages later.
3 - Re-read your scope then configure your kernel
Avoid the kitchen sink syndrome when making a kernel. Look at what you want to accomplish and then decide what the kernel has to support. You will probably want full gadget support, compatibility with file systems from other popular operating systems, security hooks appropriate for people who do a lot of browsing, etc. You don't need to support crazy RAID configurations, advanced netfilter targets and minixfs, but wifi better work. You don't need 10GBE or infiniband support. Go through the kernel configuration carefully. If you can't justify including a module by its potential use, don't check it.
Avoid pulling in out of tree patches unless you absolutely need them. From time to time, people come up with new scheduling algorithms, experimental file systems, etc. It is very, very difficult to maintain a kernel that consumes from anything else but mainline.
There are exceptions, of course. If going out of tree is the only way to meet one of your goals stated in your scope. Just remain conscious of how much additional work you'll be making for yourself in the future.
4 - Re-read your scope then select your base userland
At the very minimum, you'll need a shell, the core utilities and an editor that works without an window manager. Paying attention to dependencies will tell you that you also need a C library and whatever else is needed to make the base commands work. As Eli answered, Linux From Scratch is a good resource to check. I also strongly suggest looking at the LSB (Linux standard base), this is a specification that lists common packages and components that are 'expected' to be included with any distribution. Don't follow the LSB as a standard, compare its suggestions against your scope. If the purpose of your OS does not necessitate inclusion of something and nothing you install will depend on it, don't include it.
5 - Re-read your scope and decide on a window system
Again, referring to the everything including the kitchen sink syndrome, try and resist the urge to just slap a stock install of KDE or GNOME on top of your base OS and call it done. Another common pitfall is to install a full blown version of either and work backwards by removing things that aren't needed. For the sake of sane dependencies, its really better to work on this from bottom up rather than top down.
Decide quickly on the UI toolkit that your distribution is going to favor and get it (with supporting libraries) in place. Define consistency in UIs quickly and stick to it. Nothing is more annoying than having 10 windows open that behave completely differently as far as controls go. When I see this, I diagnose the OS with multiple personality disorder and want to medicate its developer. There was just an uproar regarding Ubuntu moving window controls around, and they were doing it consistently .. the inconsistency was the behavior changing between versions. People get very upset if they can't immediately find a button or have to increase their mouse mileage.
6 - Re-read your scope and pick your applications
Avoid kitchen sink syndrome here as well. Choose your applications not only based on your scope and their popularity, but how easy they will be for you to maintain. Its very likely that you will be applying your own patches to them (even simple ones like messengers updating a blinking light on the toolbar).
Its important to keep every architecture that you want to support in mind as you select what you want to include. For instance, if Valgrind is your best friend, be aware that you won't be able to use it to debug issues on certain ARM platforms.
Pretend you are a company and will be an employee there. Does your company pass the Joel test? Consider a continuous integration system like Hudson, as well. It will save you lots of hair pulling as you progress.
As you begin unifying all of these components, you'll naturally be establishing your own SDK. Document it as you go, avoid breaking it on a whim (refer to your scope, always). Its perfectly acceptable to just let linux be linux, which turns your SDK more into formal guidelines than anything else.
In my case, I'm rather fortunate to be working on something that is designed strictly as a server OS. I don't have to deal with desktop caveats and I don't envy anyone who does.
7 - Additional suggestions
These are in random order, but noting them might save you some time:
Maintain patch sets to every line of upstream code that you modify, in numbered sequence. An example might be 00-make-bash-clairvoyant.patch, this allows you to maintain patches instead of entire forked repositories of upstream code. You'll thank yourself for this later.
If a component has a testing suite, make sure you add tests for anything that you introduce. Its easy to just say "great, it works!" and leave it at that, keep in mind that you'll likely be adding even more later, which may break what you added previously.
Use whatever version control system is in use by the authors when pulling in upstream code. This makes merging of new code much, much simpler and shaves hours off of re-basing your patches.
Even if you think upstream authors won't be interested in your changes, at least alert them to the fact that they exist. Coordination is essential, even if you simply learn that a feature you just put in is already in planning and will be implemented differently in the future.
You may be convinced that you will be the only person to ever use your OS. Design it as though millions will use it, you never know. This kind of thinking helps avoid kludges.
Don't pull upstream alpha code, no matter what the temptation may be. Red Hat tried that, it did not work out well. Stick to stable releases unless you are pulling in bug fixes. Major bug fixes usually result in upstream releases, so make sure you watch and coordinate.
Remember that it's supposed to be fun.
Finally, realize that rolling an entire from-scratch distribution is exponentially more complex than forking an existing distribution and simply adding whatever you feel that it lacks. You need to reward yourself often by booting your OS and actually using it productively. If you get too frustrated, consistently confused or find yourself putting off work on it, consider making a lightweight fork of Debian or Ubuntu. You can then go back and duplicate it entirely from scratch. Its no different than prototyping an application in a simpler / rapid language first before writing it for real in something more difficult. If you want to go this route (first), gNewSense offers utilities to fork your own OS directly from Ubuntu. Note, by default, their utilities will strip any non free bits (including binary kernel blobs) from the resulting distro.
I strongly suggest going the completely from scratch route (first) because the experience that you will gain is far greater than making yet another fork. However, its also important that you actually complete your project. Best is subjective, do what works for you.
Good luck on your project, see you on distrowatch.
Check out Linux From Scratch:
Linux From Scratch (LFS) is a project
that provides you with step-by-step
instructions for building your own
customized Linux system entirely from
source.
Use Gentoo Linux. It is a compile from source distribution, very customizable. I like it a lot.