We have a business application that basically runs on an os-independent stack (tomcat+java+mysql) but we have always run it redhat or centos.
There is a customer that is insisting to run it on opensolaris for his own reasons (an expensive everything-is-included support agreement with Sun).
How painful can such a migration be? We have a lot of configuration file and support scripts such as:
apache
apache/tomcat connector
email interaction with postfix
customized service start/stop
a couple of cron jobs (backup, monitoring)
different users and permissions (java, mysql, email, backup...)
Our build process outputs a .tar.gz file with our business code + some shell scripts that edit all the os-configuration files.
Any previous experience on this.
The biggest issues will be with the non-POSIX (non-standard) options you've used to the GNU tools provided on Linux that are not in the Solaris standard commands. You might decide that porting the relevant tools from the GNU set is simpler than modifying your system. If you've laced the code with absolute pathnames for commands (/usr/bin/ls) but you decide to use the GNU versions instead, you've got to find a way of fixing those. I'd be extremely cautious about replacing the OpenSolaris versions with the GNU versions; you don't know when you would break something that the system relies on. So, you would put the GNU commands in a separate directory - probably not /usr/local because that is for the machine owners to populate, not you as an application-monger - and arrange for that to be used in place of the system commands. (Note: on Solaris, /bin is a symlink to /usr/bin; I assume the same is true of OpenSolaris.) AFAIK, Postfix is not standard on OpenSolaris, so you'd have to ensure you get that installed, too.
All of this is doable - there's nothing insuperable. But a lot depends on your code base.
We run both, though we don't use OpenSolaris as a web servers.
The good:
OpenSolaris comes with the gnu tools, so, get your path right and that's ok.
Most things just build and run just fine.
The not so good:
Make sure that you've installed and are using bash. Otherwise all those bashisms that you are using that you didn't think you were using will bite you.
Make sure that you're not using hard coded paths to /usr/bin or /bin. These tools are not the GNU ones and therefore have different options. Use /usr/gnu as mentioned above.
You don't have the huge number of packages that you can install straight off as you do with yum or apt. Yes, you have a package manager, it's just not quite so well populated.
As a result you probably will be installing packages by hand. They should install, it's just a bit more work for your system admins.
Are you sure that OpenSolaris runs well on your hardware? It's worth a check. You might find that some of the hardware drivers aren't as well tested.
Otherwise we find OpenSolaris to be nice. It has a lot of good ideas.
Have you looked at Nexenta - http://www.nexenta.org/os It's the OpenSolaris kernel with a Ubuntu userland.
OpenSolaris includes all the GNU utilities already, just point your scripts at /usr/gnu/bin
Installing Postfix shouldn't present any problems, and Apache/MySQL are present in a base OpenSolaris install (in truth, the Cool Web Stack stuff makes it about as easy to administer as WAMP/Instant Rails). Beyond which, SMF manifests (SMF is a replacement for rc scripts sort of like OSX's launchd, though you can still use regular init scripts) may make your life easier, since specifying dependencies and run order is somewhat nicer (it'll recursively start/stop all dependent services also).
Tomcat certainly works, though everybody I know on OpenSolaris uses GlassFish. YMMV, but deploying a .war is pretty much the same everywhere.
It may not be a bad first step to deploy into a LX branded zone (think FreeBSD jails or Linux vServer for a comparison), as the LX branded zones can run Linux binaries, and are explicitly CentOS/RHEL based.
Other than that, OpenSolaris is a Xen dom0 since b77 or something, and putting CentOS/RHEL into a domU is dead simple, if that's an option.
You also get all the Solaris goodies along with it (DTrace, ZFS, network virtualization [via CrossBow], etc). Who knows? You may even like it! Java is Java, so that shouldn't pose any issues.
you'll probably have to rewrite a big part of your scripts (user creations, service launch) as it is probably different in CentOS and OpenSolaris.
as previously written, ask your customer to install the GNU tools so you'll have less work to rewrite your scripts.
os configuration files may also not be in the same format, you'll need to check.
your tar.gz file should be extractable without troubles, but again you will have less surprises if you use GNU tools. some unix OS have tar with some limitations
Any previous experience on this.
(maybe a little offtopic)
we package and distribute our java/tomcat/postgresql/unix application with all binaries referenced in our scripts. this implies to have 1 build system for each OS we support, this implie we support our application but also external binaries, but in the end we do not have bad surprises # customers.
we also ask them to do all root operations (user creation, directory creation, sendmail config, system tuning) before we install the application.
we have written shutdown / startup scripts for all supported OS, and their installation is the only thing we do in root on the customer machine.
Beside the fact that you're a troll, somebody just said above that (Open)Solaris has:
- ZFS
- DTrace
We can understand that you are afraid of not losing your RHCE job, but you just proved me once again that my decision as an employer to ignore all the certifications when interviewing people was a good one. It seems that a large percentage of such people (especially in the Microsoft world) are not so... open-minded, to put it nicely.
Regards,
Alex
Related
My customers are novice Linux users, and so am i.
When I gave them my App packaged with ansible, they saw ansible problems, when i gave them manual steps, they also screwed that up, now i have 3 last options, either a perl/bash script or a snappy/deb/rpm package or Linux containers, can anyone share their experience on the safest way to see less problems when installing my app (Written in C)?
This depends on the nature of your application. Debs, rpms etc. are all fine but depend on which distro you're using.
If it's C application, it might make sense to make it a static binary. That way, you'll have to download a single file and just click on it to make it run. It will be big but it should work fine regardless of what else is there. Otherwise, you'll have to worry about dependencies etc.
As it was commented before it depends what you did to deploy the product.
In general, if you have dependencies (previous packages that you assume were already installed) or your installation is complex - use rpm or deb.
However if you target multi-platform bare in mind you will have at least two releases (one rpm and one deb...)
If configuration or installation is easier you can just give them an install script.
If your application requires a specific environment with specific configuration/packages I'd consider containers although I never done that personally before.
We work on scientific computing and regularly submit calculations to different computing clusters. For that we connect using linux shell and submitting jobs through SGE, Slurm, etc (it depends on the cluster). Our codes are composed of python and bash scripts and several binaries. Some of them depend on external libraries such as matplotlib. When we start to use a new cluster, it is a nightmare since we need to tell the admins all the libraries we need, and sometimes they can not install all of them, or they only have old versions that can not be upgraded. So we wonder what could we do here. I was wondering if we could somehow "pack" all libraries we need along with our codes. Do you think it is possible? Otherwise, how could we move to new clusters without the need for admins to install anything?
The key is to compile all the code you need by yourself, using the compiler/library/MPI toolchains installed by the admins of the clusters, so that
your software is compiled properly for the cluster hardware, and
you do not depend on the admin to install the software.
The following are very useful in this case:
Ansible, to upload/manage configuration files, rc files, set permissions, compile your binaries, etc. and deploy a new environment easily on new clusters
Easybuild to install your version of Python with all the needed dependencies, and install other scientific software thanks to the community supported build procedures
CDE to build a package with all dependencies for your binaries on your laptop and use it as-is on the clusters.
More specifically for Python, you can use
virtual envs to setup a consistent set of Python modules across all clusters, independently from the modules already installed; or
Anaconda or Canopy to use a Python scientific distribution
to have a consistent Python install across all clusters.
Don't get me wrong, but I think what you have to do so: stop behaving like amateurs.
Meaning: the integrity of your "system configuration" is one of the core assets of your "business". And you just told us that you are basically unable of easily re-producing your system configuration.
So, the real answer here can't be a recommendation to use this or that technology. The real answer is: you, and the other teams involved in running your operations need to come together and define a serious strategy how to fix this.
Maybe you then decide that the way to go is that your development team provides Docker buildfiles, so that your operations team can easily create images on new machines. Or you decide that you need to use something like ansible to enable centralized control over your complete environment.
That's what venv is for, it allows you to create a portable customized environment easily, with exactly what you need and nothing more.
I completely agree with https://stackoverflow.com/users/1531124/ghostcat
but here is the really bad answer that will cause you a lot of problems in near future!!!:
if you need some dynamic library and you are not planning to upgrade them in future, you can try copying all needed libs to a folder in your app and use an script to launch the app:
#!/bin/sh
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/your/lib/folder
./myAPP
but keep in mind that this is bad practice.
Create a chroot image, like here - click. Install everything you need and then you can just chroot into it on any machine.
I work on scientific clusters as well, and you are going to find that wherever you go.
I would only rely on the admins on installing the most basic stuff. That is:
- Software necessary to build your software or run the most basic stuff: compilers and most basic utilities (python, perl, binutils, autotools, cmake, etc.).
Software libraries that make use of I/O devices: MPI, file I/O libraries...
A queue system (they already have it most of the time).
Environment modules. This is not a must, but it really helps you get the job done, specially if you mess with different library versions or implementations (that's my case, for example).
From that point on, you can build and install on your own directories all the software you use most of the time.
This does not mean that you cannot ask an admin to install some libraries. If you feel that many people is going to benefit from that, then you should request its installation. In addition, you may need some specific version or some special features which are not used most of the time, but you really need them. A very good example is with BLAS libraries (basic lineal algebra subroutines):
You have lots of BLAS implementations available: the original BLAS, Intel MKL, OpenBLAS, ATLAS, cuBLAS
If that is not enough, the open source versions usually offer multiple configuration options: serial version, parallel version with PThreads, parallel version with OpenMP, parallel version with MPI...
In my particular case, most of the software that I felt was necessary for many users in the cluster ended up being installed by the admins without any problem (either me or other users requested it), but you also have to keep in mind that in a cluster there can be many users and a single person/team is not able to attend the specific requirements you need, specially if you are able to do so.
I think you want to containerize your application in some way. Two main options (because docker/rkt and similar things are way too heavyweight for your task if I understand it correctly) in my opinion are runc and snappy.
Runc relies on OCI runtime specification, you need to create an environment (that is very similar to chroot environment in that you need to copy everything you software uses in one directory) and then you'll be able to run your application with runc tool. Runc itself is just one binary, at the moment it requires root privileges to run (hello, cluster admins), but there are patches at least partly solving that, so if you build your own runc and there are no blocking things wrt root privilege requirements you may be able to run your application with no administration overhead at all.
Snappy is similar in that you need to prepare a snap package for your application, this time using snapcraft as an assistant tool. Snappy is probably a bit easier in creating an application image and IMO is certainly better for long-term support because it clearly separates your application from the data (kinda W^X, application image is a read-only squashfs file and application can only write to a limited set of directories). But at the moment it will require your cluster admins to install snapd and to perform some operations like snap installation that require root privileges. Still, it should be better than your current situation, because that's just one non-intrusive package to install.
If these tools don't fit for some reason, there is always an option to make something of your own. That won't be easy and there are many subtle details that can bite you when doing that, but it can be done, compile all of your dependencies and applications into some path, create wrapper scripts to set up PATH and LD_LIBRARY_PATH environment for your components and then bring that directory into the new cluster, run wrapper scripts instead of target binaries and that's it. It's similar to what XAMPP does, they have quite a number of integrated things packaged into one directory that works across many distributions.
update
Let's also add AppImage into the mix, theoretically it can be a savior for your case, as it specifically does not require root privileges. It's kinda inbetween Snappy and rolling your own, as you need to prepare your application directory yourself (snappy can manage some of dependencies with snapcraft when you just specify "I need this Ubuntu package"), add appropriate metadata and then it can be packaged into single executable.
I've been assigned a project to write some kind of a script that will perform a sanity check on a Linux server implementation to determine if it has a number of dependencies installed before source code is deployed to it. I need to check for the presence of applications such as PHP, Nginx, PostgreSQL, etc and likely confirm version numbers for these as well. These dependencies are required for the given source code to be able to run properly on the server.
The problem is, I'm not sure how to approach this due to my novelty in working with Linux. I've done some research on this and thought that the solution might be to use a combination of combing through the list of running services with a command such as "chkconfig --list" and pinging individual applications with commands such as "php -v" and then asserting the that results from these equate to what I'm looking for.
Pardon if that makes no sense whatsoever, I really am new to this. I was then thinking I could place these "tests" inside of a shell script or something that could be run whenever a test on the server needed to be executed. I would aggregate the true/false results of my assertions and output whether the sanity check passed based on that. Any guidance would be greatly appreciated.
Thank you.
Revision: In lieu of a shell script, I was also thinking I could write this in Python. Does anybody know of any good Python libraries that allow querying of system services?
If your target systems are managed by reasonable people, the software will be managed by the packaging system. On Redhat, Fedora, CentOS or SUSE systems that will be RPM. On any system derived from Debian it will be APT.
So your script can check for one of those two packaging systems. Although be warned that you can install RPM on a Debian system so the mere presence of RPM doesn't tell you the system type. Packages can also be named differently. For example, SUSE will name things a bit differently from Redhat.
So, use uname and/or /etc/issue to determine system type. Then you can look for a particular package version with rpm -q apache or dpkg-query -s postgresql.
If the systems are managed by lunatics, the software will be hand-built and installed in /opt or /usr/local or /home/nginx and versions will be unknown. In that case good luck.
I downloaded this shell script from this site.
It's suspiciously large for a bash script. So I opened it with text editor and noticed
that behind the code there is a lot of non-sense characters.
I'm afraid of giving the script execution right with chmod +x jd.sh. Can you advise me how to recognize if it's safe or how to set it's limited rights in the system?
thank you
The "non-sense characters" indicate binary files that are included directly into the SH file. The script will use the file itself as a file archive and copy/extract files as needed. That's nothing unusual for an SH installer. (edit: for example, makeself)
As with other software, it's virtually impossible to decide wether or not running the script is "safe".
Don't run it! That site is blocked where I work, because it's known to serve malware.
Now, as to verifying code, it's not really possible without isolating it completely (technically difficult, but a VM might serve if it has no known vulnerabilities) and running it to observe what it actually does. A healthy dose of mistrust is always useful when using third-party software, but of course nobody has time to verify all the software they run, or even a tiny fraction of it. It would take thousands (more likely millions) of work years, and would find enough bugs to keep developers busy for another thousand years. The best you can usually do is run only software which has been created or at least recommended by someone you trust at least somewhat. Trust has to be determined according to your own criteria, but here are some which would count in the software's favor for me:
Part of a major operating system/distribution. That means some larger organization has decided to trust it.
Source code is publicly available. At least any malware caused by company policy (see Sony CD debacle) would have a bigger chance of being discovered.
Source code is distributed on an appropriate platform. Sites like GitHub enable you to gauge the popularity of software and keep track of what's happening to it, while a random web site without any commenting features, version control, or bug database is an awful place to keep useful code.
While the source of the script does not seem trustworthy (IP address?), this might still be legit. With shell scripts it is possible to append binary content at the end and thus build a type of installer. Years ago, Sun would ship the JDK for Solaris in exactly that form. I don't know if that's still the case, though.
If you wanna test it without risk, I'd install a Linux in a VirtualBox (free virtual-machine software), run the script there and see what it does.
Addendum on see what it does: There's a variety of tools on UNIX that you can use to analyze a binary program, like strace, ptrace, ltrace. What might also be interesting is running the script using chroot. That way you can easily find all files that are installed.
But at the end of the day this will probably yield more binary files which are not easy to examine (as probably any developer of anti-virus software will tell you). Therefore, if you don't trust the source at all, don't run it. Or if you must run it, do it in a VM where at least it won't be able to do too much damage or access any of your data.
I'm going to rebuild my Linux box [yet] again. I have to create a few user groups, user accounts and install my standard packages. Until now I've just used the GUI tools. I was wondering if anyone has any recommendations on writing a script to create users, groups and install standard packages after I do a minimal install of my latest Fedora build? Sometimes I run Ubuntu so I'd like the script to be somewhat generic.
For .deb distros, use FAI. For .rpm distros, use Kickstart. For system management after installation, use cfengine.
Fedora and Ubuntu use totally different package managers, so you won't be able to easily do it in any sort of generic way.
In CentOS (which is RedHat Enterprise Edition with the serial numbers filed off, and so therefore pretty close to Fedora), we did this using Kickstart files. These files have a simple syntax that enabled you to specify users, groups and packages to install, and even to script some custom stuff.
While I haven't done this yet, I have a similar problem. I'm considering a virtualization host and multiple client OS (Ubuntu and CentOS being the top 2 candidates) - that way once I get the client configured as I want it, I can save it off for reloading as needed.
Doesn't get around the original setup issue, but does limit the "rebuild my Linux box [yet] again" problem.
You may want to consider it.
It may be overkill but you can check out Puppet.
From their website:
Puppet is a system for automating
system administration tasks.
I'm just starting looking for ways to automate system administration, so I don't have much experience with it yet.
If all you need to do is create users and groups and install packages then I would suggest that you just write two separate scripts.
It might be that you could share the users and groups part but only if all the distributions you use have the same policy for creating them (for example Ubuntu creates a group for each user while I am sure some distributions have a "users" group as well).
You could take a look at the useradd and groupadd commands which should be available everywhere. For Ubuntu there is also the friendlier adduser and addgroup and I would not be surprised if Fedora has a set of similar commands.
After groups are setup you just need to feed the package manager a big list of packages you need to have installed. Trying to install packages which are already installed should be safe, so you could install the packages you need on a "clean" new install and then dump a package list.
So to summarize: If you don't plan to support more than two distributions then I suggest just writing the two scripts separately.
Another option to help with constantly rebuilding a box is Norton Ghost, with ghost you can make an image and then just re-image the drive as needed. You install it and configure it to your liking, then take an image.
It's gonna be difficult to make the script generic, but you could use any sort of scripting tool (bash, or ruby or whatever) and try and check what distro is running and then run the appropriate commands to install software. There are various ways to check what distro is running here
Creating groups should be the same on all distros, and you may even be able to drop in an already configured /etc/passwd and /etc/groups (though I haven't tried that, and it may not work).
The response above, about the different distros using different methods is dead on. It's like trying to use the same part for a Chevy and a Ford (there's the car analogy, for you).
The easiest method I've found is to learn about setting up partitions for the different mount points i.e. / ; /home ; /var ; /opt are the big ones.
This lets you keep your users, groups, and many of your apps during your rebuilds. Changing distros will break a lot of things, but your user accounts should still be there.