I'm trying to set up an RPM builder that will compile all the dependent binaries and executable of my project.
The dependency looks a bit like this:
MainProject.rpm | depends on:
|
+-- subProject1.rpm | depends on:
| |
| +-- subProject2.rpm
|
+-- subProject2.rpm
I'm generating all these RPMs in this order:
-build rpm for subProject2
-install subProject2 RPM in System
-build rpm for subProject1
-install subProject1 RPM in System
-build rpm for subProject3
-install subProject3 RPM in System
-build rpm for MainProject
All my spec files are producing suitable RPMs, at the cost of myself having to install on my machine the subProject2.rpm before attempting to rpmbuild subProject1.
Same goes for the mainProject.rpm : if I want to build it, I have to install all the RPMs it depends on.
I feel like this way of doing things is very bad, because I'm installing these RPMs in my builder's filesystem.
Is there an RPMbuild option to, say, deploy the RPM dependency in a chroot-like environment to build another one? I think if there exists such thing, it also needs to take in account the RPATH.
This is only a bad thing if your build environment is not controlled as far as I'm concerned. Assuming you have a controlled, repeatable build process doing things this way should be fine I believe.
That being said the answer to your actual question is to do what Fedora/etc. are doing and use Koji or at least the underlying chroot-related piece called Mock.
You could also consider using the '--root' option. Use it with both the 'rpm' command and for the 'rpmbuild' command. With this option, all the rpm constraints and actions will be in relation to this 'chroot-like' environment. It must be a fully qualified path.
Ex:
rpmbuild --root /home/user/master-project/rpmroot
There are at least three major implications for this:
1) you must initialize an rpm database in this area before you can use it for other commands;
rpm --initdb --root /home/user/master-project/rpmroot
2) all dependencies in the alternate root must be met by some other package in the alternate root. this can get difficult if, in your case for example, 'MainProject' depends on standard libraries.
3) as you alluded, compilers/linkers must also know about the alternate root.
Ex:
LIBRARY_PATH=/home/user/master-project/rpmroot/usr/lib
C_INCLUDE_PATH=/home/user/master-project/rpmroot/usr/include
hope this helps.
Related
My knowledge on Linux administration is limited and hence wanted to check here about the pros and cons of installing any RHEL/CentOS Linux software using rpm packages over installing through tar/zip files.
Thanks
a non-exhaustive list of pros and contras:
rpm
intelligent dependency managment
conflict checking
allow easy and clean uninstall
allow for upgrades / downgrades
list all files owned by a package
a central database with all packages installed, which files they own, their interdependencies
from source
you choose yourself all compiler flags
you can choose a custom installation path
I have tried to explain the diff, pros and cons,
Tar
Basically tar is old way of dealing with in Linux. We can say its existence when the Linux was created.
Usually the tar consists of Source Code and needs to be compiled in binary format for us to use.
Pros:
Using tar packages you gain more control over the programs that you install.
If you want certain portions that avoided, you could do that on the go. Which give you the upper hand.
Cons:
The main issue comes in the maintainability of the packages installed.
They are hard to manage. Once you install, there was no way to manage the software unless and until its well documented. It also hard to version them and you are left blank on the software version you have. The possible reason for this because of the non-indexing nature of files. The files could be spread across your file system, which makes it difficult to remove or upgrade it.
Hard to automate.
It is also hard to automate because of the complexities in maintaining the packages.
Below I tried explaining how tar file are compiled to get better understanding,
Prepare(setup) environment for building
./configure
This script has lots of options that you should change. Like --prefix or --with-dir=/foo. That means every system has a different configuration. Also ./configure checks for missing libraries that should be installed. Anything wrong here causes not to build your application. That's why distros have packages that are installed on different places, because every distro thinks it's better to install certain libraries and files to certain directories. It is said to run ./configure, but in fact you should change it always.
Building the system
make
This is actually make all by default. And every make has different actions to do. Some do building, some do tests after building, some do checkout from external SCM repositories. Usually you don't have to give any parameters, but again some packages execute them differently.
Install to the system
make install
This installs the package in the place specified with configure. If you want you can specify ./configure to point to your home directory. However, lots of configure options are pointing to /usr or /usr/local. That means then you have to use actually sudo make install because only root can copy files to /usr and /usr/local.
Please go through the below link for more information on the above commands
Why always ./configure; make; make install; as 3 separate steps?
RPM
The RPM Package Manager (RPM) is an open packaging system,
RPM packages pre-compiled binary packages (as well as source packages) for an easy one-click installation experience. RPM by itself does not manage dependency and resolve conflicts. When combined with Yum or PackageKit it will resolve all the dependency for the package.
RPM makes system updates easy. Installing, uninstalling and upgrading RPM packages can be accomplished with short commands. RPM maintains a database of installed packages and their files, so you can invoke powerful queries and verification on your system. During upgrades, RPM handles configuration files carefully, so that you never lose your customisation, that you cannot accomplish with regular .tar files.
RPM feature has the ability to verify packages. If you deleted an important file for some package, you can verify the package. You will notified of changes, if any—at which point you can reinstall the package, if necessary. Any configuration files that you modified are preserved during re installation.
Pros:
Install, reinstall, remove, upgrade and verify packages
Use a database of installed packages to query and verify packages
Use metadata to describe packages, their installation instructions, and so on
Package pristine software sources into source and binary packages
Add packages to Yum repositories
Digitally sign your packages
Querying a package (if the package is on your local file system or after the package is installed)
Validating a package (checking a package has not been tampered with, before or after installation).
Cons
Not as customisable as tar.
eg on usability: We will see how to install package using Tar or rpm:
in Tar:
$ tar xvf package.tar
$ cd package
$ ./configure --prefix=PREFIX
$ make
$ make install
in RPM:
rpm -U package-2.4.x-1.i686.rpm
That simple!!.
It basically depends on the usability and the purpose of your use.
Each of them has its on pros and cons depends on how and for what we use it.
I know it a long explanation,how this will give you clear picture. I know there are more untouched such as architecture and execution. I am not pretty confident to explain those here.
In simple words you can say that rpm are prepackaged binaries. They're just ready to go, it does everything for you. But to install rpm and deb you need to be root to have some write permissions. That leaves some serious security hole in the system. You may be unknowingly installing a Torjan horse. Also if the packages are screwed up they may cause the installation to fail altogether.
I personally recommend using tar as you are in more control. It is old school, I know, that's why a bit difficult but, in my opinion best way to go.
You can further refer to the link:
https://tldp.org/HOWTO/Software-Building-HOWTO-4.html
Perl's cpan command is a powerful way to manage Perl modules. However, when maintaining modules system-wide under UNIX, Michal Ingeli notes that another possible option is
yum install 'perl(PerlModuleName)'. If available, should yum be my first resort in this case?
For example, the command cpanm CGI installs the CGI module under my ~/perl5 directory, which may be best if the CGI module is only needed by scripts run under my account. But this won't provide the CGI module to scripts run by other accounts.
I can use cpanm -l <directory> to force the cpanm command to load modules to a specific directory (e.g., cpanm -l /usr/local CGI to install CGI to /usr/local/lib/perl5), or I can edit ~/cpan/CPAN/MyConfig.pm to change the default install location cpan uses.
But on nearly all systems, multiple Perl system library locations exist (/usr/local/share/perl5, /usr/share/perl5/vendor_perl, /usr/lib64/perl5, etc.), and choosing the correct one is somewhat arbitrary since these are not generated by the cpan command.
With this in mind, should I turn to yum (if available) before cpan for system-wide UNIX Perl module management? It's easy enough to test with a command like:
yum install 'perl(LWP::Simple)'
If yum failed in this instance, I would fall back to:
cpanm -l <directory> LWP::Simple
What do you recommend in this type of case, and why?
(Note that nxadm has answered a more general question about this.)
To summarize answers so far:
If at all possible, use the system package manager to update CPAN modules. E.g., for LWP::Simple:
yum install 'perl(LWP::Simple)', or
apt-get install liblwp-simple-perl
If the preceding fails, try to implement a separate Perl environment in which to use CPAN modules not present in the system-wide libraries. Consider local::lib or Perlbrew for this;
Only if the above options don't apply, use cpanm -l <directory> to load the module to a system-wide directory.
I can't speak from experience with RPM/yum systems, but I have done a lot of work with Perl applications on Debian systems and I do highly recommend going with the system packaged versions of CPAN modules if you can. I know a lot of people disagree and historically they may have had good reason but I've been doing it for a long time and find it works very well.
In the Debian world there are an enormous number of Perl modules in pre-packaged form and if you happen to need one that isn't packaged you can build your own package with dh-make-perl and put it in your local apt repository. Being able to run apt-get install your-application and have it pull in all the required dependencies is a real time saver when your code is moving through Dev -> Staging/UAT -> Production workflows. It also gives you confidence that the version of a particular module you're deploying to production is the same as the one you tested in UAT.
One thing you absolutely should not do is use cpanm or the cpan shell as root to install modules into the system directories. If you decide to install direct from CPAN, then use local::lib to install the modules in an application-specific lib directory.
[Edit] Some sample commands as requested:
On a Debian-based system, you would first install the dh-make-perl tool:
sudo apt-get-install dh-make-perl
Then to download a package from CPAN and build it into a .deb file you would run a command like this*:
dh-make-perl --build --cpan Algorithm::CouponCode
You could install the resulting .deb file with:
sudo dpkg -i libalgorithm-couponcode-perl_1.005-1_all.deb
Managing your own apt repository is a whole other topic. In my case I'd copy the .deb to an appropriate directory on the local apt server and run a script to update the index (I think our script uses dpkg-scanpackages).
Note in my opening paragraph above I recommend using systems packages "if you can". To be clear, I meant in the case where most of the modules you want are already packaged by Debain. The example above did not build packages for any dependencies. If your app involves installing modules which have long dependency chains that are not in Debian already, then using cpanm and local::lib will simplify the install. But then you shoulder the burden of repeating that as your code advances through staging to production servers. And you may need to use cpanfile or carton to make sure you're getting the same versions at each step.
* one gotcha: if you have previously set up local::lib so that cpan installs go into a private directory (e.g.: /home/user/perl5) then that will affect the pathnames used in the .deb produced by dh-make-perl. To avoid that, run this before dh-make-perl:
unset PERL5LIB PERL_LOCAL_LIB_ROOT PERL_MB_OPT PERL_MM_OPT
Your system's perl was put there for your system's use. The folks that maintain your distribution will update it when they see fit to another version that suits the needs of your system. Using your system's Package Manager to manage it is really your best idea.
Feel free to use it, but if you need a different version, for whatever reason, you are best rolling your own into a separate location. When maintaining your own perl install, use CPAN.
Alright so after trying to chase down the dependencies for various pieces of software for the n-th time and replicating work that various people do for all the different linux distributions I would like to know if there is a better way of bundling various pieces of software into one .rpm or .deb file for easier distribution.
My current set up for doing this is a frankenstein monster of various tools but mainly Vagrant and libguestfs (built from source running in Fedora because none of the distributions actually ship it with virt-diff). Here are the steps I currently follow:
Spin up a base OS using either a Vagrant box or by create one from live CDs.
Export the .vmdk and call it base-image.
Spin up an exact replica of the previous image and go to town: use the package manager,
or some other means, to download, compile, and install all the pieces that I need. Once again, export the .vmdk and call it non-base-image.
Make both base images available to the Fedora guest OS that has libguestfs.
Use virt-diff to diff the two images and dump that data to file called diff.
Run several ruby scripts to massage diff into another format that contains the information I need and none of the stuff I don't like things in /var.
Run another script to generate a command script for guestfish with a bunch of copy-out commands.
Run the guestfish script.
Run another script to regenerate the symlinks from diff because guestfish can't do it.
Turn the resulting folder structure into a .deb or .rpm file and ship it.
I would like to know if there is a better way to do this. You'd think there would be but I haven't figured it out.
I would definitely consider something along the lines of:
A)
yum list (select your packages/dependencies whatever)
use yumdownloader on the previous list (or use th pkgs you have already downloaded)
createrepo
ship on media with install script that adds the cd repo to repolist, etc.
or B)
first two steps as above, then pack the rpms into an archive build a package that contains all of the above and kicks off the actual install of the rpms (along the lines of rpm -Uvh /tmp/repo/*) as a late script (in the cleanup phase, maybe). Dunno if this can be done avoiding locks on the rpm database.
I think you reached the point of complexity - indeed a frankenstein monster - where you should stop fearing of making proper packages with dependencies. We did this in my previous work - we had a set of fabricated rpm packages - and it was very easy and straightforward, including:
pre/post install scripts
uninstall scripts
dependencies
We never had to do anything you just described. And for the customer, installing even a set of packages was very easy!
You can follow a reference manual of how to build RPM package for more info.
EDIT: If you need a single installation package, then create this master packge, that would contain all the other packages (with dependencies set properly) and installed them in the post-install script (and uninstalled them in the uninstall script).
There are mainly 3 steps to make a package with all dependencies (let it be A, B & C).
A. Gather required files.
There are many way to gather files of the main software and its dependencies. In order to get all the dependices and for error free run you need to use a base OS (i.e live system)
1. Using AppDirAssistant
This app is used by www.portablelinuxapps.org to create portable app directory. They scan and watch for the files accessed by the app to find required.
2. Using chroot & overlayfs
In this method you don't need to boot into live cd instead chroot into it.
a. mount the .iso # /cdrom and
b. mount the filesystem(filesystem.squashfs) # another place, say # /tmp/union/root
c. Bind mount /proc # /tmp/union/root/proc
d. Overlay on it
mount -t overlayfs overlayfs /tmp/union/root -o lowerdir=/tmp/union/root,upperdir=/tmp/union/rw
e. Chroot
chroot /tmp/union/root
Now you can install packages using apt-get or another method (only from the chrooted terminal). All the changed files are stored # /tmp/union/rw. Take files from there.
3. Using manually collected packages
Use package manager to collect dependencies. For example
apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted).
B. Clean garbage files
After gathering files remove unwanted files
C. Pack files
1. Using appImageAssistance
If you manually gathered files then you need to copy appname.desktop file from ./usr/share/applications to root of directory tree. Also copy file named AppRun from another app or extract it from AppDirAssistance.
2. Make a .deb or .rpm using gathered files.
Is the problem primarily that of ensuring that your customers have installed all the standard upstream distro packages necessary for your package to run?
If that's the case, then I believe the most straightforward solution would be to leverage the yum and apt infrastructure to have those tools track down and install the necessary prerequisite packages.
If you supply a native yum/apt repository with complete pre-req specs (the hard work you've apparently already completed). Then the standard system install tool takes care of the rest. See link below for more on creating a personal repository for yum/apt.
For off-line customers, you can supply media with your software, and a mirror - or mirror subset - of the upstream distro, and instructions for adding them to yum config/apt config.
Yum
Creating a Yum Repository in the Fedora Deployment Guide
Apt
How To Setup A Debian Repository on the Debian Wiki
So your customers aren't ever going to install any other software that might specify a different version of those dependencies that you are walking all over, right?
Why not just create your own distro if you're going to go that far?
Or you can just give them a bunch of packages and a single script that does rpm -i dep1 dep2 yourpackage
I have a libfoo-devel rpm that I can create, using the trick to override _topdir. Now I want to build a package "bar" which has a BuildRequires 'libfoo-devel". I can't seem to find the Right Way to get access to the contents of libfoo-devel without having to install it on the build host. How should I be doing it?
EDIT:
My build and target distros are both SuSE.
I prefer solutions that don't require mock, since I believe SuSE does not include it in its stock repo.
Subsequent EDIT:
I believe that the answer I seek is in the build package. Perhaps it's SuSE's answer to mock? Or it's the distributed version of the oBS service?
DESCRIPTION
build is a tool to build SuSE Linux
RPMs in a safe and clean way. build
will install a minimal SuSE Linux as
build system into some directory and
will chroot to this system to compile
the package. This way you don't risk
to corrupt your working system (due to
a broken spec file for example), even
if the package does not use BuildRoot.
build searches the spec file for a
BuildRequires: line; if such a line is
found, all the specified rpms are
installed. Otherwise a selection of
default packages are used. Note that
build doesn't automatically resolve
missing dependencies, so the specified
rpms have to be sufficient for the
build.
Note that if you really don't need libfoo-devel installed to build package bar the most sensible alternative would be to remove libfoo-devel from the BuildRequires directive (and maybe put the requirement where it belongs).
However, if you cannot do that for some reason, create a "development" rpm database. Basically it involves using rpm --initdb --root /path/to/fake/root. Then populate it with all of the "target packages" of your standard distro installation.
That's a lot of rpm --install --root /path/to/fake/root --justdb package-name.rpm commands, but maybe you can figure out a way to copy over your /var/lib/rpm/* database files and use those as a starting point. Once you have the alternative rpm database, you can fake the installation of the libfoo-devel package with a --justdb option. Then you'll be home free on the actual rpm build.
If neither mock nor the openSUSE Build Service are a viable choice then you will have to buckle down and install the package, either directly or in a chroot; the package provides files that the SRPM packager has decided are required to build, and hence is in the BuildRequires tag.
It seems that the RPM logic is quite different from what I know already and I am having some issues understanding the "RPM logic". For my work, I have to create a documentation on "How-to create a RPM package on Red Hat 5".
I'm used to Debian and it's derivatives (Ubuntu, and so on) and thus to Debian packages (aka. .deb files).
From what I read, it seems that ones need to be root to create a RPM package. While I understand why root could be required to install a package, I still don't understand why elevated privileges should be needed just to create one.
If I try to create a RPM package as a user, changing the buildroot it fails on the %installstep because I don't have permission to write files into /usr/bin. Fair enough but... why does it want to copy my files into /usr/bin at this step?! I just want to create the package, not install it!
I'm sure I'm missing something here. Is there anyone who could give me at least a basic understanding of how rpmbuild works and why?
Will this do?
You don't need to be root to build RPM packages. I recommend you to read this two part article to get you started.
The official Maximum RPM book also has a chapter on Having RPM Use a Different Build Area, which allows non-root users to build RPMs.