I am packaging some Python libraries as RPMs. Some of the libraries are only available as source distributions (no wheels).
In my RPM spec I do:
pip install --root=%{buildroot} --prefix=/x/y tornado
When rpmbuild finishes up it runs check-buildroot, and the build fails with errors like:
Binary file /a/b/c/BUILDROOT/my-rpm-1.0.0-1.el7.x86_64/x/y/lib64/python2.7/site-packages/tornado/speedups.so matches
I see the %{buildroot} path listed if I run strings tornado.so | grep BUILDROOT.
How can I sanitize the .so files? Or more generally, how can I make check-buildroot pass?
I figured out how to remove the paths from the SO files.
I determined that the paths were embedded debug information using this command:
readelf --debug-dump=line speedups.so | less
The strip command can remove debug information from SO files, so I added this to my RPM spec:
BuildRequires: binutils
set +e
find "%{buildroot}{%_prefix}/lib64/python2.7/site-packages" -type f -name "*.so" | while read so_file
do
strip --strip-debug "$so_file"
done
set -e
Note: strip segfaults on some SO files, and it's not clear why. I disabled immediate exits with set +e so that the build ignores them.
Related
I am trying to compile grammar parser https://github.com/RichardMoot/Grail into Linux program according to instructions https://github.com/RichardMoot/Grail/blob/master/README and http://www.labri.fr/perso/moot/tutorial/install.html. There is manual how to create Linux executable from SWI-Prolog code http://www.swi-prolog.org/FAQ/UnixExe.html. All that is fine. But I can not find in the Makefile https://github.com/RichardMoot/Grail/blob/master/Makefile any compilation command. SWI-Prolo uses swipl command for compilation but this Makefile swipl calls only once - for displaying the version of the swipl.
I experience some hardship in installation and compilation, that is fine, I can execute/debug Makefile line by line and arrive at the result. But there is problem in my case - I can not see the ultimate goal in my makefile: which lines are responsible for the production of object files (if necessary) and which lines are responsible for the creation of the final Linux executable.
This is windowed program. The source code and documentation contains warnings about incompatibility with the SWI-Prolog 7, but that is fine, I can resolvem them myself, but as I said - I can not see the Makefile lines for creation of exe.
The source code is created by eminent scientist and I certainly don't want to disturb him by so low-level technical question. I would be happy if he continues work on theory and doesn't waste time on low level programming questions. Hope, that there are SWI-Prolog experts.
I am using latest (7.x) SWI-Prolog on Ubuntu 16.x and I have already installed all the mentioned prerequisites.
If you look closely at the provided Makefile, you'll find that the rules all and install are defined as follows (comments added by me):
all:
-cd source ; $(edit) g3 > g3.tmp # Replaces placeholders for your
# ... GRAIL_ROOT install directory.
-cd source ; mv -f g3.tmp g3 # Overwrites `g3` with the filled file.
cd source ; chmod a+x g3 # Makes it executable.
install: # Essentially copies all files to
-mkdir $(datarootdir) # ... your install directory.
-mkdir $(datadir)
cp -f $(images) $(datadir)
-mkdir $(bindir)
cp -f source/insertdot $(bindir)
chmod a+x $(bindir)/insertdot
cp -f $(resources) $(datadir)
cp -f source/*.pl $(bindir)
cp -f source/g3 $(bindir)
If you then do the common make && make install you'll end up with two folders installed in your Grail directory: bin and share. Inside the binary directory you'll have the g3 file that, regardless of being a SWI-Prolog source, has this initial line:
#!/usr/bin/swipl -q -g start -f
% [... prolog code.]
This header should allow your console terminal to determine what interpreter to use for this script (in this case, swipl). In my case, executing Grail with ./g3 returned a SWI-Prolog message indicating that wrong options/command arguments were used.
According to the man, Unix systems have to use option -s at the end of the header (but this didn't work either in my case):
From the manual:
-s file
Load file as a script. This option may be used from the shell to
make Prolog load a file before entering the toplevel.
It is also used to turn a file into an executable Prolog script
on Unix systems using the following first line
#!/usr/bin/swipl option ... -s
If you want to run this program, simply call the same command from your terminal:
swipl -q -g start -s g3
As I understand it, the purpose of the Source: header in an rpm spec file is to specify a file (often a tar archive) that is used as the package payload. This source file is typically generated beforehand, perhaps by make, and then rpmbuild is executed afterwards.
I'm wondering if it's possible to cut make out of the picture and just use rpmbuild. Can the source file be created as part of the rpmbuild process itself, perhaps in the %build or %install step in the spec file?
If so, what does one use as the Source: header? I can't just leave it blank, because rpmbuild complains. Is there a way to tell rpm that the payload file is generated from within the spec file, and not supplied externally?
I don't want to create the source file separately because that would mean keeping track of the package name and version number in two places: in the rpm spec file, and also in whatever makefile or other script creates the payload file. It seems like I should be able to do everything from within the spec file.
You want to call
%setup -q -c -T
see [1]. Example:
Name: test
Version: 1
Release: 1%{?dist}
Summary: Test
License: ...
%description
Test
%prep
%setup -q -c -T
%build
echo "int main() { return 0; } " > test.cpp
gcc -o test test.cpp
%install
install -Dpm 0755 test %{buildroot}%{_bindir}/test
%files
%{_bindir}/test
%changelog
...
[1] http://www.rpm.org/max-rpm/s1-rpm-specref-macros.html
I had the a similar problem recently. I ended up writing a script that read the spec file and did all the work. We no longer had to manage Makefiles and spec files and whatever other ways some of our developers were managing this.
The script is publicly posted on github at:
https://github.com/alexnelsone/jenkins-rpm-build/blob/master/README.md.
It parses the info out of the spec file and then does all the work for you.
Is there any efficient way (maybe by abusing the gcc preprocessor?) to get a set of stripped kernel sources where all code not needed according to .config is left out?
Well got some steps into a solution.
First, one can obtain the used compiler commands by
make KBUILD_VERBOSE=1 | tee build.log
grep '^ gcc' build.log
For now, I select only one gcc command line for further steps. For example the build of kernel/kmod.c, it looks like:
gcc <LIST OF MANY OPTIONS> -c -o kernel/kmod.o kernel/kmod.c
I now remove the option -c, -o ... and add -E, thus disabling compilation and writing preprocessor output to the screen. Further I add -fdirectives-only to prevent macro expansion and -undef to remove the GNU defined macro definitions. -nostdinc to remove the standard c headers is already added by the kernel makefile.
Now includes are still included and thus expanded on the preprocessor output. Thus I pipe the input file through grep removing them: grep -v '#include' kernel/kmod.c. Now only one include is left: autoconf.h is included by the Makefile's command line. This is great as it actually defines the macros used by #ifdef CONFIG_... to select the active kernel code.
The only thing left is to filter out the preprocessor comments and the remaining #defines from autoconf.h by means of grep -v '^#'.
The whole pipe looks like:
grep -v '#include' kernel/kmod.c | gcc -E -fdirectives-only -undef <ORIGINAL KERNEL BUILD GCC OPTIONS WITHOUT -c AND -o ...> - |grep -v '^#'
and the result is a filtered version of kernel/kmod.c containing the code that is actually build into kmod.o.
Questions remain: How to do that for the whole source tree? Are there files that are actually build but never used and stripped at linking?
Kernel Minimization Script :
A project inspired by this question and providing an easy answer...
It contains a Python script that generate a minimized sources code during build time. The new minimized source tree will only contain used sources. (project page)
Info :
The script is tested working with the kernel v4.14.x, however building the kernel one more time from those generated minimized sources require to copy make files and Kconfig files etc... at least we could easily isolate only used source for investigations and development
Usage :
cd /kernel/sources
make
wget https://github.com/Hitachi-India-Pvt-Ltd-RD/minimization/raw/master/minimize.py
export PATH=$PATH:`pwd`
make C=2 CHECK=minimize.py CF="-mindir ../path-to-minimized-source-tree/"
Note & Reminder :
If we are building within and against the targeted machine, we also have the make localmodconfig command that shrink the current config file with only the currently used modules, if used before "Minimization" it will generate further more stripped sources
Compile everything and use atime to find out which files were not used. It might not be very accurate but it's probably worth a try.
Is there any programming way (system call?) to list shared library dependency on Linux? Instead of using ldd ...
readelf -Wa lib.so|grep NEEDED
Gentoo Linux has an lddtree.sh
http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-projects/pax-utils/lddtree.sh?revision=1.22&content-type=text%2Fplain
You may find it helpful.
Set LD_TRACE_LOADED_OBJECTS environment variable to non-empty string and run your binary. Look at this man page.
LD_TRACE_LOADED_OBJECTS
(ELF only) If set to non-empty string, causes the program to list its dynamic library dependencies, as if run by ldd(1), instead of running normally.
This is the simple bash script I use myself on Fedora, it relies on find-requires of rpm package, you can look inside find-requires to find what tools it internally uses.
#!/bin/bash
#
# Use rpm to recursively list dependencies of all files in a directory
#
# Syntax:
# lsdep path/to/directory
# Example:
# lsdep /usr/src/kernels/`uname -r`/
find $1 -type f -exec sh -c 'res=`echo '{}' | /usr/lib/rpm/find-requires`; [ -n "$res" ] && (echo;echo file '{}'; echo $res)' \;
How does one find out what is the lib that the above flag is referring to?
How would I do it for some other one?
The -l option takes the name of the library as the argument so in this case the library would be named libXi.a (or libXi.so or something similar). To find the library look in the standard library locations (/usr/lib, /lib, /usr/local/lib, etc.) available in your distribution. There may also be additional library directories specified using the -L option to the linker.
If your program compiled successfully, or if you have another program which uses -lXi, then you can do:
ldd /path/to/that/program | grep libXi
For example:
$ ldd /usr/X11R6/bin/audacity | grep libXi
libXinerama.so.1 => /usr/lib/libXinerama.so.1 (0x00007f53faaba000)
libXi.so.6 => /usr/lib/libXi.so.6 (0x00007f53f8e2c000)
And that will most likely tell you where that library is. (It's not 100% because the build process could alter the search path but that's usually not likely for standard libraries like X11.)
From there, you can ask your distro which package has that file, if you care. For example on Ubuntu or a .deb-base distro:
$ dpkg --search /usr/lib/libXi.so.6
libxi6: /usr/lib/libXi.so.6
If you can't use LDD, then check your system's /etc/ld.so.conf which will indicate the search path for runtime shared library linking. (/lib/ and /usr/lib are included by default.)
Worst case, you could just find for it:
find / -regex '.*libXi\.\(a\|so\).*' 2> /dev/null
strace -f -e open gcc ... -lXi
Look for libXi in the output.