Compiling static library for Google Native Client using SCons - scons

I'm working on a few multi platform projects that all depend on common framework.
I want to add support for Google Native-Client (NaCl). The way I aproached the problem is first to compile the framework as static library (this is how I've been doing it on all other platforms).
I have to say that I have never used SCons before. I think I start grasping it. Starting from a build.scons from a tutorial I can get some code compiling and linking. Now I would want to skip the linking process but seems like the nacl_env was never intended to compile static libraries.
Reading the SCons help didn't help me much since the Library node is missing from the nacl_env.
I don't think I understand SCons enough to write the whole build process from scratch so I was hopping to not have to do so.
1. Am I approaching the problem correctly?
2. Any tips or sample nacl static libs, build using SCons?

Ok, what I did is way more trickery than what you probably need.
I wanted my static library to handle the initialization steps of the NaCl module, and then call some project-specific function.
I ended up turning my whole framework and the contents of the built-in libppapi_cpp.a into a single .o file, and then that into a single .a file, a static library.
I needed a single .o file, because otherwise I would run into dependency problems releated to initialization, I could not solve.
build_lib.sh (framework):
#!/bin/bash -e
SDK="/home/kalmi/ik/nacl_sdk/pepper_15"
function create_allIn_a {
TMPDIR="`mktemp -d`"
echo $TMPDIR
cp $O_FILES $TMPDIR
pushd $TMPDIR &> /dev/null
$AR x $LIBPPAPI_CPP_A
$LD -Ur * -o ALL.o
$AR rvs $OUTPUT_NAME ALL.o
$RANLIB $OUTPUT_NAME
popd &> /dev/null
}
./scons
BIN_BASE="$SDK/toolchain/linux_x86/bin"
LD="$BIN_BASE/i686-nacl-ld"
AR="$BIN_BASE/i686-nacl-ar"
RANLIB="$BIN_BASE/i686-nacl-ranlib"
LIBPPAPI_CPP_A="$SDK/toolchain/linux_x86_newlib/x86_64-nacl/lib32/libppapi_cpp.a"
O_FILES="`find $(pwd)/opt_x86_32 | grep .o$ | grep --invert-match my_main.o | tr "\n" " "`"
LIBDIR="../../../bin/lib/lib32"
mkdir -p $LIBDIR
if [ -f $LIBDIR/libweb2grid_framework.a ]; then
rm $LIBDIR/libweb2grid_framework.a
fi
OUTPUT_NAME="`readlink -m $LIBDIR/libweb2grid_framework.a`"
create_allIn_a
BIN_BASE="$SDK/toolchain/linux_x86/bin"
LD="$BIN_BASE/x86_64-nacl-ld"
AR="$BIN_BASE/x86_64-nacl-ar"
RANLIB="$BIN_BASE/x86_64-nacl-ranlib"
LIBPPAPI_CPP_A="$SDK/toolchain/linux_x86_newlib/x86_64-nacl/lib64/libppapi_cpp.a"
O_FILES="`find $(pwd)/opt_x86_64 | grep .o$ | grep --invert-match my_main.o | tr "\n" " "`"
LIBDIR="../../../bin/lib/lib64"
mkdir -p $LIBDIR
if [ -f $LIBDIR/libweb2grid_framework.a ]; then
rm $LIBDIR/libweb2grid_framework.a
fi
OUTPUT_NAME="`readlink -m $LIBDIR/libweb2grid_framework.a`"
create_allIn_a
./scons -c
The my_main.o file is excluded from the static library, because that file contains the function that is to be provided by the project that uses this framework.
The build.scons file for the framework is truly ordinary.
build.scons (for some project that uses this framework):
#! -*- python -*-
#What to compile:
sources = [ 'src/something.cpp', 'src/something_helper.cpp' ]
###############################################################x
import make_nacl_env
import nacl_utils
import os
nacl_env = make_nacl_env.NaClEnvironment(
use_c_plus_plus_libs=False,
nacl_platform=os.getenv('NACL_TARGET_PLATFORM'))
nacl_env.Append(
# Add a CPPPATH that enables the full-path #include directives, such as
# #include "examples/sine_synth/sine_synth.h"
CPPPATH=[os.path.dirname(os.path.dirname(os.path.dirname(os.getcwd())))],
LIBS=['web2grid_framework','srpc'],
LIBPATH=['../../../bin/lib/lib32','../../../bin/lib/lib64'],
LINKFLAGS=['-pthread']
)
nacl_env.AllNaClModules(sources, 'client')
Some lines worth highlighting:
use_c_plus_plus_libs=False,
LIBS=['web2grid_framework','srpc'],
LIBPATH=['../../../bin/lib/lib32','../../../bin/lib/lib64'],
LINKFLAGS=['-pthread']
I am not saying that this is a clean method, but it gets the job done.

So, there's two questions here
1. Using SCONS:
NaCl uses SCONS for it's examples, simply to help compiling of the examples easier. In reality, SCONS simply directs to the GCC/G++ compilers in the SDK build directories. (SCONS will take the input scripts, and create the final param string to send to GCC)
GCC is a common compiler, and is well documented on the net : http://gcc.gnu.org/
How you integrate NaCl compilation into your work-flow is up to you (ie you're not forced to use SCONS).
For instance, if you'd like to go to GCC directly, you can simply call :
<path to bin>/x86_64-nacl-gcc -m64 -o test.nexe main.c
For a more detailed look into how to compile NaCl modules, please read the documentation # gonacl.com on compiling which will detail how to compile with and without SCONS.
2.Compilng Static libs with GCC
Here is an example : http://www.adp-gmbh.ch/cpp/gcc/create_lib.html
~Main

Related

makefile: link to a library if it installed. How to? and how to disable?

I am building rocksdb
seem rocksdb's makefile will detect my system's compression library(snappy, libz), if installed rocksdb will link to it.
I am interested in how rocksdb do that. The makefile is so complicate and I can't find which part do the job.
I hope to prevent this action, I hope rocksdb only link snappy, not zlib which download in my linux machine.
makefile:https://github.com/facebook/rocksdb/blob/main/Makefile
Thank to #John Bollinger 's comment.
This makefile work as configuration script generates the make_config.mk file, and the PLATFORM_LDFLAGS in make_config.mk indicates the library that the system has.
build_detect_platform is responsible for generating make_config.mk. It work like this
if ! test $ROCKSDB_DISABLE_SNAPPY; then
# Test whether Snappy library is installed
# http://code.google.com/p/snappy/
$CXX $PLATFORM_CXXFLAGS -x c++ - -o test.o 2>/dev/null <<EOF
#include <snappy.h>
int main() {}
EOF
if [ "$?" = 0 ]; then
COMMON_FLAGS="$COMMON_FLAGS -DSNAPPY"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lsnappy"
JAVA_LDFLAGS="$JAVA_LDFLAGS -lsnappy"
fi
fi
so I can set ROCKSDB_DISABLE_ZLIB to prevent link to zlib

Is there a way to define custom implicit GNU Make rules?

I'm often creating png files out of dot (graphviz format) files. The command to do so is the following:
$ dot my_graph.dot -o my_graph.png -Tpng
However, I would like to be able to have a shorter command format like $ make my_graph.dot to automatically generate my png file.
For the moment, I'm using a Makefile in which I've defined the following rule, but the recipe is only available in the directory containing the Makefile
%.eps: %.dot
dot $< -o $# -Teps
Is it possible to define custom implicit GNU Make recipes ? Which would allow the above recipe to be available system-wide
If not, what solution do you use to solve those kind of problem ?
Setup:
Fedora Linux with ZSH/Bash
You could define shell functions in your shell's startup files, e.g.
dotpng()
{
echo dot ${1%.dot}.dot -o ${1%.dot}.png -Tpng;
}
This function can be called like
dotpng my_graph.dot
or
dotpng my_graph
The code ${1%.dot}.dot strips .dot from the file name if present and appends it (again) to allow both my_graph.dot and my_graph as function argument.
Is it possible to define custom implicit GNU Make recipes ?
Not without modifying the source code of GNU Make.
If not, what solution do you use to solve those kind of problem ?
I wouldn't be a fan o modyfying the system globally, but you could do:
Create a file /usr/local/lib/make/myimplicitrules.make with the content
%.eps: %.dot
dot $< -o $# -Teps
Use include /usr/local/lib/make/myimplicitrules.make in your Makefile.
I would rather use a git submodule or similar to share common configuration between projects, rather than depending on global configuration. Depending on global environment will make your program hard to test and non-portable.
I would rather go with a shell function, something along:
mymake() {
make -f <(cat <<'EOF'
%.eps: %.dot
dot $< -o $# -Teps
EOF
) "$#"
}
mymake my_graph.dot
GNU Make lets you specify extra makefiles to read using the MAKEFILES
environment variable. Quoting from info '(make)MAKEFILES Variable':
the default goal is never taken from one of these makefiles (or any
makefile included by them) and it is not an error if the files listed
in 'MAKEFILES' are not found
if you are running 'make' without a specific makefile, a makefile
in 'MAKEFILES' can do useful things to help the built-in implicit
rules work better
As an example, with no makefile in the current directory and the
following .mk files in make's include path (e.g. via
MAKEFLAGS=--include-dir="$HOME"/.local/lib/make/) you can create
subdir gen/ and convert my_graph.dot or dot/my_graph.dot by
running:
MAKEFILES=dot.mk make gen/my_graph.png
To further save some typing it's tempting to add MAKEFILES=dot.mk
to a session environment but defining MAKEFILES in startup files
can make things completely nontransparent. For that reason I prefer
seeing MAKEFILES=… on the command line.
File: dot.mk
include common.mk
genDir ?= gen/
dotDir ?= dot/
dotFlags ?= $(if $(DEBUG),-v)
Tvariant ?= :cairo:cairo
vpath %.dot $(dotDir)
$(genDir)%.png $(genDir)%.svg $(genDir)%.eps : %.dot | $(genDir).
dot $(dotFlags) $< -o $# -T'$(patsubst .%,%,$(suffix $#))$(Tvariant)'
The included common.mk is where you'd store general definitions to
manage directory creation, diagnostics etc., e.g.
.PRECIOUS: %/. ## preempt 'unlink: ...: Is a directory'
%/. : ; $(if $(wildcard $#),,mkdir -p -- $(#D))
References:
?= = := … - info '(make)Reading Makefiles'
vpath - info '(make)Selective Search'
order-only prerequisites (e.g. | $(genDir).) - info '(make)Prerequisite Types'
.PRECIOUS - info '(make)Chained Rules'

Pattern syntax %.3: man/libfoo.man in Automake with different base name

I wrote a library libfoo providing functions bar and baz.
I want the user to be able to find the same man-page (from mans/libfoo.man) when they call man libfoo, man bar and man baz (Similar to man fprintf, man sprintf all pointing to the same page.)
My current setup has the files mans/libfoo.man and Makefile.am
To 'tell' automake that I want to end up with the three man-pages I specified the dist_man3_MANS variable.
Makefile.am:
dist_man3_MANS = mans/libfoo.3 mans/bar.3 mans/baz.3
Coming from GNU make, I thought I could just write
%.3: mans/libfoo.man
ln -S libfoo.man $#
to create links temporarily and then let Automake install those accordingly, but Automake errors out with Makefile.am:115: warning: '%'-style pattern rules are a GNU make extension. I want to do it properly and take this warning seriously by not relying on GNU Make to be as portable as possible.
The Automake manual suggests to add a target
.man.3:
$(LN_S) $^ $#
but that just tells Automake that xx.man can be compiled to xx.3, requiring the base name to be the same. I don't want to carry around those xx.man files, so this approach does not work.
I could hack it in with putting a rule
dist_man3_MANS = mans/libfoo.3 mans/bar.3 mans/baz.3
$(dist_man3_MANS): mans/libfoo.man
$(LN_S) libfoo.man $#
but that seems like a dirty hack, because I am not giving it a recipe to compile .man to .3, but rather say: "Hey, you can create those files with this rule", which for this case may work coincidental.
I would follow the example from the Automake info page section Extending Automake Rules and do something along the lines of
LIBFOO_MAN_ALIASES = bar baz
install-data-hook:
set -e; \
cd $(DESTDIR)$(man3dir) && \
for manalias in $(LIBFOO_MAN_ALIASES); do \
$(LN_S) libfoo.3 $${manalias}.3; \
done
uninstall-hook:
cd $(DESTDIR)$(man3dir) && \
for manalias in $(LIBFOO_MAN_ALIASES); do \
rm -f $${manalias}.3; \
done
relying on AC_PROG_LN_S to make sure that $(LN_S) does something reasonable for the system (symlink, hardlink, copy) to create a file name which can be open(2)ed and read.
FTR, I have just taken a look at three different systems' man pages and found them using three different methods to make the fprintf(3) man page show the same man page as printf(3) does:
Debian 10 uses symlinks
Fedora 35 uses a /usr/share/man/man3/fprintf.3 file containing .so man3/printf.3 (while some other man pages use symlinks to achieve the same effect)
FreeBSD 13 uses hardlinks, and find /usr/share/man -type l does not find any symlinks on my relatively clean system. However, manually testing both symlinks and the .so man3/printf.3 method suggests that FreeBSD man(1) does not treat symlinks in any special way and therefore opens the symlinked man page, and it also interprets the .so command just like Fedora 35's man(1) does.
I do not know how portable each of those methods is. Each of these three methods could set up on make install by using an appropriate install-data-hook, but any man file which can be opened using open(2) appears to be work, and therefore $(LN_S) looks like a good bet.

How to replace paths to executables in source code with Nix that are not in PATH

I wish to write some Haskell that calls an executable as part of its work; and install this on a nixOS host. I don't want the executable to be in my PATH (and to rely on that would disrupt the beautiful dependency model of nix).
If this were, say, a Perl script, I would have a simple builder that looked for strings of a certain format, and replaced them with the executable names, based upon dependencies declared in the .nix file. But that seems somewhat harder with the cabal-based building common to haskell.
Is there a standard idiom for encoding the paths to executables at build time (including during development, as well as at install time) within Haskell code on nix?
For the sake of a concrete example, here is a trivial "script":
import System.Process ( readProcess )
main = do
stdout <- readProcess "hostname" [] ""
putStrLn $ "Hostname: " ++ stdout
I would like to be able to compile run this (in principle) without relying on hostname being in the PATH, but rather replacing hostname with the full /nix/store/-inetutils-/bin/hostname path, and thus also gaining the benefits of dependency management under nix.
This could possibly be managed by using a shell (or similar) script, built using a replacement scheme as defined above, that sets up an environment that the haskell executable expects; but still that would need some bootstrapping via the cabal.mkDerivation, and since I'm a lover of OptParse-Applicative's bash completion, I'm loathe to slow that down with another script to fire up every time I hit the tab key. But if that's what's needed, fair enough.
I did look through cabal.mkDerivation for some sort of pre-build step, but if it's there I'm not seeing it.
Thanks,
Assuming you're building the Haskell app in Nix, you can patch a configuration file via your Nix expression. For an example of how to do this, have a look at this small project.
The crux is that you can define a postConfigure hook like this:
pkgs.haskell.lib.overrideCabal yourProject (old: {
postConfigure = ''
substituteInPlace src/Configuration.hs --replace 'helloPrefix = Nothing' 'helloPrefix = Just "${pkgs.hello}"'
'';
})
What I do with my xmonad build in nix1 is refer to executable paths as things like ##compton##/bin/compton. Then I use a script like this to generate my default.nix file:
#!/usr/bin/env bash
set -eu
packages=($(grep '##[^#]*##' src/Main.hs | sed -e 's/.*##\(.*\)##.*/\1/' | sort -u))
extra_args=()
for p in "${packages[#]}"; do
extra_args+=(--extra-arguments "$p")
done
cabal2nix . "${extra_args[#]}" \
| head -n-1
echo " patchPhase = ''";
echo " substituteInPlace src/Main.hs \\"
for p in "${packages[#]}"; do
echo " --replace '##$p##' '\${$p}' \\"
done
echo " '';"
echo "}"
What it does is grep through src/Main.hs (could easily be changed to find all haskell files, or to some specific configuration module) and pick out all the tags surrounded by## like ##some-package-name##. It then does 2 things with them:
passes them to cabal2nix as extra arguments for the nix expression it generates
post-processes nix expression output from cabal2nix to add a patch phase, which replaces the ##some-package-name## tag in the Haskell source file with the actual path to the derivation.2
This generates a nix-expression like this:
{ mkDerivation, base, compton, networkmanagerapplet, notify-osd
, powerline, setxkbmap, stdenv, synapse, system-config-printer
, taffybar, udiskie, unix, X11, xmonad, xmonad-contrib
}:
mkDerivation {
pname = "xmonad-custom";
version = "0.0.0.0";
src = ./.;
isLibrary = false;
isExecutable = true;
executableHaskellDepends = [
base taffybar unix X11 xmonad xmonad-contrib
];
description = "My XMonad build";
license = stdenv.lib.licenses.bsd3;
patchPhase = ''
substituteInPlace src/Main.hs \
--replace '##compton##' '${compton}' \
--replace '##networkmanagerapplet##' '${networkmanagerapplet}' \
--replace '##notify-osd##' '${notify-osd}' \
--replace '##powerline##' '${powerline}' \
--replace '##setxkbmap##' '${setxkbmap}' \
--replace '##synapse##' '${synapse}' \
--replace '##system-config-printer##' '${system-config-printer}' \
--replace '##udiskie##' '${udiskie}' \
'';
}
The net result is I can just write Haskell code and a cabal package file; I don't have to worry much about maintaining the nix package file as well, only re-running my generate-nix script if my dependencies change.
In my Haskell code I just write paths to executables as if ##the-nix-package-name## was an absolute path to a folder where that package is installed, and everything magically works.
The installed xmonad binary ends up containing hardcoded references to the absolute paths to the executables I call, which is how nix likes to work (this means it automatically knows about the dependency during garbage collection, for example). And I don't have to worry about keeping the things I called in my interactive environment's PATH, or maintaining a wrapper that sets up PATH just for this executable.
1 I have it set up as a cabal project that gets built and installed into the nix store, rather than having it dynamically recompile itself from ~/.xmonad/xmonad.hs
2 Step 2 is a little meta, since I'm using a bash script to generate nix code with an embedded bash script in it
This is not indented to be the answer but if I post this in comment section it would turn out to be ugly formatted.
Also I am not sure if this hack is the right way to do the job.
I notice that if I use nix-shell I can get full path to nix store
Assume hash is always the same, AFAIK I believe it is, you can use it to hard-coded in build recipe.
$ which bash
/run/current-system/sw/bin/bash
[wizzup# ~]
$ nix-shell -p bash
[nix-shell:~]$ which bash
/nix/store/wb34dgkpmnssjkq7yj4qbjqxpnapq0lw-bash-4.4-p12/bin/bash
Lastly, I doubt if you have to to any of this if you use buildInput, it should be the same path.

Ocamldoc "Unbound module Thread" without ocamlfind

Is it possible to use ocamldoc for a project with Threads without using ocamlfind? More importantly, how?
-thread or -package aren't supported by ocamldoc, and -I -thread doesn't work.
-thread or -package aren't supported by ocamldoc, and -I -thread doesn't work.
Try with -I +threads instead. This will tell ocamldoc where to look for the thread library files.
On a side note, I use ocamlbuild for generating documentation, when I am already using it for my project builds (which is most of the time). With this tool, you only need to list all the documented ml files in a single .odocl file, and ask for the corresponding .docdir/index.html with the same parameters as a compilation command to get the documentation generated. If your project compiles with ocamlbuild, it should be able to generate documentation without hiccups with it as well.
$ ls src
foo.ml bar.ml baz.zip
$ ls -1 src/*.ml | cut -f1 -d'.' > project.odocl
$ cat project.odocl
src/foo
src/bar
$ ocamlbuild project.docdir/index.html
[...]
$ ls project.docdir
Bar.html
Foo.html
index.html
[...]

Resources