Scons LIBS linking, how to remove duplicate LIBS? - scons

We use indirection when specifying library dependencies, for instance,
env.Program(TARGET, SOURCE, LIBS = [ $LIBS_A, $LIBS_B, $LIBS_C ])
LIBS_A, LIBS_B, and LIBS_C are specified in other SConscript files.
Suppose:
LIBS_A/Sconscript has LIBS_A = [ foo, bar, baz ]
LIBS_B/Sconscript has LIBS_B = [ foo, bar, bat ]
LIBS_C/Sconscript has LIBS_C = [ foo, bar, ben ]
Then the final linker line will have:
-lfoo -lbar -lbaz -lfoo -lbar -lbat -lfoo -lbar -lben
How can we remove duplicates and have the linker line read:
-lfoo -lbar -lbaz -lbat -lben

How's this. files fully listed.
main.cxx
#include <iostream>
int main()
{
std::cout << "Hello World!";
}
SConstruct
env=Environment()
Export('env')
for l in ['A','B','C']:
env.SConscript("LIBS_%s/SConscript"%l)
env.Program('myprogram',['main.cxx'], LIBS = list(set(env['LIBS_A'] + env['LIBS_B'] + env['LIBS_C'])))
LIBS_A/SConscript
Import('env')
env['LIBS_A'] = ['foo','bar','baz']
LIBS_B/SConscript
Import('env')
env['LIBS_B'] = ['foo','bar','bat']
LIBS_C/SConscript
Import('env')
env['LIBS_C'] = ['foo','bar','ben']
Yields:
g++ -o myprogram main.o -lbaz -lbat -lfoo -lbar -lben
If none of those would work for you, here's a more complicated way that should:
class foo(object):
def __init__(self, arg):
self.arg = arg
def __call__(self, target, source, env, for_signature):
print "LIBS_A: %s LIBS_B: %s"%(env['LIBS_A'],env['LIBS_B'])
rc = list(set(env['LIBS_A']+env['LIBS_B']))
print "Unique: %s"%rc
print "SIG:%s"%for_signature
return rc
# Will call our foo method to uniquify LIBS_*
env=Environment(FOO=foo, BAR="${FOO('$LIBS_A $LIBS_B')}")
env.Command('foo.XXXXX','foo.in',action="echo LIBS: $LIBS",LIBS="$BAR",
LIBS_A=['a','b','c','d','e','f','f','b'],
LIBS_B=['b','d','e','g','x'])

This worked for me. Big thanks to #bdbaddog
class uniquify_libs(object):
def __init__(self, arg):
self.arg = arg
def __call__(self, target, source, env, for_signature):
rc = list(set(env.subst(env[self.arg])[0].split()))
return rc
LIBS = ['$LIBS_A', '$LIBS_B'] ## defined in other SConscripts
Program(
target = TARGET,
source = SOURCE,
UNIQUE = uniquify_libs,
MYLIBS = LIBS,
LIBS = "${UNIQUE('MYLIBS')}",
)

Related

Why would dlopen reuse the address of a previously loaded symbol?

I just debugged a strange issue where I have two libraries let's call it libA.so and libB.so
Application dlopens libA.so(EDIT: it isn't: it's linked by the -l option) which is a thin library which then loads libB.so which is the actual implementation.
The dlopen are called using the RTLD_NOW option, no other options are passed.
And both libraries use the same logger module where the logger's state is stored in a global variable, since both use the same logger and linked to them statically the global variable in both of them is of the same name.
When libB is loaded the two global variables are sitting at the same address and conflicting. So the dynamic linker reused the address of the variable to use the same variable in libB.
If it matters this variable is defined deep within a .cpp file, I'm not sure if linking between C and C++ is different.
Reading the dlopen's documentation it says:
RTLD_GLOBAL
The symbols defined by this library will be made available for symbol resolution of subsequently loaded libraries.
RTLD_LOCAL
This is the converse of RTLD_GLOBAL, and the default if neither flag is specified. Symbols defined in this library are not made available to resolve references in subsequently loaded libraries.
So RTLD_LOCAL is supposed to be the default, that is libA's symbols shouldn't be used when resolving libB's symbols. But it's still happening. Why?
As a workaround I added visibility("hidden") option to this global to avoid exporting. And raised a ticket to make all symbols hidden by default, so collisions like this shouldn't happen in the future, but I'm still wondering why this happens when it shouldn't.
EDIT2:
Source example:
commonvar.h:
#pragma once
#include <iostream>
struct A
{
A()
{
std::cout << "A inited. Address: " << this << "\n";
}
virtual ~A() {}
};
extern A object;
struct POD
{
int x, y, z;
};
extern POD pod;
commonvar.cpp:
#include <string>
#include "commonvar.h"
A object;
POD pod = {1, 2, 3};
a.h:
#pragma once
extern "C" void foo();
a.cpp:
#include <iostream>
#include "commonvar.h"
using FnFoo = void (*)();
extern "C" void foo()
{
std::cout << "A called.\n";
std::cout << "A: Address of foo is: " << &object << "\n";
std::cout << "A: Address of pod is: " << &pod << "\n";
std::cout << "A: {" << pod.x << ", " << pod.y << ", " << pod.z << "}\n";
pod.x = 42;
}
b.cpp:
#include <iostream>
#include <string>
#include "commonvar.h"
extern "C" void foo()
{
std::cout << "B called.\n";
std::cout << "B: Address of foo is: " << &object << "\n";
std::cout << "B: Address of pod is: " << &pod << "\n";
std::cout << "B: {" << pod.x << ", " << pod.y << ", " << pod.z << "}\n";
}
main.cpp:
#include <dlfcn.h>
#include <iostream>
#include <cassert>
#include "a.h"
using FnFoo = void (*)();
int main()
{
std::cout << "Start of program.\n";
foo();
std::cout << "Loading B\n";
void *b = dlopen("libb.so", RTLD_NOW);
assert(b);
FnFoo fnB;
fnB = FnFoo(dlsym(b, "foo"));
assert(fnB);
fnB();
}
Build script:
#!/bin/bash
g++ -fPIC -c commonvar.cpp
ar rcs common.a commonvar.o
g++ -fPIC -shared a.cpp common.a -o liba.so
g++ -fPIC -shared b.cpp common.a -o libb.so
g++ main.cpp liba.so -ldl -o main
Dynamic symbols of main:
U __assert_fail
0000000000202010 B __bss_start
U __cxa_atexit
w __cxa_finalize
U dlopen
U dlsym
0000000000202010 D _edata
0000000000202138 B _end
0000000000000bc4 T _fini
U foo
w __gmon_start__
0000000000000860 T _init
w _ITM_deregisterTMCloneTable
w _ITM_registerTMCloneTable
U __libc_start_main
U _ZNSt8ios_base4InitC1Ev
U _ZNSt8ios_base4InitD1Ev
0000000000202020 B _ZSt4cout
U _ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc
Dynamic symbols of liba.so:
0000000000202064 B __bss_start
U __cxa_atexit
w __cxa_finalize
0000000000202064 D _edata
0000000000202080 B _end
0000000000000e6c T _fini
0000000000000bba T foo
w __gmon_start__
0000000000000a30 T _init
w _ITM_deregisterTMCloneTable
w _ITM_registerTMCloneTable
0000000000202070 B object
0000000000202058 D pod
U _ZdlPvm
0000000000000dca W _ZN1AC1Ev
0000000000000dca W _ZN1AC2Ev
0000000000000e40 W _ZN1AD0Ev
0000000000000e22 W _ZN1AD1Ev
0000000000000e22 W _ZN1AD2Ev
U _ZNSolsEi
U _ZNSolsEPKv
U _ZNSt8ios_base4InitC1Ev
U _ZNSt8ios_base4InitD1Ev
U _ZSt4cout
U _ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc
0000000000201dd0 V _ZTI1A
0000000000000ed5 V _ZTS1A
0000000000201db0 V _ZTV1A
U _ZTVN10__cxxabiv117__class_type_infoE
Dynamic symbols of libb.so:
$ nm -D libb.so
0000000000202064 B __bss_start
U __cxa_atexit
w __cxa_finalize
0000000000202064 D _edata
0000000000202080 B _end
0000000000000e60 T _fini
0000000000000bba T foo
w __gmon_start__
0000000000000a30 T _init
w _ITM_deregisterTMCloneTable
w _ITM_registerTMCloneTable
0000000000202070 B object
0000000000202058 D pod
U _ZdlPvm
0000000000000dbe W _ZN1AC1Ev
0000000000000dbe W _ZN1AC2Ev
0000000000000e34 W _ZN1AD0Ev
0000000000000e16 W _ZN1AD1Ev
0000000000000e16 W _ZN1AD2Ev
U _ZNSolsEi
U _ZNSolsEPKv
U _ZNSt8ios_base4InitC1Ev
U _ZNSt8ios_base4InitD1Ev
U _ZSt4cout
U _ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc
0000000000201dd0 V _ZTI1A
0000000000000ec9 V _ZTS1A
0000000000201db0 V _ZTV1A
U _ZTVN10__cxxabiv117__class_type_infoE
Output:
A inited. Address: 0x7efd6cf97070
Start of program.
A called.
A: Address of foo is: 0x7efd6cf97070
A: Address of pod is: 0x7efd6cf97058
A: {1, 2, 3}
Loading B
A inited. Address: 0x7efd6cf97070
B called.
B: Address of foo is: 0x7efd6cf97070
B: Address of pod is: 0x7efd6cf97058
B: {42, 2, 3}
As it can be seen the addresses of the variables collide but the function's address doesn't.
Moreover the C++ initialization are peculiar: aggregates the pod variable are initialized only once you can see that call to foo() modifies it, but when B is loaded it won't reinitialize it, but does call the constructor for the full object when libb.so is loaded.
The key to answering this question is whether the main executable exports the same symbol in its dynamic symbol table. That is, what is the output from:
nm -D a.out | grep ' mangled_name_of_the_symbol'
If the output is empty, the two libraries should indeed use separate (their own) copies of the symbol. But if the output is not empty, then both libraries should reuse the symbol defined in the main binary (this happens because UNIX dynamic linking attempts to emulate what would have happened if everything was statically linked into the main binary -- UNIX support for shared libraries happened long after UNIX itself became popular, and in that context this design decision made sense).
Demonstration:
// main.c
#include <assert.h>
#include <dlfcn.h>
#include <stdio.h>
int foo = 12;
int main()
{
printf("main: &foo = %p, foo = %d\n", &foo, foo);
void *h = dlopen("./foo.so", RTLD_NOW);
assert (h != NULL);
void (*fn)(void) = (void (*)()) dlsym(h, "fn");
fn();
return 0;
}
// foo.c
#include <assert.h>
#include <dlfcn.h>
#include <stdio.h>
int foo = 42;
void fn()
{
printf("foo: &foo = %p, foo = %d\n", &foo, foo);
void *h = dlopen("./bar.so", RTLD_NOW);
assert (h != NULL);
void (*fn)(void) = (void (*)()) dlsym(h, "fn");
fn();
}
// bar.c
#include <stdio.h>
int foo = 24;
void fn()
{
printf("bar: &foo = %p, foo = %d\n", &foo, foo);
}
Build this with:
gcc -fPIC -shared -o foo.so foo.c && gcc -fPIC -shared -o bar.so bar.c &&
gcc main.c -ldl && ./a.out
Output:
main: &foo = 0x5618f1d61048, foo = 12
foo: &foo = 0x7faad6955040, foo = 42
bar: &foo = 0x7faad6950028, foo = 24
Now rebuild just the main binary with -rdynamic (which causes foo to be exported from it): gcc main.c -ldl -rdynamic. The output changes to:
main: &foo = 0x55ced88f1048, foo = 12
foo: &foo = 0x55ced88f1048, foo = 12
bar: &foo = 0x55ced88f1048, foo = 12
P.S.
You can gain much insight into the behavior of dynamic linker by running with:
LD_DEBUG=symbols,bindings ./a.out
Update:
It turns out I asked a wrong question ... Added source example.
If you look at LD_DEBUG output, you'll see:
165089: symbol=object; lookup in file=./main [0]
165089: symbol=object; lookup in file=./liba.so [0]
165089: binding file ./liba.so [0] to ./liba.so [0]: normal symbol `object'
165089: symbol=object; lookup in file=./main [0]
165089: symbol=object; lookup in file=./liba.so [0]
165089: binding file ./libb.so [0] to ./liba.so [0]: normal symbol `object'
What this means: liba.so is in the global search list (by virtue of having been directly linked to by main). This is approximately equivalent to having done dlopen("./liba.so", RTLD_GLOBAL).
It should not be a surprise then that the symbols in it are available for subsequently loaded shared libraries to bind to, which is exactly what the dynamic loader does.
A possible solution for this issue is using the RTLD_DEEPBIND flag of dlopen (however, it's Linux specific, not POSIX standard), which will make the loaded library try to resolve symbols against itself (and its own dependencies) before going through the ones in the global scope.
For this to work properly, the executable has to be built with -fPIE, otherwise some violated ODR assumptions made by libstdc++ will likely cause a segfault (alternatively, if iostream is replaced with cstdio, it works without -fPIE).

why linking to a static library is generally faster than linking to individual source files?

case 1. gcc -o program main.o file1.o file2.o
case 2. ar crv foo.a file1.o file2.o
then, gcc -o program main.o foo.a
At compile time, linking to a static library (case 2) is generally faster than linking to individual source files.(case 1)
Why and what cases?
Any help is appreciated.
/* Filename :lib.h */
void file2(char *);
void file1(int);
main.c
#include "lib.h"
int main()
{
file1(3);
file2("Hello World");
return 0;
}
file1.c
#include <stdio.h>
#include "lib.h"
void file1(int arg) {
printf("you passed %d\n", arg);
} 
file2.c
#include <stdio.h>
#include "lib.h"
void file2(char* arg) {
printf("you passed %s\n", arg);
}
The only case I can think of is when your object files are rather large.
Linking against a library will only "pull in" referenced items. So your executable can be smaller and the functions closer in memory for shorter jumps. But that is an unlikely thing in a modern platform.

Compiling/Linking CUDA and CPP Source Files

I am working through a sample program that uses both C++ source code as well as CUDA. This is the essential content from my four source files.
matrixmul.cu (main CUDA source code):
#include <stdlib.h>
#include <cutil.h>
#include "assist.h"
#include "matrixmul.h"
int main (int argc, char ** argv)
{
...
computeGold(reference, hostM, hostN, Mh, Mw, Nw); //reference to .cpp file
...
}
matrixmul_gold.cpp (C++ source code, single function, no main method):
void computeGold(float * P, const float * M, const float * N, int Mh, int Mw, int Nw)
{
...
}
matrixmul.h (header for matrixmul_gold.cpp file)
#ifndef matrixmul_h
#define matrixmul_h
extern "C"
void computeGold(float * P, const float * M, const float * N, int Mh, int Mw, int Nw);
#endif
assist.h (helper functions)
I am trying to compile and link these files so that they, well, work. So far I can get matrixmul_gold.cpp compiled using:
g++ -c matrixmul_gold.cpp
And I can compile the CUDA source code with out errors using:
nvcc -I/home/sbu/NVIDIA_GPU_Computing_SDK/C/common/inc -L/home/sbu/NVIDIA_GPU_Computing_SDK/C/lib matrixmul.cu -c -lcutil_x86_64
But I just end up with two .O files. I've tried a lot of different ways to link the two .O files but so far it's a no-go. What's the proper approach?
UPDATE: As requested, here is the output of:
nm matrixmul_gold.o matrixmul.o | grep computeGold
nm: 'matrixmul.o': No such file
0000000000000000 T _Z11computeGoldPfPKfS1_iii
I think the 'matrixmul.o' missing error is because I am not actually getting a successful compile when running the suggested compile command:
nvcc -I/home/sbu/NVIDIA_GPU_Computing_SDK/C/common/inc -L/home/sbu/NVIDIA_GPU_Computing_SDK/C/lib -o matrixmul matrixmul.cu matrixmul_gold.o -lcutil_x86_64
UPDATE 2: I was missing an extern "C" from the beginning of matrixmul_gold.cpp. I added that and the suggested compilation command works great. Thank you!
Conventionally you would use whichever compiler you are using to compile the code containing the main subroutine to link the application. In this case you have the main in the .cu, so use nvcc to do the linking. Something like this:
$ g++ -c matrixmul_gold.cpp
$ nvcc -I/home/sbu/NVIDIA_GPU_Computing_SDK/C/common/inc \
-L/home/sbu/NVIDIA_GPU_Computing_SDK/C/lib \
-o matrixmul matrixmul.cu matrixmul_gold.o -lcutil_x86_64
This will link an executable binary called matrimul from matrixmul.cu, matrixmul_gold.o and the cutil library (implicitly nvcc will link the CUDA runtime library and CUDA driver library as well).

How to place `library (RgoogleMaps)` inside the code of Rcpp?

#include "rtest.h"
#include <iostream>
SEXP rcpp_hello_world ()
{
using namespace Rcpp ;
CharacterVector x = CharacterVector::create( "foo", "bar" );
NumericVector y = NumericVector::create( 0.0, 1.0 );
List z = List::create (x, y);
return z;
}
void funcA ()
{
std :: cout << "\nsdfsdfsdf\n";
}
int main () {return 0;}
How to place
library(RgoogleMaps)
and
png (filename="Rg.png", width=480, height=480)
inside the above code?
I run it as: R CMD SHLIB rtest.cpp
> sessionInfo()
R version 2.15.1 (2012-06-22)
Platform: x86_64-unknown-linux-gnu (64-bit)
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=C LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
>
Rcpp's version is 0.9.13
I tried:
R CMD SHLIB -lRgoogleMaps rtest.cpp
It resulted in:
anisha#linux-y3pi:~/> R CMD SHLIB -lRgoogleMaps rtest.cpp
g++ -I/usr/lib64/R/include -DNDEBUG -I/usr/local/include -I/usr/lib64/R/library/Rcpp/include -fpic -fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -c rtest.cpp -o rtest.o
g++ -shared -L/usr/local/lib64 -o rtest.so rtest.o -lRgoogleMaps -L/usr/lib64/R/lib -lR
/usr/lib64/gcc/x86_64-suse-linux/4.5/../../../../x86_64-suse-linux/bin/ld: cannot find -lRgoogleMaps
collect2: ld returned 1 exit status
make: *** [rtest.so] Error 1
I think you have a bit of a conceptual problem here between what
library(RgoogleMaps)
does in R, and what a library is for a compiler
-lfoo -Lpath/to/library
The two are not the same, despite the fact that we use the English noun "library" in both cases.
You may need to brush up a little with a text on programming, compilers, linkers, ...
Why would you want to do this? Rcpp is designed for interfacing C++ code within an R session so that you can exploit faster computation or re-use existing C++ libraries.
Write an R wrapper that calls the Rcpp code, have that wrapper arrange for the package to be made available (using require(RgoogleMaps)).
Second, you don't want to hard code the plotting device. Again, you could do this in R:
png(filename="Rg.png", width=480, height=480)
##
## call Rcpp function
## in here
dev.off()
Rcpp isn't meant for writing standalone C++ applications, you still want to interact with it from an R session.

Unknown symbol when loading a kernel module but the symbol is exported in kernel

I am writing a kernel module to manipute GPIOs. In initialization, the moduel needs to change the muplex of GPIOs.
[tld.c]
#include <mach-omap2/mux.h>
.
.
.
int open_gpio(void){
int result;
result = gpio_request_array(leds_gpios, ARRAY_SIZE(leds_gpios));
if(result != 0) {
printk("tld: cannot request gpio ports\n");
}
result = gpio_direction_output(LED_LE_PIN, 0);
omap_mux_set_gpio(OMAP_MUX_MODE7, LED_LE_PIN);
if(result != 0) {
printk("tld: cannot change GPIO muplex.\n");
gpio_free_array(leds_gpios, ARRAY_SIZE(leds_gpios));
}
return result;
}
static int tld_init(void) {
.
.
/* open gpio ports */
result = open_gpio();
.
.
}
.
module_init(tld_init);
.
Function omap_mux_set_gpio() is declared in Kernel_Source/arch/arm/mach-omap2/mux.h. So I include it in the source file.
The Makefile is also customized.
[Makefile]
CONFIG_CROSS_COMPILE =
CROSS_COMPILE ?= $(CONFIG_CROSS_COMPILE:"%"=%)
ARCH ?= arm
SOURCE_DIR ?= /home/me/kerner_source
AS = $(CROSS_COMPILE)as
LD = $(CROSS_COMPILE)ld
CC = $(CROSS_COMPILE)gcc
CPP = $(CC) -E
AR = $(CROSS_COMPILE)ar
NM = $(CROSS_COMPILE)nm
STRIP = $(CROSS_COMPILE)strip
OBJCOPY = $(CROSS_COMPILE)objcopy
OBJDUMP = $(CROSS_COMPILE)objdump
obj-m += tld.o
ccflags-y += -I$(SOURCE_DIR)/arch/arm
all:
make ARCH=$(ARCH) CROSS_COMPILE=$(CROSS_COMPILE) -C $(SOURCE_DIR) M=$(PWD) modules
clean:
rm *.o *.ko *.mod.*
I can get it compiled but there is warning:
WARNING: "omap_mux_set_gpio" [/home/me/projects/tld.ko] undefined!
When i run insmod, there is error in dmesg:
tld: Unknown symbol omap_mux_set_gpio (err 0)
I have checked the kernel symbol list by runing:
cat /proc/kallsyms | grep omap_mux_set_gpio
An function called omap_mux_set_gpio does exist.
I think the way I include the header file probably is causing the problem. The mux.h belongs to the kernel itself but i treat it like an external one.
What is right way to do it?
Thank you very much.
If a symbol is not marked as EXPORT_SYMBOL or EXPORT_SYMBOL_GPL, then you can not use it in kernel modules. This symbol does not appear to be marked in that way.

Resources