I am currently trying to understand the build process for the Linux kernel. While looking through the Makefiles, I found several rules in the form
scripts_basic:
$(Q)$(MAKE) $(build)=scripts/basic
$(Q)rm -f .tmp_quiet_recordmcount
which all recursively call other make processes and also pass the directory to process. Simultaneously, there seems to be a variable, which is passed, indicating what to do with the subdirectory (the $(build) part.
Looking at the make process, as far as I can see, this always seems to be obj, I cannot find any other value for this variable so far during the make process. Also, I cannot seem to find any place where this variable is set.
So what exactely is this variable for and how is it used (e.g., where is set and processed).
Not exactly. The relevant bit is in scripts/Kbuild.include, where it says
build := -f $(if $(KBUILD_SRC),$(srctree)/)scripts/Makefile.build obj
What this means is that if $(KBUILD_SRC) is not empty, the path to scripts/Makefile.build is given as an absolute path (or at least with a path that can be found from the working directory) by prepending the path to the top of the kernel source tree. As far as I can tell, this is to make the sub-makes all use the same Makefile and avoid having the same make code several dozen times.
Related
I use an sh script to start an application in the background after setting several environment variables. I use temporary variables to start binaries from different places using LD_LIBRARY_PATH and temporary variables. Problem is that the application loads one *.so file from a hardcoded path which I cannot change. Currently I solve this problem manually by replacing the hardcoded file location with a symbolic link.
Can you tell me if there is a clean solution to solve this from the sh script? Basically what I want is that a certain file location is switched with a different binary only for open calls from the application the script starts, for all other processes it should stay the same.
Regards.
Example methods to replace open("/the/original") with open("/some/other"):
One simple method is modifying a pathname within an executable.
First copy the original executable to something like "modified" and run
a utility like bvi (similar to vim).
Consider 2 cases from comparing the length of the new vs. original pathnames:
when (new-length <= original-length), overwrite the original pathname in place
when (new-length > original-length), create a short symlink that references
the new longer pathname like the example below, then overwrite the original pathname
with the short symlink pathname,
ln -s /full/path/some/new/file /shorter
For both cases, remember to include a trailing NULL byte.
After saving changes with bvi, test the newly copied+altered executable.
Once "modified" is working to plan, could also rename for convenience:
mv xyz xyz.orig
mv modified xyz
Another method can serve a transparency requirement.
Create a dynamic library (eg custom.so) with a wrapper routine for
open() which conditionally replaces the pathname and calls the real
libc open(). Run the unmodified original executable (xyz) with another
environment variable, eg:
LD_PRELOAD=/path/to/custom.so xyz
There are some tradeoffs with versatility and modest complexity;
the original xyz is left unchanged and can always be run with/without LD_PRELOAD;
some might consider added overhead from a wrapper as undesirable;
doesn't work with statically-linked or set-uid executables.
Many articles provide instructions for creating a preload library, re-using
an original symbol like open() (frequent example is malloc()), call dlsym() once to find the regular libc open(), saving the result as a function pointer, and calling libc open() indirectly.
everyone,
when I deploy my package to a linux environment, I met this error:
.../Linux-2.6c2.5-i686/Ncurses/Ncurses-15766.0-0/lib/libncurses.so.5 is encountered a second time at /apollo/_env/FBAMerchantAutoRemovalDaemon-swit1na.1755067.237551097.1107633519/perl/lib/perl5.8-dist/File/Find.pm line 542.
though I read the perl script, I have no idea what is wrong. I suspect my environment is tainted. Does anyone have idea what is wrong and how can I debug this problem? Thanks a lot in advance!
Zhe
From perldoc File::Find
follow
Causes symbolic links to be followed. Since directory trees with symbolic links (followed) may contain files more than once and may even have cycles, a hash has to be built up with an entry for each file. This might be expensive both in space and time for a large directory tree. See "follow_fast" and "follow_skip" below. If either follow or follow_fast is in effect:
It is guaranteed that an lstat has been called before the user's wanted() function is called. This enables fast file checks involving _. Note that this guarantee no longer holds if follow or follow_fast are not set.
There is a variable $File::Find::fullname which holds the absolute pathname of the file with all symbolic links resolved. If the link is a dangling symbolic link, then fullname will be set to undef.
So, if, for the purposes of your application, if it is OK to follow symlinks, invoke find with the follow option set:
find({ wanted => \&process, follow => 1 }, $dir);
Or, consider if one of the other follow_skip behaviors is more appropriate for your application:
follow_skip
follow_skip==1, which is the default, causes all files which are neither directories nor symbolic links to be ignored if they are about to be processed a second time. If a directory or a symbolic link are about to be processed a second time, File::Find dies.
follow_skip==0 causes File::Find to die if any file is about to be processed a second time.
follow_skip==2 causes File::Find to ignore any duplicate files and directories but to proceed normally otherwise.
It may be that follow_skip => 2 is more appropriate for your application. Only you can make that decision.
I was wondering, is there any guidlines for identifying symlink related function in an application binary?
Let's take BusyBox for example, /bin/ping is a symlink to /bin/BusyBox.
How can I know identify the ping related functions within the BusyBox binary?
Thanks in advance :)
You can't generally do that.
In case of BusyBox, it checks upon startup which commandline was invoked to execute the binary (including the path to the binary itself). It then calls functions that provide the functionality that was selected based on the basename of the binary / symlink.
Again in case of BusyBox, most of the times the funktion names are closely related to the command name. But this is basically just coincidence: it could well be that someone created an exectable "A" that would call a function "X" when started via a symlink name "B" and function "Y" when called as "C".
I'm trying to set up a build system involving a code generator. The exact files generated are unknown until after the generator is run, but I'd like to be able to run further build steps by pattern matching (run some program on all files with some extension). Is this possible?
Some of the answers here involving code generation seem to assume that the output is known or a listing of generated files is created. This isn't impossible in my case, but I'd like to avoid it since it makes things more complicated.
https://bitbucket.org/scons/scons/wiki/DynamicSourceGenerator seems to indicate that it's possible to add additional targets during Builder actions, but while I could get the build to run and list the generated files, any build steps introduced don't run.
https://bitbucket.org/scons/scons/wiki/NonDeterministicDependencies uses Scanners to add build steps. I put a glob(...) in a scanner, and it succeeds in detecting the generated files, but the files are inexplicably deleted before it actually runs the dependent step.
Is this use case possible? And why is SCons deleting my generated files?
A toy example
source (the file referenced in SConscript)
An example generator, constructs 3 files (not easily known to the build system) and puts them in the argument folder
echo "echo 1" > $1/gen1.txt
echo "echo 2" > $1/gen2.txt
echo "echo 3" > $1/gen3.txt
SConstruct
Just sets up a variant_dir
SConscript('SConscript', variant_dir='build')
SConscript
The goal is for it to:
"Compile" the generator (in this toy example, just copies a file called 'source' and adds execute permissions
Run the "compiled" generator ('source' is a script that generates files)
Perform some operation on each of those generated files by extension. This example just runs the "compile" copy operation on them (for simplicity).
env = Environment()
env.Append(BUILDERS = {'ExampleCompiler' :
Builder(action=[Copy('$TARGET', '$SOURCE'),
Chmod('$TARGET', 0755)])})
generator = env.ExampleCompiler('generator', 'source')
env.Append(BUILDERS = {'GeneratorRun' :
Builder(action=[Mkdir('$TARGET'),
'$SOURCE $TARGET'])})
generated_dir = env.GeneratorRun(Dir('generated'), generator)
Everything's fine up to here, where all the targets are explicitly known to the build system ahead of time.
Attempting to use this block of code to glob over the generated files causes SCons to delete (!!) the generated files:
for generated in generated_dir[0].glob('*.txt'):
generated_run = env.ExampleCompiler(generated.abspath + '.sh', generated)
Attempting to use an action to update the build tree results in additional actions not being run:
def generated_scanner(target, source, env):
for generated in source[0].glob('*.txt'):
print "scanned " + generated.abspath
generated_target = env.ExampleCompiler(generated.abspath + '.sh', generated)
Alias('TopLevelAlias', generated_target)
env.Append(BUILDERS = {'GeneratedOperation' :
Builder(action=[generated_scanner])})
dummy = env.GeneratedOperation(generated_dir[0].File('#dummy'), generated_dir)
Alias('TopLevelAlias', dummy)
The Alias operations are suggested in above dynamic source generator guide, but don't seem to do anything. The prints do execute and indicate that the action gets run.
Running some build pattern on special file extensions is possible with SCons. For C/CPP files this is the preferred scheme, for example:
env = Environment()
env.Program('main', Glob('*.cpp'))
The main task of SCons, as a build system, is to do the minimum amount of work such that all your targets are up-to-date. This makes things complicated for the use case you've described above, because it's not clear how you can reach a "stable" situation where no generated files are added and all targets are built.
You're probably better off by using a simple Python script directly...I really don't see how using SCons (or any other build system for that matter) is mission-critical in this case.
Edit:
At some point you have to tell SCons about the created files (*.txt in your example above), and for tracking all dependencies properly, the list of *.txt files has to be complete. This the task of the Emitter within SCons, which is responsible for returning the list of resulting target and source files for a Builder call. Note, that these files don't have to exist physically during the "parse" phase of SCons. Please also have a look at my answer to Scons: create late targets , which goes into some more detail.
Once you have a proper Emitter in place (see also https://bitbucket.org/scons/scons/wiki/ToolsForFools , "Using Emitters") you should be able to use the Glob('*.txt') call, which will detect and track your created files automatically.
Finally, on our page "Talks and Slides" ( https://bitbucket.org/scons/scons/wiki/TalksAndSlides ) you can find my talk from the PyCon FR.2014, "Why SCons is Not Slow", which explains shortly how SCons works internally. This might be helpful in understanding this problem better and coming up with a full solution.
If I have an executable in /usr/bin and call it while I'm located at ~/dev/wikis/ (that is user#HAL:~/dev/wikis$ the_executable), the ioutil.ReadFile("file.txt") function in the executable will look in /home/user/dev/wikis/file.txt, but is it possible to make it look in /usr/bin/file.txt instead without the user or developer knowing beforehand that the executable will be located in /usr/bin (it could just as well be located in /home/user/dev/my_program/the_executable)?
And then to add a layer of complexity, another situation, say I call the executable from a symbolic link in /usr/bin with the "source" of the executable actually being in /home/user/dev/my_program/the_executable, and I want the program to know about /home/user/dev/my_program/ in this case, dynamically, rather then /usr/bin.
In short: How can an executable dynamically resolve its location on the filesystem or its actual "resting place" vs the working directory of the user (which can easily be gotten through os.Getwd() and which other commands like ioutil.ReadFile use, or use something comparible, for resolving paths).
My best bet is that I have to get the PID of the running program (os.Getpid) and then somehow use that integer to access information about the instance of the program running under that PID, and hopefully that information contains a string of its directory which I can then use.
In Linux (and perhaps in other Unixy systems) you'll find a symbolic link to the actual executable running as pid under /proc/pid/exe, checking that one will give what you want if it is a binary. If it is a script of some sort, that will probably just give the interpreter.
But note that it is perfectly possible to start a process and delete the executable while it is running, leaving either a dangling link or just nothing.
This will depend on the language but as far as I know most programming languages have a way to get either the 'basename' of the program, e.g. helloworld, or the full path, e.g. /usr/bin/helloworld. Often this is provided as the first argument to the program, inserted by the OS/runtime library/interpreter, etc. For example, in C, argv[0] (as we start counting from 0 in C) gives the name of the current program, but the calling process initialises this special argument so the exact format can vary, and in bash, $0 will expand into the path of the script as given when executed (I think). From here: https://gobyexample.com/command-line-arguments, "os.Args provides access to raw command-line arguments. Note that the first value in this slice is the path to the program, and os.Args[1:] holds the arguments to the program." So it appears you don't need to worry about /proc but if you're interested, How do I find the location of the executable in C?
The exported variable os.Args (which is a slice: []string) holds the program arguments, its first element being the executable name with full path.
If the executable is not a symlink, you can use the path or filepath packages to get the folder of the executable like this:
folder := filepath.Dir(os.Args[0])
And you can use os.Readlink() to resolve a symbolic link.
And to test if your executable is a symlink, you can use os.Lstat() which makes no attempt to follow a link (as opposed to os.Stat()).
So your final version should look like this:
s := os.Args[0]
fi, err := os.Lstat(s)
if err != nil {
panic(err) // Failed to get stats
}
// Check if it's a symlink and if so, try to resolve it
if fi.Mode()&os.ModeSymlink != 0 {
if s, err = os.Readlink(s); err != nil {
panic(err) // Failed to resolve symlink
}
}
s = filepath.Dir(s) // We only want the folder part