We have a fairly large codebase worked on by two dozen programmers, across linux, windows and mac. We are adding some opencl code (initially targeting intel haswell) and are looking for ideas for how to do this with minimum disruption to the team. How can we arrange things so they can all happily compile against the correct opencl headers and libraries? In other words, what can we safely assume about their environments to allow us to set up the codebase to work out of the box for everyone?
There are quite a few differences between OpenCL installations on different OSs. The best way to handle it is to start with a cross-platform build system - I personally use CMake. The newest versions of CMake include a module for finding the headers and libraries, so in theory you don't need to do anything special. However, in my experience, the module is not written well enough to support all possible SDKs and OSs. I've had to add more locations to look for the relevant files.
Once you have the headers and libraries located, you'll need to include the opencl.lib to your build, which you should be able to do in a platform-independed manner through your build system of choice.
The final part is how to include the headers in your code. Basically, there is a difference between where the cl.h is on Windows and Linux vs OS X. If you look at any OpenCL example, you will see what I mean. You'll need to have some #ifdefs around that, which I recommend you isolate to your own include file.
You will also need to decide how to handle your kernels. For any decent size kernel, you will want to keep the source in a separate file. I keep my kernels in .cl files and then I use the STRINGIFY macro to include the source directly in my CPP code into a string variable.
I hope this gives you enough to start with.
There can be many things to do but I am writing down a standard one:
Create a offline standard portable intermediate representation(SPIR) of opencl kernels across different OS and distribute. The main goal of SPIR was to enable application developers to avoid shipping their kernels in a source form, while maintaining portability between vendors and devices.
Refer this for more detail:
https://software.intel.com/en-us/articles/using-spir-for-fun-and-profit-with-intel-opencl-code-builder
Related
I'm building a .a from C++ code. It only depends on the standard library (libc++/libstdc++). From general reading, it seems that portability of binaries depends on
compiler version (because it can affect the ABI). For gcc, the ABI is linked to the major version number.
libc++/libstdc++ versions (because they could pass a vector<T> into the .a and its representation could change).
I.e. someone using the .a needs to use the same (major version of) the compiler + same standard library.
As far as I can see, if compiler and standard library match, a .a should work across multiple distros. Is this right? Or is there gubbins relating to system calls, etc., meaning a .a for Ubuntu should be built on Ubuntu, .a for CentOS should be built on CentOS, and so on?
Edit: see If clang++ and g++ are ABI incompatible, what is used for shared libraries in binary? (though it doens't answer this q.)
Edit 2: I am not accessing any OS features explicitly (e.g. via system calls). My only interaction with the system is to open files and read from them.
It only depends on the standard library
It could also depend implicitly upon other things (think of resources like fonts, configuration files under /etc/, header files under /usr/include/, availability of /proc/, of /sys/, external programs run by system(3) or execvp(3), specific file systems or devices, particular ioctl-s, available or required plugins, etc...)
These are kind of details which might make the porting difficult. For example look into nsswitch.conf(5).
The evil is in the details.
(in other words, without a lot more details, your question don't have much sense)
Linux is perceived as a free software ecosystem. The usual way of porting something is to recompile it on -or at least for- the target Linux distribution. When you do that several times (for different and many Linux distros), you'll understand what details are significant in your particular software (and distributions).
Most of the time, recompiling and porting a library on a different distribution is really easy. Sometimes, it might be hard.
For shared libraries, reading Program Library HowTo, C++ dlopen miniHowTo, elf(5), your ABI specification (see here for some incomplete list), Drepper's How To Write Shared Libraries could be useful.
My recommendation is to prepare binary packages for various common Linux distributions. For example, a .deb for Debian & Ubuntu (some particular versions of them).
Of course a .deb for Debian might not work on Ubuntu (sometimes it does).
Look also into things like autoconf (or cmake). You may want at least to have some externally provided #define-d preprocessor strings (often passed by -D to gcc or g++) which would vary from one distribution to the next (e.g. on some distributions, you print by popen-ing lp, on others, by popen-ing lpr, on others by interacting with some CUPS server etc...). Details matter.
My only interaction with the system is to open files
But even these vary a lot from one distribution to another one.
It is probable that you won't be able to provide a single -and the same one- lib*.a for several distributions.
NB: you probably need to budget more work than what you believe.
I want to build from source a recent Linux kernel (e.g. 4.13.4 in end of september 2017) on my Debian/Sid/x86-64 with all (or most) currently loaded modules configured as hard-built in the new kernel.
(I believe that I have read something like this somewhere, but can't remember where and can't find it)
It would be some make configfromloadedmodules (but of course it is not exactly configfromloadedmodules makefile target, but some other target that I did not easily find).
That is, for most (ideally all) currently loaded modules (as given by lsmod) it would answer Y (not m) for each of them at make config time and give me some good enough .config; but I don't want a bloated kernel with all drivers -even those that I don't use and which are not currently loaded- built in.
Does that exist, or was what I have probably read some wish or some non standard feature of experimental kernels?
This would avoid any initrd thing and give me a kernel suited for my hardware and habits.
The current kernel is a standard Debian one 4.12.0-2-amd64, so I have its /boot/config-4.12.0-2-amd64 (so I want to automatize the replacement of CONFIGxxx=m with CONFIGxxx=y there, according to the currently loaded modules, e.g. as given by lsmod).
See also this answer; I still believe that device trees are not essential to Linux, but they are a useful convenience.
A near variant of my question is how to easily configure a kernel, suited to my computer and hardware and set-up, without ìnitrd, without any modules (e.g. with CONFIG_MODULES=n) and without (or with very few) useless drivers, which would work as nicely as my current Debian kernel.
I believe, you should read about "make localmodconfig" and "make localyesconfig" and use one as per your requirement.
This, This and This are helpful links.
I'm looking for the best way to develop and package different variants of a library with different compile settings but for the same ABI and then selecting the best fit at runtime. In more concrete terms, I'd like a NEON and non-NEON armeabi-v7a build.
The native library has a public C interface that third parties link to. They seem to need to link to one of the variants to prevent link errors, but I'd like to load the alternative variant at runtime if it's a better fit for the device, and have the runtime loader do the correct relocations.
From what I see so far it seems I need to give both variants the same file name, so need to put them in different folders. Subfolders under the abi folder don't seem to get copied by the package installation process so that approach doesn't work. The best suggestion I've seen so far is to manually copy one variant from the res folder to a known device path and to call System.loadLibrary() with a full path. Reference: https://groups.google.com/forum/#!topic/android-ndk/zu_dmcmUlMo
Is this still the best/recommended approach?
How will this interact with the binary translation done on non-arm devices? (Although I can supply an x86 build, some third parties may leave it out of their apk).
I'm assuming cpufeatures on a device using binary translation will not report the cpu family as ARM, so my proposed solution would be to build a standard armeabi-v7a library in the normal way (which I guess will get binary translated), and ship a NEON-supporting library in res/raw. Then at runtime if cpufeatures reports an ARM CPU with NEON support then copy out that library and call loadLibrary with the full path. Can anyone see any problems with that approach?
If you explicitly want to have two different builds of a lib, then yes, it's probably the best compromise.
First off - do note that many libraries that can use NEON can be built with those parts runtime-enabled so that you can have a normal ARMv7 build which doesn't strictly require NEON but can enable those codepaths at runtime if detected - e.g. libav/FFmpeg do that, and the same goes for many other similar libraries. This allows you to have one single ARMv7 binary that fully utilizes NEON where applicable, while still works on the few ARMv7 devices without NEON.
If you're trying to use compiler autovectorization, or if this is a library where the NEON routines aren't easily confined to restricted parts that are enabled at runtime (or hoping to gain extra performance by building the whole library with NEON enabled), your approach sounds sane.
Keep in mind that you want to have at least one native library that is packaged "normally" (which you seem to have, but which has been an issue in e.g. https://stackoverflow.com/a/29329413/3115956). On installation, the installer picks the best match of the bundled architectures and only extracts the libs from that one, and runs the process in that mode. On devices with multiple ABIs (32 and 64 bit), this is essential since if the process is started in a different mode it's too late to switch mode once you try to load a library in a different form.
On an x86 device that emulates ARM binaries, at least the cpufeatures library will return ARM if the process is running in ARM mode. If you use system properties to find the primary and secondary ABIs, you won't know which of them the current process is using though.
EDIT: x86 devices with binary translation actually seem to be able to load an armeabi library even if the same process already has loaded some bundled x86 libraries as well. So apparently this translation is done on a per library basis, not like 32 vs 64 bit, where a certain mode is chosen for the process at startup, which excludes loading any libraries of the other variant.
I understand the benefits of dynamic linking (old code can take advantage of library upgrades automatically, it's more space efficient), but it definitely has downsides, especially in the heterogeneous Linux ecosystem. It makes it difficult to distribute a distribution-agnostic binary that "just works" and makes a previously working program more likely to break due to a system upgrade that breaks backwards compatibility or introduces regressions into a shared library.
Given these disadvantages, why does dynamic linking seem to be so universally preferred? Why is it so hard to find statically linked, distribution-agnostic Linux binaries, even for small applications?
There are three big reasons:
GNU libc doesn't support static linkage to itself, because it makes extensive internal use of dlopen. (This means that static linkage to anything else is less worthwhile, because you can't get a totally static binary without replacing the C library.)
Distributions don't support static linkage to anything else, because it increases the amount of work they have to do when a library has a security vulnerability.
Distributions have no interest whatsoever in distribution-agnostic binaries. They want to get the source and build it themselves.
You should also keep in mind that the Linux-not-Android software ecology is entirely source-based. If you are shipping binaries and you're not a distribution vendor you are Doing It Wrong.
There are several reasons we prefer dynamic linkage:
Licensing. This is a particular issue with the LGPL, though there are other licenses with similar strictures.
Basically, it's legal for me to send you a binary built against LGPL libfoo.so.*, and even to give you a binary for that library. I have a various responsibilities, such as responding to requests for the source for the LGPL'd library, but the important thing here is that I don't have to give you the source for my program, too. Since glibc is LGPL and almost every binary on a Linux box is linked to it, that alone will force dynamic linkage by default.
Bandwidth costs. People like to say bandwidth is free, but that's true only in principle. In many practical cases, bandwidth still matters.
My company's main C++-based system packs up into a ~4 MB RPM, which takes a few minutes to upload over the slow DSL uplinks at most of our customers' sites. We still have some customers only accessible via modem, too, and for those an upload is a matter of "start it, then go to lunch." If we were shipping static binaries, these packages would be much larger. Our system is composed of several cooperating programs, most of which are linked to the same set of dynamic libraries, so the RPM would contain redundant copies of the same shared code. Compression can squeeze some of that out, but why keep shipping it again and again for each upgrade?
Management. Many of the libraries we link against are part of the OS distro, so we get free updates to those libraries independent from our program. We don't have to manage it.
We do separately ship some libraries which aren't part of the OS, but they have to change much less often than our code does. Typically, these are installed on the system when we build the server, then never updated again. This is because we are most often more interested in stability than new features from these libraries. As long as they're working, we don't touch them.
I have been looking into MeeGo, maemo, Android architecture.
They all have Linux Kernel, build some libraries on it, then build middle layer libraries [e.g telephony, media etc...].
Suppose i wana build my own system, say Linux Kernel, with some binariers like glibc, Dbus,.... UI toolkit like GTK+ and its binaries.
I want to compile every project from source to customize my own linux system for desktop, netbook and handheld devices. [starting from netbook first :)]
How can i build my own customize system from kernel to UI.
I apologize in advance for a very long winded answer to what you thought would be a very simple question. Unfortunately, piecing together an entire operating system from many different bits in a coherent and unified manner is not exactly a trivial task. I'm currently working on my own Xen based distribution, I'll share my experience thus far (beyond Linux From Scratch):
1 - Decide on a scope and stick to it
If you have any hope of actually completing this project, you need write an explanation of what your new OS will be and do once its completed in a single paragraph. Print that out and tape it to your wall, directly in front of you. Read it, chant it, practice saying it backwards and whatever else may help you to keep it directly in front of any urge to succumb to feature creep.
2 - Decide on a package manager
This may be the single most important decision that you will make. You need to decide how you will maintain your operating system in regards to updates and new releases, even if you are the only subscriber. Anyone, including you who uses the new OS will surely find a need to install something that was not included in the base distribution. Even if you are pushing out an OS to power a kiosk, its critical for all deployments to keep themselves up to date in a sane and consistent manner.
I ended up going with apt-rpm because it offered the flexibility of the popular .rpm package format while leveraging apt's known sanity when it comes to dependencies. You may prefer using yum, apt with .deb packages, slackware style .tgz packages or your own format.
Decide on this quickly, because its going to dictate how you structure your build. Keep track of dependencies in each component so that its easy to roll packages later.
3 - Re-read your scope then configure your kernel
Avoid the kitchen sink syndrome when making a kernel. Look at what you want to accomplish and then decide what the kernel has to support. You will probably want full gadget support, compatibility with file systems from other popular operating systems, security hooks appropriate for people who do a lot of browsing, etc. You don't need to support crazy RAID configurations, advanced netfilter targets and minixfs, but wifi better work. You don't need 10GBE or infiniband support. Go through the kernel configuration carefully. If you can't justify including a module by its potential use, don't check it.
Avoid pulling in out of tree patches unless you absolutely need them. From time to time, people come up with new scheduling algorithms, experimental file systems, etc. It is very, very difficult to maintain a kernel that consumes from anything else but mainline.
There are exceptions, of course. If going out of tree is the only way to meet one of your goals stated in your scope. Just remain conscious of how much additional work you'll be making for yourself in the future.
4 - Re-read your scope then select your base userland
At the very minimum, you'll need a shell, the core utilities and an editor that works without an window manager. Paying attention to dependencies will tell you that you also need a C library and whatever else is needed to make the base commands work. As Eli answered, Linux From Scratch is a good resource to check. I also strongly suggest looking at the LSB (Linux standard base), this is a specification that lists common packages and components that are 'expected' to be included with any distribution. Don't follow the LSB as a standard, compare its suggestions against your scope. If the purpose of your OS does not necessitate inclusion of something and nothing you install will depend on it, don't include it.
5 - Re-read your scope and decide on a window system
Again, referring to the everything including the kitchen sink syndrome, try and resist the urge to just slap a stock install of KDE or GNOME on top of your base OS and call it done. Another common pitfall is to install a full blown version of either and work backwards by removing things that aren't needed. For the sake of sane dependencies, its really better to work on this from bottom up rather than top down.
Decide quickly on the UI toolkit that your distribution is going to favor and get it (with supporting libraries) in place. Define consistency in UIs quickly and stick to it. Nothing is more annoying than having 10 windows open that behave completely differently as far as controls go. When I see this, I diagnose the OS with multiple personality disorder and want to medicate its developer. There was just an uproar regarding Ubuntu moving window controls around, and they were doing it consistently .. the inconsistency was the behavior changing between versions. People get very upset if they can't immediately find a button or have to increase their mouse mileage.
6 - Re-read your scope and pick your applications
Avoid kitchen sink syndrome here as well. Choose your applications not only based on your scope and their popularity, but how easy they will be for you to maintain. Its very likely that you will be applying your own patches to them (even simple ones like messengers updating a blinking light on the toolbar).
Its important to keep every architecture that you want to support in mind as you select what you want to include. For instance, if Valgrind is your best friend, be aware that you won't be able to use it to debug issues on certain ARM platforms.
Pretend you are a company and will be an employee there. Does your company pass the Joel test? Consider a continuous integration system like Hudson, as well. It will save you lots of hair pulling as you progress.
As you begin unifying all of these components, you'll naturally be establishing your own SDK. Document it as you go, avoid breaking it on a whim (refer to your scope, always). Its perfectly acceptable to just let linux be linux, which turns your SDK more into formal guidelines than anything else.
In my case, I'm rather fortunate to be working on something that is designed strictly as a server OS. I don't have to deal with desktop caveats and I don't envy anyone who does.
7 - Additional suggestions
These are in random order, but noting them might save you some time:
Maintain patch sets to every line of upstream code that you modify, in numbered sequence. An example might be 00-make-bash-clairvoyant.patch, this allows you to maintain patches instead of entire forked repositories of upstream code. You'll thank yourself for this later.
If a component has a testing suite, make sure you add tests for anything that you introduce. Its easy to just say "great, it works!" and leave it at that, keep in mind that you'll likely be adding even more later, which may break what you added previously.
Use whatever version control system is in use by the authors when pulling in upstream code. This makes merging of new code much, much simpler and shaves hours off of re-basing your patches.
Even if you think upstream authors won't be interested in your changes, at least alert them to the fact that they exist. Coordination is essential, even if you simply learn that a feature you just put in is already in planning and will be implemented differently in the future.
You may be convinced that you will be the only person to ever use your OS. Design it as though millions will use it, you never know. This kind of thinking helps avoid kludges.
Don't pull upstream alpha code, no matter what the temptation may be. Red Hat tried that, it did not work out well. Stick to stable releases unless you are pulling in bug fixes. Major bug fixes usually result in upstream releases, so make sure you watch and coordinate.
Remember that it's supposed to be fun.
Finally, realize that rolling an entire from-scratch distribution is exponentially more complex than forking an existing distribution and simply adding whatever you feel that it lacks. You need to reward yourself often by booting your OS and actually using it productively. If you get too frustrated, consistently confused or find yourself putting off work on it, consider making a lightweight fork of Debian or Ubuntu. You can then go back and duplicate it entirely from scratch. Its no different than prototyping an application in a simpler / rapid language first before writing it for real in something more difficult. If you want to go this route (first), gNewSense offers utilities to fork your own OS directly from Ubuntu. Note, by default, their utilities will strip any non free bits (including binary kernel blobs) from the resulting distro.
I strongly suggest going the completely from scratch route (first) because the experience that you will gain is far greater than making yet another fork. However, its also important that you actually complete your project. Best is subjective, do what works for you.
Good luck on your project, see you on distrowatch.
Check out Linux From Scratch:
Linux From Scratch (LFS) is a project
that provides you with step-by-step
instructions for building your own
customized Linux system entirely from
source.
Use Gentoo Linux. It is a compile from source distribution, very customizable. I like it a lot.