We are creating a CICD to compile different FPGA IP Cores (imagine different software libraries). All the compilations are based on Makefile and are invoked with a simple make.
The main problem is that our collection is increasing more and more, and we just realize that the job is always the same, changing only the name of the IP Core (or the library). So we copy and paste from the previous job. This is a problem if we need to mantain our .gitlab-cicd.yml in a future.
We came to this example https://icinga.com/blog/2022/03/24/parameterizing-gitlab-ci-cd/ and it is pretty good explained. The main disadvantage is that we use Ubuntu 18.04 and Jsonnet is not supported in this version.
Do you know any alternatives or examples? Maybe with jinja?
Thanks in advance!
Related
I created a Perl script reading information from an XLSX sheet. Since on one machine it worked well, and on another it did not, I included a short debug section:
$sheetdate = ($sheet -> {Cells} [0] [$sheet->{MaxCol}]) -> value();
print "value: $sheetdate\n";
$sheetdate = ($sheet -> {Cells} [0] [$sheet->{MaxCol}]) -> get_format();
print "getformat: $sheetdate\n";
On one machine it printed:
value: 2016-01-18
getformat: yyyy-mm-dd
While on the other:
value: 1-18-16
getformat: m-d-yy
Same script, same worksheet, different results. I believe that something in the environment makes the difference, but I do not know what exactly.
Any hints?
"Same script, same worksheet, different results. I believe that something in the environment makes the difference, but I don not know what exactly."
You sort-of indicate here yourself that you're not really seeking the solution to a perl or XLSX problem so much as some assistance with troubleshooting your environment.
Without access to the environment its difficult to offer a solution per se, but I can say this - you need to;
1) Re-arrange things so that you do get the same result from both environments;
2) Identify a list of differences between the original, problem environment and the one that now "works"; and
3) Modify one thing on the list at a time - moving towards the environment that works - checking each time until it becomes clear what the key variable (not in a programming sense) is.
With regards to (1), take a look at Strawberry Perl. Using Strawberry, its relatively easy to set up what some call Perl on a stick (see Portable ZIP edition) - a complete perl environment on a USB stick. Put your document on the same USB and then try the two environments - this time with absolute certainty of having the same environment. If different results persist, try booting from a "live environement" DVD (linux or widows as appropriate), and then using the USB.
Ultimately, I'd suggest there's something (such as a spreadsheet template ) at play that is different between the environments. You just need to go through a process of elimination to find out what it is.
With the benefit of hindsight, I think its worth revisiting this to produce a succinct answer for those who come across this problem in the future.
The original question was how could a perl script produce two different results when the excel data file fed into it is identical (which was confirmed with MD5 checksums). As programmers, our focus tends to be on the scripts we write and the data that goes into them. What slips to the back of the mind is the myriad of ways that perl itself can be installed and configured.
The three things that should assist in determining where the difference between two installs lie are;
(1) Use strawberry perl on a stick as described above to take the environment out of the equation and thereby (if the problem "disappears") confirm that the problem is something to do with the environment.
(2) Use Data::Dumper liberally throughout to find where the flow of execution "forks."
(3) Compare the output of perl -V (note capital V) to find out if there are differences in how the respective perls were built and configured.
The root cause of the problem was an outdated Spreadsheet::XLSX cpan module installed as RPM from the distribution repository. The RPM included version 0.13 of this module, in CPAN there was already version 0.15, and since version 0.14 the module's behaviour was changed in this particular respect. So once I replaced the pre-compiled module with the version downloaded directly from CPAN and compiled locally, the problem was solved.
I'm working on a project now in which I configure the cabal file to build several executables which share the library built by the same cabal file. The cabal project is structured much like this one, with one library section followed by several executable sections that include this library in their build-depends sections.
I'm using this approach so I can make common functions available to any number of executables, and create more executables easily as needed.
Yet in his Monad Reader article on Hoogle p.33, Neil Mitchell advocates bundling up Haskell projects into a single executable with multiple modes (e.g. by using Neil Mitchell's CmdArgs library.) So there might be one mode to start a web server, another mode to query the database from the command line, etc. Quote:
Provide one executable
Version 3 had four executable programs – one to generate ranking
information, one to do command line searching, one to do web
searching, and one to do regression testing. Version 4 has one
executable, which does all the above and more, controlled by flags.
There are many advantages to providing only one end program – it
reduces the chance of code breaking without noticing it, it makes the
total file size smaller by not duplicating the Haskell run-time system,
it decreases the number of commands users need to learn. The move to
one multipurpose executable seems to be a common theme, which tools
such as darcs and hpc both being based on one command with multiple
modes.
Is a single multimode executable really the better way to go? Are there countervailing reasons to stick with separate executables sharing the same library?
Personally, I'm more of a fan of the Unix philosophy "write programs that do one thing and do it well". However there are reasons for doing either way, so the only reasonable answer here is: it depends.
One example where it makes senses to bundle everything into same executable, is when you're targeting a platform that is very limited on resources (e.g, embedded system). This is the approach taken by BusyBox.
On the other hand if you divide into multiple executables, you give your clients the option of just using those that matter to them. With a single executable, even if your client really just wanted one functionality, he'll have no way to get rid of the extra baggage.
I'm sure there are a lot of more reasons for going either way, but this just goes to show that there's no definitive answer. It depends on the use case.
In my open-source project Artha I use libnotify for showing passive desktop notifications to the user.
Instead of statically linking libnotify, a lookup at runtime is made for the shared object (.so) file via dlload, if available on the target machine, Artha exposes the notification feature in it's GUI. On app. start, a call to dlload with filename param as libnotify.so.1 is made and if it returns a non-null pointer, then the feature is exposed.
A recurring problem with this model is that every time the version number of the library is bumped, Artha's code needs to be updated, currently libnotify.so.4 is the latest to entail such an occurance.
Is there a linux system call (irrespective of the distro the app. is running on), which can tell me if a particular library's shared object is available at runtime? I know that there exists the bruteforce option of enumerating the library by going from 1 to say 10, I find the solution ugly and inelegant.
Also, if this can be addressed via autoconf, then that solution is welcome too I.e. at build time, based on the target machine, the configure.h generated should've the right .so name that can be passed to dlload.
P.S.: I think good distros follow the style of creating links to libnotify.so.x so that a programmer can just do dlload("libnotify.so", RTLD_LAZY) and the right version numbered .so is loaded; unfortunately not all distros follow this, including Ubuntu.
The answer is: you don't.
dlopen() is not designed to deal with things like that, and trying to load whichever soversion you find on the system just because it happens to have the symbols you need is not a good way to do it.
Different sonames have different ABIs, and different ABIs means that you may be calling the same exact symbol name that is expecting a different set (or different size) of parameters, which will cause crashes or misbehaviour that are extremely difficult do debug.
You should have a read on how shared object versions work and what an ABI is.
The libfoo.so link is there for the link editor (ld) and is usually installed with the -devel packages for that reason; it might also very well not be a link but rather a text file with a linker script, often times on purpose to avoid exactly what you're trying to do.
I wonder if we can a reduce just a little bit the effort around packages
under linux/unix OS environments and software installations.
It is my stance that there is too much redundant effort about $subject.
I have been pondering about ways to connect build systems of $subject
with some next "stage build tools", like: easybuild (1) & openbuildservice (2);
read below for more details.
To be more specific, I was able last week to take pkgsrc's repository,
process the Makefiles via a tiny "pkg2eb" script and produce *.eb files
for easybuild, then fed many parallel gcc compilations with them.
That "blindly-driven process" ended up in >600 successful builds,
ie. these were packages that simply needed 'wget/configure/make/make install';
It's not bad for a first run, just wonder if it can be done any better.
So:
According to your experience, which OS has the cleanest/leanest
pkgsrc/port structure to be sourced & fed to other external tools?
This is NOT the same as which has the most available packages!
Have you heard of any similar efforts trying to massively produce
packages from eg. a common source list in a structured manner?
(I mean, in a transferable way across different build systems)
So,
much relevant information is visible here:
http://www.mancoosi.org/edos/packages/ # lengthy description of various packaging formats
this one shows the higher level picture:
http://www.mancoosi.org/edos/suggestions/ (esp. 2.1.1 Expressivity shortcomings)
Anyway, to answer to original question, the best bets as of now are:
RPM's .spec files
DEB control files
pkgsrc; possible but some hackery is still needed
portage; quite clean, distinguishes between DEPEND and RDEPEND
macports; easy to parse; very detailed dependencies aspects
ports; like pkgsrc; multiple dependencies defined
I have been looking into MeeGo, maemo, Android architecture.
They all have Linux Kernel, build some libraries on it, then build middle layer libraries [e.g telephony, media etc...].
Suppose i wana build my own system, say Linux Kernel, with some binariers like glibc, Dbus,.... UI toolkit like GTK+ and its binaries.
I want to compile every project from source to customize my own linux system for desktop, netbook and handheld devices. [starting from netbook first :)]
How can i build my own customize system from kernel to UI.
I apologize in advance for a very long winded answer to what you thought would be a very simple question. Unfortunately, piecing together an entire operating system from many different bits in a coherent and unified manner is not exactly a trivial task. I'm currently working on my own Xen based distribution, I'll share my experience thus far (beyond Linux From Scratch):
1 - Decide on a scope and stick to it
If you have any hope of actually completing this project, you need write an explanation of what your new OS will be and do once its completed in a single paragraph. Print that out and tape it to your wall, directly in front of you. Read it, chant it, practice saying it backwards and whatever else may help you to keep it directly in front of any urge to succumb to feature creep.
2 - Decide on a package manager
This may be the single most important decision that you will make. You need to decide how you will maintain your operating system in regards to updates and new releases, even if you are the only subscriber. Anyone, including you who uses the new OS will surely find a need to install something that was not included in the base distribution. Even if you are pushing out an OS to power a kiosk, its critical for all deployments to keep themselves up to date in a sane and consistent manner.
I ended up going with apt-rpm because it offered the flexibility of the popular .rpm package format while leveraging apt's known sanity when it comes to dependencies. You may prefer using yum, apt with .deb packages, slackware style .tgz packages or your own format.
Decide on this quickly, because its going to dictate how you structure your build. Keep track of dependencies in each component so that its easy to roll packages later.
3 - Re-read your scope then configure your kernel
Avoid the kitchen sink syndrome when making a kernel. Look at what you want to accomplish and then decide what the kernel has to support. You will probably want full gadget support, compatibility with file systems from other popular operating systems, security hooks appropriate for people who do a lot of browsing, etc. You don't need to support crazy RAID configurations, advanced netfilter targets and minixfs, but wifi better work. You don't need 10GBE or infiniband support. Go through the kernel configuration carefully. If you can't justify including a module by its potential use, don't check it.
Avoid pulling in out of tree patches unless you absolutely need them. From time to time, people come up with new scheduling algorithms, experimental file systems, etc. It is very, very difficult to maintain a kernel that consumes from anything else but mainline.
There are exceptions, of course. If going out of tree is the only way to meet one of your goals stated in your scope. Just remain conscious of how much additional work you'll be making for yourself in the future.
4 - Re-read your scope then select your base userland
At the very minimum, you'll need a shell, the core utilities and an editor that works without an window manager. Paying attention to dependencies will tell you that you also need a C library and whatever else is needed to make the base commands work. As Eli answered, Linux From Scratch is a good resource to check. I also strongly suggest looking at the LSB (Linux standard base), this is a specification that lists common packages and components that are 'expected' to be included with any distribution. Don't follow the LSB as a standard, compare its suggestions against your scope. If the purpose of your OS does not necessitate inclusion of something and nothing you install will depend on it, don't include it.
5 - Re-read your scope and decide on a window system
Again, referring to the everything including the kitchen sink syndrome, try and resist the urge to just slap a stock install of KDE or GNOME on top of your base OS and call it done. Another common pitfall is to install a full blown version of either and work backwards by removing things that aren't needed. For the sake of sane dependencies, its really better to work on this from bottom up rather than top down.
Decide quickly on the UI toolkit that your distribution is going to favor and get it (with supporting libraries) in place. Define consistency in UIs quickly and stick to it. Nothing is more annoying than having 10 windows open that behave completely differently as far as controls go. When I see this, I diagnose the OS with multiple personality disorder and want to medicate its developer. There was just an uproar regarding Ubuntu moving window controls around, and they were doing it consistently .. the inconsistency was the behavior changing between versions. People get very upset if they can't immediately find a button or have to increase their mouse mileage.
6 - Re-read your scope and pick your applications
Avoid kitchen sink syndrome here as well. Choose your applications not only based on your scope and their popularity, but how easy they will be for you to maintain. Its very likely that you will be applying your own patches to them (even simple ones like messengers updating a blinking light on the toolbar).
Its important to keep every architecture that you want to support in mind as you select what you want to include. For instance, if Valgrind is your best friend, be aware that you won't be able to use it to debug issues on certain ARM platforms.
Pretend you are a company and will be an employee there. Does your company pass the Joel test? Consider a continuous integration system like Hudson, as well. It will save you lots of hair pulling as you progress.
As you begin unifying all of these components, you'll naturally be establishing your own SDK. Document it as you go, avoid breaking it on a whim (refer to your scope, always). Its perfectly acceptable to just let linux be linux, which turns your SDK more into formal guidelines than anything else.
In my case, I'm rather fortunate to be working on something that is designed strictly as a server OS. I don't have to deal with desktop caveats and I don't envy anyone who does.
7 - Additional suggestions
These are in random order, but noting them might save you some time:
Maintain patch sets to every line of upstream code that you modify, in numbered sequence. An example might be 00-make-bash-clairvoyant.patch, this allows you to maintain patches instead of entire forked repositories of upstream code. You'll thank yourself for this later.
If a component has a testing suite, make sure you add tests for anything that you introduce. Its easy to just say "great, it works!" and leave it at that, keep in mind that you'll likely be adding even more later, which may break what you added previously.
Use whatever version control system is in use by the authors when pulling in upstream code. This makes merging of new code much, much simpler and shaves hours off of re-basing your patches.
Even if you think upstream authors won't be interested in your changes, at least alert them to the fact that they exist. Coordination is essential, even if you simply learn that a feature you just put in is already in planning and will be implemented differently in the future.
You may be convinced that you will be the only person to ever use your OS. Design it as though millions will use it, you never know. This kind of thinking helps avoid kludges.
Don't pull upstream alpha code, no matter what the temptation may be. Red Hat tried that, it did not work out well. Stick to stable releases unless you are pulling in bug fixes. Major bug fixes usually result in upstream releases, so make sure you watch and coordinate.
Remember that it's supposed to be fun.
Finally, realize that rolling an entire from-scratch distribution is exponentially more complex than forking an existing distribution and simply adding whatever you feel that it lacks. You need to reward yourself often by booting your OS and actually using it productively. If you get too frustrated, consistently confused or find yourself putting off work on it, consider making a lightweight fork of Debian or Ubuntu. You can then go back and duplicate it entirely from scratch. Its no different than prototyping an application in a simpler / rapid language first before writing it for real in something more difficult. If you want to go this route (first), gNewSense offers utilities to fork your own OS directly from Ubuntu. Note, by default, their utilities will strip any non free bits (including binary kernel blobs) from the resulting distro.
I strongly suggest going the completely from scratch route (first) because the experience that you will gain is far greater than making yet another fork. However, its also important that you actually complete your project. Best is subjective, do what works for you.
Good luck on your project, see you on distrowatch.
Check out Linux From Scratch:
Linux From Scratch (LFS) is a project
that provides you with step-by-step
instructions for building your own
customized Linux system entirely from
source.
Use Gentoo Linux. It is a compile from source distribution, very customizable. I like it a lot.