I'm learining Rust and want to build app that will run on another server, however, server's target is the same as local/CI.
I found there is a way to specify target-cpu property for compilier.
I tried to build for target-cpu=skylake (local) and for target-cpu=skylake-avx512 (server) and noticed slight difference in output binary size.
Will building for a specific CPU improve performance or change application behaviour?
Related
I'm relatively new to linux and building binaries using CMAKE.
In Windows, I'm accustomed to seeing small binaries with the debug info stored in an external pdb. However, in linux I asked this question Is there a way to determine what code is bloating a linux shared object? and it was noted that the build option -g included the DebugInfo directly into the binary which caused the code bloat.
At this point, I have two concerns:
Performance
Luckily, after reading this online, RelWithDebInfo performance is equivalent to Release mode:
Code build with RelwithDebInfo mode should not have it’s performance
degraded in comparison to Release mode, as the symbol table and debug
metadata that are generated do not live in the executable code
section, so they should not affect the performance when the code is
run without a debugger attached.
Memory
Some of the linux binaries created with RelWithDebInfo are ~100MB, so in a Google Cloud VM environment, how does running many of these binaries affect memory usage/cost? I google'd but didn't find any definitive answers.
Question
Can RelWithDebInfo negatively affect memory usage/cost, and if so, are there methods to avoid this, but still retain the DebugInfo?
I'm working on a HTTP/1.1 server in C as a learning experience and want to make it performant while still being dynamic. Performing a get or post on static files or scripts was easy enough, but I'd like to add the ability to call compiled binaries for greater speed.
Currently, I link these compiled binaries directly into the server binary, but I'd like to be able to update and hot swap them. I considered dynamically linking them as shared libraries, but I don't want to relink them to handle every request. I also considered creating a new process to run them, however that incurs significant overhead every request and makes getting the response back to the client difficult (I'm using OpenSSL sockets).
How could I efficiently relink these compiled binaries when they update, without shutting down the server?
I'm testing on Debian Sid and running on an AWS ECS instance with CentOS 7. Both have Linux kernel versions 4.19+
I'd like to be able to update and hot swap them. I considered dynamically linking them as shared libraries
You appear to believe that you can update a shared library (on disk) that a running server binary is currently using, and expect that running server process to start using the updated library.
That is not how shared libraries work. If you try that, your server process will either crash, or continue using the old library (depending on exactly how you update the library on disk).
This can be made to work in limited circumstances if you use dlopen to load the library, and if you can quiesce your server, and have it dlclose to unload the previously loaded version, and then dlopen updated version. But the exact details of making this work are quite tricky.
How can I use puppet to build from source without using multiple Exec commands?. Do we have modules for it on forge that I could use?
It's possible to use Puppet to build applications from source without using execs, possibly with a custom written type and provider. Otherwise, yes, it'd have to be a few different exec resources with onlyif, creates etc. statements to stop them running every time the agent ran.
Puppet's model of configuration management is known as a desired state model: you define the end state of the system and let the system. This is why exec's are generally avoided in Puppet: they don't fit a desired state model. It also makes things like updating the application, or dealing with unknowns like a partial failure of the compilation that creates a required file.
In my opinion, I would not recommend using configuration management to build applications from source at all. There are a few issue inherent with doing so (this is not just for Puppet, but most config management languages):
Slower runs, as running the compilation can be longer and detecting that it's complete is normally a slightly trickier tasks
Issues with half complete state or failure: if the compilation breaks halfway through it's both harder to detect and resolve
Making the compilation idempotent: You have to wrap the command in logic that detects if the installation has already been done. However, this is difficult, as things like the detection of a flag file or particular binary could occur even when the compilation ends in failure
Upgrading or changing: There's no easy way to upgrade or change the application. A package would be easier to do this with.
This sounds like something that would be better served by packaging, using tools such as FPM or just native package building tools such as rpmbuild.
For our Travis-CI builds of the Jailhouse hypervisor, we have a rather costly environment setup which consists of a partial distribution update to pull in a recent make version (>=3.82, the default one is still only 3.81 - Ubuntu...), a cross toolchain for ARM and a 100 MB package of prebuilt Linux kernel sources that we need to compile an out-of-tree module.
To reduce the build time and the number of kernel downloads, we currently build all configuration variants sequentially in a single run (make; make clean; make...). That was fine for checking for build breakages, but with the addition of a Coverity scan, which depends on the build outputs, it no longer works. Switching to a build matrix seems the obvious solution, at the price of multiple installations because Travis-CI seems to be unable to reuse them during such builds. While we currently only have 3 configuration variants, this will increase in the future (e.g. every ARM board added will increase it by one), thus the approach does not really scale.
Do we have any alternatives? I already looked at caching, available via the docker-based build, but lacking sudo support there prevents this approach. Other ideas?
You should change your build to do this
cov-build --idir <target1> make; make clean
...
Use different intermediate directories for each build. Then go back later and run
cov-analyze --idir <target1>
cov-commit-defects --idir <target1> --stream <target1>
For the first time, I am writing a web service that will call upon external programs to process requests in batch. The front-end will accept file uploads and then place them in a queue. The workers on the backend will take that file, run it through ffmpeg and the rest of my pipeline, and send an email when the process is complete.
I have my backend process working on my computer (Ubuntu 10.04). The question is: should I try to re-create that pipeline using binaries that I've compiled from scratch? Or is it okay to use apt when configuring in The Real World?
Not all hosting services uses Ubuntu, and not all give me root access. (I haven't chosen a host yet.) However, they will let me upload binaries to execute, and many give me shell access with gcc.
Usually this would be a no-brainier and I'd compile it all from scratch. But doing so - not to mention trying to figure out how to create a platform-independent .tar.gz binary - will be quite a task which ultimately doesn't really help me ship my product.
Do you have any thoughts on the best way to set up my stack so that I'm not tied to a specific hosting provider? Should I try creating my own .deb, which contains Ubuntu's version of ffmpeg (and other tools) with the configurations I need?
Short of a setup where I manage my own servers/VMs (which may very well be what I have to do), how might I accomplish this?
The question is: should I try to re-create that pipeline using binaries that I've compiled from scratch? Or is it okay to use apt when configuring in The Real World?
It is in reverse: it is not okay to deploy unpackaged in The Real World IMHO
and not all give me root access
How would you be deploying a .deb without root access. Chroot jails?
But doing so - not to mention trying to figure out how to create a platform-independent .tar.gz binary - will be quite a task which ultimately doesn't really help me ship my product.
+1 You answer you own question. Don't meddle unless you have to.
Do you have any thoughts on the best way to set up my stack so that I'm not tied to a specific hosting provider?
Only depend on wellpackaged standard libs (such as ffmpeg). Otherwise include them in your own deployment. This problem isn't too hard too solve for 10s of thousand Linux applications over decades now, so it would probably be feasible for you too.
Out of the box:
Look at rightscale and other cloud providers/agents that have specialized images/tool chains especially for video encoding.
A 'regular' VPS provider (with Xen or Virtuozzo) will not normally be happy with these kinds of workload, but EC2, Rackspace and their lot will be absolutely fine with that.
In general, I wouldn't believe that a cloud infrastructure provider that doesn't grant root access will allow for computationally intensive workloads. $0.02