I can set number of threads for the build process using -j argument. For example, I have 4 cores +4 virtual. When I write: make -j8 the speed increases 4 times.
Is it possible to set that value as default? (For example, in Linux Gentoo, in config file, it's possible to set this default value).
p.s. I have Arch Linux
Your question is not about threads, but processes (jobs) executed by make.
The simple, way to set this, when make is used from the console is adding:
alias make="/usr/bin/make -j 8"
to your .profile file.
You can also use setenv MAKEFLAGS '-j 8', but MAKEFLAGS can ignore this parameter in some scenarios, because keeping desired number of processes requires communicating with recursive make calls. Happily this method works with current versions of GNU Make.
setenv MAKEFLAGS '-j8'
Hope this helps!
Here's how I've done it:
CORES ?= $(shell sysctl -n hw.ncpu || echo 1)
all:; #$(MAKE) _all -j$(CORES)
_all: install lint test
.PHONY: all _all
…
I've basically "aliased" my default target all to a "private" _all. The command to figure out the number of cores is OSX specific, AFAIK, so you could just improve it to be more cross platform if you will. And because of the ?= assignment, we can just override it with and env variable if/when needed.
EDIT:
You can also append to your MAKEFLAGS from within the makefile itself, like so:
CPUS ?= $(shell sysctl -n hw.ncpu || echo 1)
MAKEFLAGS += --jobs=$(CPUS)
…
EDIT 2:
You may also use the following, ff you want it to be more cross-platform:
CPUS ?= $(shell (nproc --all || sysctl -n hw.ncpu) 2>/dev/null || echo 1)
Related
I use the following in my script to find how many available cores I can use.
This is for Linux:
NPROCS = $(shell grep -c 'processor' /proc/cpuinfo)
This is for Mac (darwin):
NPROCS = $(shell sysctl -n hw.ncpu)
How should it be in Windows?
You can use one of those commands:
c:wmic cpu get NumberOfCores, NumberOfLogicalProcessors/Format:List
NumberOfCores=6
NumberOfLogicalProcessors=12
NumberOfCores=6
NumberOfLogicalProcessors=12
(here you have two physical processors)
Or as suggested in comments:
c:echo %NUMBER_OF_PROCESSORS%
24
In Makefile the record should be something like:
NPROCS = $(shell echo %NUMBER_OF_PROCESSORS%)
P.S. And the commands you write in question are wrong! You must remove space around =
I need to create kernel panic and I tried following
sysctl kernel.panic=0 && echo c > /proc/sysrq-trigger
When I ran the commands above. I see system always reboots. I need system to be in panic mode without rebooting
Use -w option when you want to change a sysctl setting under RHEL.
Multiple commands example:
> sysctl -w kernel.panic="0"
> echo c > /proc/sysrq-trigger
Notice that if you want to preserve kernel settings after reboot, it's always better to add them to the /etc/sysctl.conf file. However the quickly setting method maybe enough for your testing requirments.
Also make sure you don't paste both commands "sysctl -w kernel.panic=0 echo c > /proc/sysrq-trigger" together. (I'm always giving this recommendation when i see multiple shell commands posted together, like i see in your question). Or use && operator to execute the next command like this:
Single line example:
sysctl -w kernel.panic="0" && echo c > /proc/sysrq-trigger
I'm trying to build embedded system using buildroot. Everything seems to work. All modules are starting, the system is stable. The problem is that /etc/init.d/rcS does not start during initialization of the system. If I run it manually everything is OK. I have it in my inittab file.
# /etc/inittab
#
# Copyright (C) 2001 Erik Andersen <andersen#codepoet.org>
#
# Note: BusyBox init doesn't support runlevels. The runlevels field is
# completely ignored by BusyBox init. If you want runlevels, use
# sysvinit.
#
# Format for each entry: <id>:<runlevels>:<action>:<process>
#
# id == tty to run on, or empty for /dev/console
# runlevels == ignored
# action == one of sysinit, respawn, askfirst, wait, and once
# process == program to run
# Startup the system
null::sysinit:/bin/mount -t proc proc /proc
null::sysinit:/bin/mount -o remount,rw /
null::sysinit:/bin/mkdir -p /dev/pts
null::sysinit:/bin/mkdir -p /dev/shm
null::sysinit:/bin/mount -a
null::sysinit:/bin/hostname -F /etc/hostname
# now run any rc scripts
::sysinit:/etc/init.d/rcS
# Put a getty on the serial port
ttyFIQ0::respawn:/sbin/getty -L -n ttyFIQ0 115200 vt100 # GENERIC_SERIAL
# Stuff to do for the 3-finger salute
::ctrlaltdel:/sbin/reboot
# Stuff to do before rebooting
null::shutdown:/etc/init.d/rcK
null::shutdown:/bin/umount -a -r
null::shutdown:/sbin/swapoff -a
Any idea what could be wrong?
/bin/init needs to be on your filesystem.
/bin/sh needs to be on your filesystem.
/etc/init.d/rcS needs to be executable and have #!/bin/sh as its first line.
Init
Are you sure you where invoking Busybox init? What was the kernel command line? If no init= option was supplied to the the kernel, the kernel will look for an executable at /init.
For instance, if your busybox binary resides in /bin/busybox, you need to create the following symlink :
ln -s /bin/busybox /init
If you want your init to reside in /sbin, to comply with the inittab, also create a symlink there. Note that the kernel will not respect init= setting if you don't mount root and your busybox only runs in an initramfs.
ln -s /bin/busybox /sbin/init
Inittab
Also, you could try not using an inittab. The things you try to run from inittab, might very well fit in rcS and any descendant scripts. From the same source you found your example inittab:
# Note: BusyBox init works just fine without an inittab. If no inittab is
# found, it has the following default behavior:
# ::sysinit:/etc/init.d/rcS
# ::askfirst:/bin/sh
# ::ctrlaltdel:/sbin/reboot
# ::shutdown:/sbin/swapoff -a
# ::shutdown:/bin/umount -a -r
# ::restart:/sbin/init
# tty2::askfirst:/bin/sh
# tty3::askfirst:/bin/sh
# tty4::askfirst:/bin/sh
rcS
Make sure /etc/init.d/rcS is executable:
chmod +x chroot chroot /bin/busybox
And try with:
#!/bin/busybox sh
echo "Hello world!"
Please note that this sentence can get buried between kernel log messages, so you might want to pass the quiet kernel command line option to see if it appears.
Busybox symlinks
Are the symlinks installed into the file system or not? If not it is not a disaster. Make sure that /etc/init.d/rcS starts with:
#!/bin/busybox sh
mkdir -pv /sbin
/bin/busybox --install -s
In addition to the scripts themselves being executable and having a correct shebang line, the kernel also needs to be compiled with the CONFIG_BINFMT_SCRIPT option enabled.
CONFIG_BINFMT_SCRIPT:
Say Y here if you want to execute interpreted scripts starting with
#! followed by the path to an interpreter.
You can build this support as a module; however, until that module
gets loaded, you cannot run scripts. Thus, if you want to load this
module from an initramfs, the portion of the initramfs before loading
this module must consist of compiled binaries only.
Most systems will not boot if you say M or N here. If unsure, say Y.
Without this option, you may receive the error message can't run '/etc/init.d/rcS': Exec format error.
From the information given, everything looks correct.
Some things to try:
Check ownership of your rcS script.
Comment out everything from rcS, and add something very simple:
echo "This worked" > /tmp/test
There might be something in your script related to a startup race condition that is causing it to exit. Also curious if your script is starting syslogd.
Please take a look at the following code,
snap=snapshot.file
touch snapshot.file-1
$ [ -a $snap-1 ] && echo yes
yes
What does the test -a command tests for here?
I tried info coreutils 'test invocation' and searched for -a, but didn't find it in the file characteristic tests section, but rather in the connectives for test section.
Is such test -a command an undocumented one?
-a is used for an and expression. You would usually use it with two operands:
$[ $snap0 -a $snap1 ]
not sure what context it is used in here, but it's possible that someone removed the first operand without removing the -a operator.
I have a script that runs some data processing command 10K times.
foreach f (folderName/input*.txt)
mycmd $f
end
I have timed the runtime for each "mycmd $f" to be 0.25 secs.
With 10K runs, it adds up to be more than 1 hr.
I'm running it on a 16 cores nehalem.
It's a huge waste to not run on the remaining 15 cores.
I have tried & with sleep, somehow the script just dies with a warning or error around 3900 iterations, see below. The shorter the sleep, that faster it dies.
foreach f (folderName/input*.txt)
mycmd $f & ; sleep 0.1
end
There has got to be a better way.
Note: I would prefer shell script solutions, let's not wander into C/C++ land.
Thanks
Regards
Pipe the list of files to
xargs -n 1 -P 16 mycmd
For example:
echo folderName/input*.txt | xargs -n 1 -P 16 mycmd
There are a few other solutions possible using one of the following applications:
xjobs
Parallel
PPSS - Parallel Processing Shell Script
runpar.sh
Submit the jobs with batch; that should fix load balancing and resource starvation issues.
for f in folderName/input.*; do
batch <<____HERE
mycmd "$f"
____HERE
done
(Not 100% sure whether the quotes are correct and/or useful.)
With GNU Parallel you can do:
parallel mycmd ::: folderName/input*.txt
From: http://git.savannah.gnu.org/cgit/parallel.git/tree/README
= Full installation =
Full installation of GNU Parallel is as simple as:
./configure && make && make install
If you are not root you can add ~/bin to your path and install in
~/bin and ~/share:
./configure --prefix=$HOME && make && make install
Or if your system lacks 'make' you can simply copy src/parallel
src/sem src/niceload src/sql to a dir in your path.
= Minimal installation =
If you just need parallel and do not have 'make' installed (maybe the
system is old or Microsoft Windows):
wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel
cp parallel sem
mv parallel sem dir-in-your-$PATH/bin/
Watch the intro video for a quick introduction:
https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1