I use the following in my script to find how many available cores I can use.
This is for Linux:
NPROCS = $(shell grep -c 'processor' /proc/cpuinfo)
This is for Mac (darwin):
NPROCS = $(shell sysctl -n hw.ncpu)
How should it be in Windows?
You can use one of those commands:
c:wmic cpu get NumberOfCores, NumberOfLogicalProcessors/Format:List
NumberOfCores=6
NumberOfLogicalProcessors=12
NumberOfCores=6
NumberOfLogicalProcessors=12
(here you have two physical processors)
Or as suggested in comments:
c:echo %NUMBER_OF_PROCESSORS%
24
In Makefile the record should be something like:
NPROCS = $(shell echo %NUMBER_OF_PROCESSORS%)
P.S. And the commands you write in question are wrong! You must remove space around =
Related
I know I can perf to record what's happening in core 2 using :
perf record -e sched:sched_switch -C 2
I am curious if there is another tool like top or else , allow me to stare at the screen and monitor what pid(s) executing in core 2 ?!
This is answered in this post, but the short answer is this:
Using ps -e -o psr,pid,%cpu,%mem,args you can get the (virtual) core used under the PSR column, and you can grep for a certain core (in this case 10) with:
ps -e -o psr,pid,%cpu,%mem,args | grep -E '^(PSR|[[:space:]]*10)'
This gives you output like this:
If you want to monitor in real-time you can run the command in a while loop like this, replacing 10 with the core of your choice:
while true; do clear; ps -e -o psr,pid,%cpu,%mem,args | grep -E '^(PSR|[[:space:]]*10 )'; sleep 2; done
You can also add a PROCESSOR column to top: go into top, press f to open the Fields Management menu and choose P (last used CPU). You can then filter for processor core by pressing o and typing in: P=8, replacing 8 with the core you want to monitor.
I want to print certain columns only from ps output that is PID, PPID, command, memory utilization and CPU utilization columns.
when I run ps command I get the following output.
Now I only want some columns from this output so I use -o flag as mentioned in this tutorial.
But I am getting this error.
I don't understand where is the problem. I have also tried usin --help and it is not showing -o flag. So I am confused here.
I am using the windows operating system. And using Git Bash terminal to run all these Linux commands.
Git Bash is a terminal for Windows that emulates the Linux bash (shell) functionality. It is not 100% compatible to a "real" bash shell. As you've empirically seen, its ps executable doesn't support all the flags you're used to from Linux. The --help option will show you what flags are supported.
Hello
Maybe put 2 things together, ps and grep? Then try this...
ps | grep -o -E "^[ 0-9]{1,9}"
...and is this working on your system?
( The Space in [ ] is important )
I'm working on developing a process that will eventually be resident on a CentOS (latest) virtual machine, I'm developing in Ubuntu 12.04 LTS...
So, I have incron set to monitor my drop folder with IN_CLOSE_WRITE so that when a file is written into it a rather resource intensive script is then run on the file (Images, imagemagick). this all works fine; unless too many files are dropped at once. the script, as I said, is rather resource intensive and if more than 4 or so instances run concurrently my development machine is brought to its knees (the eventual virtual machine will be beefier, but I foresee instances where perhaps HUNDREDS of files will be dropped at once!)
dangerous incrontab:
/path/to/dropfolder IN_CLOSE_WRITE bash /path/to/resourceintensivescript.sh $#/$#
so the question is: how to limit the number of jobs spawned by incrond? I tried using gnu parallel but couldn't figure out how to make that work...
for example:
/path/to/dropfolder IN_CLOSE_WRITE parallel --gnu -j 4 bash /path/to/resourceintensivescript.sh $#/$#
seems to do nothing :/
and:
/path/to/dropfolder IN_CLOSE_WRITE;IN_NO_LOOP bash /path/to/resourceintensivescript.sh $#/$#
ends up missing files :P
Ideas on how to deal with this?
A very basic way to do this is to simply use grep and count the processes... something like:
processName=myprocess
if [ $(ps -ef |grep -v grep|grep ${processName} |wc -l) -le 4 ]
then
do something
fi
With the loop suggestion:
processName=myprocess
while true
do
if [ $(ps -ef |grep -v grep|grep ${processName} |wc -l) -le 4 ]
then
do something
break
fi
sleep 5
done
You can use the sem utility that comes with parallel:
/path/to/dropfolder IN_CLOSE_WRITE sem --gnu --id myjobname -j 4 /path/to/resourceintensivescript.sh $#/$#
I have a script that runs some data processing command 10K times.
foreach f (folderName/input*.txt)
mycmd $f
end
I have timed the runtime for each "mycmd $f" to be 0.25 secs.
With 10K runs, it adds up to be more than 1 hr.
I'm running it on a 16 cores nehalem.
It's a huge waste to not run on the remaining 15 cores.
I have tried & with sleep, somehow the script just dies with a warning or error around 3900 iterations, see below. The shorter the sleep, that faster it dies.
foreach f (folderName/input*.txt)
mycmd $f & ; sleep 0.1
end
There has got to be a better way.
Note: I would prefer shell script solutions, let's not wander into C/C++ land.
Thanks
Regards
Pipe the list of files to
xargs -n 1 -P 16 mycmd
For example:
echo folderName/input*.txt | xargs -n 1 -P 16 mycmd
There are a few other solutions possible using one of the following applications:
xjobs
Parallel
PPSS - Parallel Processing Shell Script
runpar.sh
Submit the jobs with batch; that should fix load balancing and resource starvation issues.
for f in folderName/input.*; do
batch <<____HERE
mycmd "$f"
____HERE
done
(Not 100% sure whether the quotes are correct and/or useful.)
With GNU Parallel you can do:
parallel mycmd ::: folderName/input*.txt
From: http://git.savannah.gnu.org/cgit/parallel.git/tree/README
= Full installation =
Full installation of GNU Parallel is as simple as:
./configure && make && make install
If you are not root you can add ~/bin to your path and install in
~/bin and ~/share:
./configure --prefix=$HOME && make && make install
Or if your system lacks 'make' you can simply copy src/parallel
src/sem src/niceload src/sql to a dir in your path.
= Minimal installation =
If you just need parallel and do not have 'make' installed (maybe the
system is old or Microsoft Windows):
wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel
cp parallel sem
mv parallel sem dir-in-your-$PATH/bin/
Watch the intro video for a quick introduction:
https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
I can set number of threads for the build process using -j argument. For example, I have 4 cores +4 virtual. When I write: make -j8 the speed increases 4 times.
Is it possible to set that value as default? (For example, in Linux Gentoo, in config file, it's possible to set this default value).
p.s. I have Arch Linux
Your question is not about threads, but processes (jobs) executed by make.
The simple, way to set this, when make is used from the console is adding:
alias make="/usr/bin/make -j 8"
to your .profile file.
You can also use setenv MAKEFLAGS '-j 8', but MAKEFLAGS can ignore this parameter in some scenarios, because keeping desired number of processes requires communicating with recursive make calls. Happily this method works with current versions of GNU Make.
setenv MAKEFLAGS '-j8'
Hope this helps!
Here's how I've done it:
CORES ?= $(shell sysctl -n hw.ncpu || echo 1)
all:; #$(MAKE) _all -j$(CORES)
_all: install lint test
.PHONY: all _all
…
I've basically "aliased" my default target all to a "private" _all. The command to figure out the number of cores is OSX specific, AFAIK, so you could just improve it to be more cross platform if you will. And because of the ?= assignment, we can just override it with and env variable if/when needed.
EDIT:
You can also append to your MAKEFLAGS from within the makefile itself, like so:
CPUS ?= $(shell sysctl -n hw.ncpu || echo 1)
MAKEFLAGS += --jobs=$(CPUS)
…
EDIT 2:
You may also use the following, ff you want it to be more cross-platform:
CPUS ?= $(shell (nproc --all || sysctl -n hw.ncpu) 2>/dev/null || echo 1)