Why the "while" concurrent is running slower and slower in shell script? - linux

Question

I want zip lots of files using more cpus at onetime by using shell's concurrent function like this:
#!/bin/bash
#set -x
function zip_data()
{
while true
do
{
echo "do zip something"
}&
done
}
zip_data
wait
At the begin time ,the speed of loop onetime is quickly .
But with the increase of loop times when running,
the speed is more and more slowly.
Why ???
I think the reason may be there are too many child processes are runing .
So I try that make the while function running one loop at onetime like this:
#!/bin/bash
#set -x
function trap_exit
{
exec 1000>&-;exec 1000<&-
kill -9 0
}
trap 'trap_exit; exit 2' 1 2 3 15 9
mkfifo testfifo ; exec 1000<>testfifo ; rm -rf testfifo
function zip_data()
{
echo >&1000
while true
read -u 1000
do
{
echo "do something"
echo >&1000
}&
done
}
zip_data
wait
However the phenomenon is same like before .
So I don't understand that reason why the speed is more and more slowly when runing.
#
Today I try like this but it don't work
#!/bin/bash
#set -x
c=0
while true
do
c=$(jobs -p | wc -l)
while [ $c -ge 20 ]; do
c=$(jobs -p | wc -l)
sleep 0.01
done
{
echo "$c"
sleep 0.8
}&
done
So i try other way to finish this function like this ,Thank you!
#!/bin/bash
#set -x
function EXPECT_FUNC()
{
para=$1
while true
do
{
do something $1
}
done
}
EXPECT_FUNC 1 &
EXPECT_FUNC 2 &
EXPECT_FUNC 3 &
EXPECT_FUNC 4 &
wait

Answer 1

Any single-threaded util can run in well managed concurrent threads with parallel. man parallel offers dozens of examples, e.g.:
Create a directory for each zip-file and unzip it in that dir:
parallel 'mkdir {.}; cd {.}; unzip ../{}' ::: *.zip
Recompress all .gz files in current directory using bzip2 running 1 job
per CPU core in parallel:
parallel "zcat {} | bzip2 >{.}.bz2 && rm {}" ::: *.gz
A particularly interesting example that only works with gzip shows how to use several CPUs to simultaneously work one archive, with a single-threaded archiver, which sounds impossible:
To process a big file or some output you can use --pipe to split up
the data into blocks and pipe the blocks into the processing program.
If the program is gzip -9 you can do:
cat bigfile | parallel --pipe --recend '' -k gzip -9 >bigfile.gz
This will split bigfile into blocks of 1 MB and pass that to
gzip -9 in parallel. One gzip will be run per CPU core. The output
of gzip -9 will be kept in order and saved to bigfile.gz
If parallel is too complex, here are some compression utils with built-in parallel archiving:
XZ: pixz
LZMA: plzip, pxz
GZIP: pigz
BZIP2: pbzip2