Run shell script for every file in directory - linux

I have a bunch of files in a directory all named YYYY_MM_DD
-rw-r--r-- 1 root root 480K Apr 21 13:17 2012_04_05
-rw-r--r-- 1 root root 483K Apr 21 13:17 2012_04_06
-rw-r--r-- 1 root root 484K Apr 21 13:17 2012_04_07
-rw-r--r-- 1 root root 480K Apr 21 13:17 2012_04_08
-rw-r--r-- 1 root root 344K Apr 21 13:17 2012_04_09
-rw-r--r-- 1 root root 66K Apr 21 13:17 2012_04_10
-rw-r--r-- 1 root root 461K Apr 21 13:17 2012_04_11
-rw-r--r-- 1 root root 475K Apr 21 15:09 2012_04_17
-rw-r--r-- 1 root root 480K Apr 21 15:10 2012_04_18
-rw-r--r-- 1 root root 474K Apr 21 15:10 2012_04_19
-rw-r--r-- 1 root root 474K Apr 21 15:10 2012_04_20
I have a shell script that accepts a file as a paramater and calculates figures based on the data in the file, i call the script like this
sh Calculate.sh MyFile
I want to run this shell script for every file in this directory.
How would i go about doing this, xargs ??

Have you tried the find command with execution ?
My sample will echo the files, but you can call a shell script with the filename as a parameter
find . -maxdepth 1 -type f -exec echo {} \;

A simple for loop in the shell:
for file in *; do sh Calculate.sh "$file"; done

find . -maxdepth 1 -type f | xargs -n 1 -I % Calculate.sh %

./Calculate.sh 2012_04_{05..20}

Related

linux find files by created in last X minutes + sort does not return expected list

I have a folder with many jpg files, where each file name presents the creation time in the form of: "20220204T160311.746....jpg" ("..." represent additional data).
I'm trying to get all files created on the last X minutes. There are tons of results for such query if one asks Google, nevertheless, I'm getting weird results when trying the following:
find . -cmin -50 \( ! -iname "*.csv" ! -iname ".*" \) -ls | sort -n
the results do not start ok ("now" is 16:26):
14466561 2636 -rw-r--r-- 1 root root 2695349 Feb 4 16:03 ./20220204T160348.526-cam4-791-AREASbw-32358.jpg
14466569 2208 -rw-r--r-- 1 root root 2257356 Feb 4 16:03 ./20220204T160354.069-cam4-614.jpg
14466571 1044 -rw-r--r-- 1 root root 1068758 Feb 4 16:03 ./20220204T160354.069-cam4-614__chng_182.jpg
14466579 2208 -rw-r--r-- 1 root root 2258697 Feb 4 16:03 ./20220204T160359.296-cam4-352.jpg
14466581 1048 -rw-r--r-- 1 root root 1070111 Feb 4 16:03 ./20220204T160359.296-cam4-352__chng_175.jpg
14466589 2216 -rw-r--r-- 1 root root 2268675 Feb 4 16:04 ./20220204T160404.682-cam4-179.jpg
14466591 1048 -rw-r--r-- 1 root root 1070053 Feb 4 16:04 ./20220204T160404.682-cam4-179__chng_273.jpg
14466597 2204 -rw-r--r-- 1 root root 2253706 Feb 4 16:04 ./20220204T160409.957-cam4-571.jpg
14466599 1048 -rw-r--r-- 1 root root 1070335 Feb 4 16:04 ./20220204T160409.957-cam4-571__chng_154.jpg
14466600 1044 -rw-r--r-- 1 root root 1068675 Feb 4 16:04 ./20220204T160409.957-cam4-571__chng_309.jpg
14466601 2652 -rw-r--r-- 1 root root 2712638 Feb 4 16:04 ./20220204T160409.957-cam4-571-AREASbw-31696.jpg
14466610 2208 -rw-r--r-- 1 root root 2259408 Feb 4 16:04 ./20220204T160415.421-cam4-959.jpg
14466611 1048 -rw-r--r-- 1 root root 1070365 Feb 4 16:04 ./20220204T160415.421-cam4-959__chng_272.jpg
14466620 2212 -rw-r--r-- 1 root root 2264606 Feb 4 16:04 ./20220204T160420.666-cam4-742.jpg
14466621 1048 -rw-r--r-- 1 root root 1070276 Feb 4 16:04 ./20220204T160420.666-cam4-742__chng_173.jpg
and when I continue to scroll up, it arrives to a part where the files are wrongly sorted (like what supposed to be the start of the list):
14457644 2240 -rw-r--r-- 1 root root 2293209 Feb 4 16:26 ./20220204T162633.985-cam4-188.jpg
14457645 1056 -rw-r--r-- 1 root root 1080805 Feb 4 16:26 ./20220204T162633.985-cam4-188__chng_171.jpg
14457658 2236 -rw-r--r-- 1 root root 2286664 Feb 4 16:26 ./20220204T162639.299-cam4-801.jpg
14457659 1056 -rw-r--r-- 1 root root 1080042 Feb 4 16:26 ./20220204T162639.299-cam4-801__chng_158.jpg
14462048 2200 -rw-r--r-- 1 root root 2252735 Feb 4 15:49 ./20220204T154927.284-cam4-095.jpg
14462049 1044 -rw-r--r-- 1 root root 1065249 Feb 4 15:49 ./20220204T154927.284-cam4-095__chng_134.jpg
14462088 2204 -rw-r--r-- 1 root root 2255657 Feb 4 15:49 ./20220204T154932.490-cam4-571.jpg
14462089 1044 -rw-r--r-- 1 root root 1066449 Feb 4 15:49 ./20220204T154932.490-cam4-571__chng_228.jpg
14462118 2204 -rw-r--r-- 1 root root 2254481 Feb 4 15:49 ./20220204T154937.767-cam4-237.jpg
14462127 1044 -rw-r--r-- 1 root root 1066700 Feb 4 15:49 ./20220204T154937.767-cam4-237__chng_79.jpg
How come in the middle it goes back from 15:49 to 16:26?
It happens on every query I'm running, unless it is on a short term like 10 min.
By default sort uses the first column. Use -k to specify column number to sort by.
find . -cmin -50 \( ! -iname "*.csv" ! -iname ".*" \) -ls | sort -n -k 7

Linux sort by month and year

I am trying to sort this data by the date such as the Oct 30 2020 column however I am not sure how
here is my current command
sudo find / -maxdepth 4 -size +1M -exec ls -lh {} \; | sort -k 6hr
Here is my output
-rw-r--r-- 1 root root 80M Oct 30 2020 /usr/lib/x86_64-linux-gnu/libLLVM-11.so.1
-rw-r--r-- 1 root root 38M Aug 27 13:02 /var/cache/apt/srcpkgcache.bin
-rw-r--r-- 1 root root 38M Sep 6 15:57 /var/cache/apt/pkgcache.bin
-rw-r--r-- 1 root root 27M Mar 17 2020 /usr/lib/x86_64-linux-gnu/libicudata.so.66.1
-rwxr-xr-x 1 root root 27M Feb 2 2021 /usr/lib/snapd/snapd
-rwxr-xr-x 1 root root 23M Feb 2 2021 /usr/bin/snap
-rwxr-xr-x 1 root root 16M Feb 2 2021 /usr/lib/snapd/snap-bootstrap
-rwxr-xr-x 1 root root 12M Feb 2 2021 /usr/lib/snapd/snap-repair
-r--r--r-- 1 root root 9.5M Feb 19 2021 /usr/lib/udev/hwdb.bin
-rw-r--r-- 1 root root 9.0M Dec 18 2020 /usr/lib/x86_64-linux-gnu/libvulkan_intel.so
-rwxr-xr-x 1 root root 8.8M Feb 2 2021 /usr/lib/snapd/snap-preseed
-rw-r--r-- 1 root root 8.7M May 29 00:49 /usr/lib/x86_64-linux-gnu/libtsan.so.0.0.0
-rwxr-xr-x 1 root root 8.7M Feb 2 2021 /usr/lib/snapd/snap-recovery-chooser
-rwxr-xr-x 1 root root 8.3M Feb 2 2021 /usr/lib/snapd/snapctl
-rw-r--r-- 1 root root 7.2M Dec 18 2020 /usr/lib/x86_64-linux-gnu/libvulkan_radeon.so
-rw-r--r-- 1 root root 6.3M Dec 16 2020 /usr/lib/x86_64-linux-gnu/libpthread.a
-rwxr-xr-x 1 root root 5.8M Feb 2 2021 /usr/lib/snapd/snap-update-ns
-rw-r--r-- 1 root root 5.6M Jan 16 2020 /usr/lib/file/magic.mgc
-rw-r--r-- 1 root root 5.5M Dec 16 2020 /usr/lib/x86_64-linux-gnu/libc.a
-rwxr-xr-x 1 root root 5.3M Jul 28 2020 /usr/bin/python3.8
-rw-r--r-- 1 root root 5.2M Jul 28 2020 /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0
-r-xr-xr-x 2 admin1 admin1 4.7M Aug 26 02:01 /mnt/c/Windows/explorer.exe
-rwxr-xr-x 1 root root 4.6M Feb 2 2021 /usr/lib/snapd/snap-failure
-rwxrwxrwx 1 admin1 admin1 4.3M Jun 28 16:08 /mnt/c/Python39/python39.dll
-rwxr-xr-x 1 root root 3.9M Feb 2 2021 /usr/lib/snapd/snap-exec
-rw-r--r-- 1 root root 3.4M Oct 19 2020 /usr/lib/x86_64-linux-gnu/libperl.so.5.30.0
-rwxr-xr-x 2 root root 3.4M Oct 19 2020 /usr/bin/perl
Thanks so much!!
Or a version that uses a different magic spell for ls:
sudo find / -maxdepth 4 -size +1M -exec ls -lh --time-style=long-iso {} \; | sort -k6
Because it thoroughly annoys me that ls doesn't show years by default when timestamps lie within the last 6 months and MMM DD YYYY makes no sense to me in the first place I've exported TIME_STYLE=long-iso in my ~/.bashrc ... so much more logical and easy to sort.
You can do this with what is called a decorate + sort + undecorate approach where you decorate each line by adding a new column at the end (a new column 10) in the form yyyymmdd and then sort by the new last column and then remove (undecorate) the output by removing the last column.
awk is the normal choice, and this works quite well as long as the filenames do NOT contain whitespace (which would skew the column (field) count for awk).
In your case you could do:
sudo find / -maxdepth 4 -size +1M -exec ls -lh {} \; |
awk 'BEGIN {
m["Jan"]=1; m["Feb"]=2; m["Mar"]=3; m["Apr"]=4; m["May"]=5; m["Jun"]=6;
m["Jul"]=7; m["Aug"]=8; m["Sep"]=9; m["Oct"]=10; m["Nov"]=11; m["Dec"]=12}
{
year = ($8 ~ /:/) ? 2021 : $8
mon = (length(m[$6]) < 2) ? "0"m[$6] : m[$6]
day = (length($7) < 2) ? "0"$7 : $7
$(NF+1) = year mon day
}1
' | sort -k 10 | awk '{$NF = ""}1'
Above, the first call to awk decorates each line with the new yyyymmdd column (new column (field) 10) and then the results are sorted by field 10 with sort. The second awk command undecorates each line.
Example Use/Output
Taking your shown output and placing it in the file file you could sort as:
awk 'BEGIN {
m["Jan"]=1; m["Feb"]=2; m["Mar"]=3; m["Apr"]=4; m["May"]=5; m["Jun"]=6;
m["Jul"]=7; m["Aug"]=8; m["Sep"]=9; m["Oct"]=10; m["Nov"]=11; m["Dec"]=12}
{
year = ($8 ~ /:/) ? 2021 : $8
mon = (length(m[$6]) < 2) ? "0"m[$6] : m[$6]
day = (length($7) < 2) ? "0"$7 : $7
$(NF+1) = year mon day
}1
' file | sort -k 10 | awk '{$NF = ""}1'
-rw-r--r-- 1 root root 5.6M Jan 16 2020 /usr/lib/file/magic.mgc
-rw-r--r-- 1 root root 27M Mar 17 2020 /usr/lib/x86_64-linux-gnu/libicudata.so.66.1
-rw-r--r-- 1 root root 5.2M Jul 28 2020 /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0
-rwxr-xr-x 1 root root 5.3M Jul 28 2020 /usr/bin/python3.8
-rw-r--r-- 1 root root 3.4M Oct 19 2020 /usr/lib/x86_64-linux-gnu/libperl.so.5.30.0
-rwxr-xr-x 2 root root 3.4M Oct 19 2020 /usr/bin/perl
-rw-r--r-- 1 root root 80M Oct 30 2020 /usr/lib/x86_64-linux-gnu/libLLVM-11.so.1
-rw-r--r-- 1 root root 5.5M Dec 16 2020 /usr/lib/x86_64-linux-gnu/libc.a
-rw-r--r-- 1 root root 6.3M Dec 16 2020 /usr/lib/x86_64-linux-gnu/libpthread.a
-rw-r--r-- 1 root root 7.2M Dec 18 2020 /usr/lib/x86_64-linux-gnu/libvulkan_radeon.so
-rw-r--r-- 1 root root 9.0M Dec 18 2020 /usr/lib/x86_64-linux-gnu/libvulkan_intel.so
-rwxr-xr-x 1 root root 12M Feb 2 2021 /usr/lib/snapd/snap-repair
-rwxr-xr-x 1 root root 16M Feb 2 2021 /usr/lib/snapd/snap-bootstrap
-rwxr-xr-x 1 root root 23M Feb 2 2021 /usr/bin/snap
-rwxr-xr-x 1 root root 27M Feb 2 2021 /usr/lib/snapd/snapd
-rwxr-xr-x 1 root root 3.9M Feb 2 2021 /usr/lib/snapd/snap-exec
-rwxr-xr-x 1 root root 4.6M Feb 2 2021 /usr/lib/snapd/snap-failure
-rwxr-xr-x 1 root root 5.8M Feb 2 2021 /usr/lib/snapd/snap-update-ns
-rwxr-xr-x 1 root root 8.3M Feb 2 2021 /usr/lib/snapd/snapctl
-rwxr-xr-x 1 root root 8.7M Feb 2 2021 /usr/lib/snapd/snap-recovery-chooser
-rwxr-xr-x 1 root root 8.8M Feb 2 2021 /usr/lib/snapd/snap-preseed
-r--r--r-- 1 root root 9.5M Feb 19 2021 /usr/lib/udev/hwdb.bin
-rw-r--r-- 1 root root 8.7M May 29 00:49 /usr/lib/x86_64-linux-gnu/libtsan.so.0.0.0
-rwxrwxrwx 1 admin1 admin1 4.3M Jun 28 16:08 /mnt/c/Python39/python39.dll
-r-xr-xr-x 2 admin1 admin1 4.7M Aug 26 02:01 /mnt/c/Windows/explorer.exe
-rw-r--r-- 1 root root 38M Aug 27 13:02 /var/cache/apt/srcpkgcache.bin
-rw-r--r-- 1 root root 38M Sep 6 15:57 /var/cache/apt/pkgcache.bin
Note, you use of -exec in the find command is horribly inefficient and you are better served using the -printf option with available format specifiers that can create the file output in any order you like and decorate the output for you with yyyymmdd. That would save calling ls -lh on each file to produce the file listing.
There are other ways to approach this, so do not think this is the only one-way to do this. Further note the limitation on whitespace in the filename corrupting the fields if this approach is used.
GNU date can parse the format printed by ls. The following converts the date into Unix epoch seconds and uses a Schwartzian transform to sort the lines. The transform needs four steps:
Calculate the sort key
Prefix the data with the sort key
Sort the prefixed data
Remove the prefix
while read line; do
date=$(sed 's/ / /g' <<< "$line" | cut -d ' ' -f 6-8)
epoch=$(date -d "$date" +%s)
printf '%s %s\n' "$epoch" "$line"
done | sort -n | sed 's/^[0-9]* //'
But: Why you shouldn't parse the output of ls(1)
Instead it would be better to tell find to print just the time and the file name. This can be sorted, the sort key can be removed, and finally the file names can be passed to ls. But you have to use the option -f to keep the sort order.
sudo find / -maxdepth 4 -size +1M -printf "%C# %p\n" |
sort -n |
sed 's/^[0-9.]* //' |
xargs ls -flh

Using grep and * to get the exact file name

I have following heapdump files
AppSrv01]# ls -ltr *heapdump*
-rw-r--r-- 1 root root 0 Sep 13 16:44 hbsbdheapdump.3436
-rw-r--r-- 1 root root 0 Sep 13 16:44 hbsbdheapdump.3435
-rw-r--r-- 1 root root 0 Sep 13 16:44 hbsbdheapdump.3434
-rw-r--r-- 1 root root 0 Sep 13 16:44 hbsbdheapdump.3433
-rw-r--r-- 1 root root 0 Sep 13 16:44 hbsbdheapdump.3432
-rw-r--r-- 1 root root 0 Sep 13 16:44 hbsbdheapdump.3431
-rw-r--r-- 1 root root 0 Sep 13 17:03 heapdump.dfsdf
now if I use
ls -ltr| grep heapdump*
I get following output
AppSrv01]# ls -ltr |grep heapdump*
-rw-r--r-- 1 root root 0 Sep 13 17:03 heapdump.dfsdf
But I use
ls -ltr |grep *heapdump*
I get no output. Could anybody help where is my mistake and how to search the heapdump files.
My expected output is
ls -ltr | grep *heapdump*
should give me
-rw-r--r-- 1 root root 0 Sep 13 16:44 hbsbdheapdump.3436
-rw-r--r-- 1 root root 0 Sep 13 16:44 hbsbdheapdump.3435
-rw-r--r-- 1 root root 0 Sep 13 16:44 hbsbdheapdump.3434
-rw-r--r-- 1 root root 0 Sep 13 16:44 hbsbdheapdump.3433
-rw-r--r-- 1 root root 0 Sep 13 16:44 hbsbdheapdump.3432
-rw-r--r-- 1 root root 0 Sep 13 16:44 hbsbdheapdump.3431
-rw-r--r-- 1 root root 0 Sep 13 17:03 heapdump.dfsdf
due to some reason I could not use
ls -ltr *heapdump*
It looks like you're confusing glob patterns with regular expressions, which is what grep uses.
The correct way to print a list of files containing the word heapdump is like this:
printf '%s\n' *heapdump*
Here, *heapdump* is glob-expanded by the shell to match anything followed by "heapdump" followed by anything else. All the files that match the pattern are passed to printf, which prints each one followed by a newline.
If you want to filter the output of ls -ltr, then you can use:
ls -ltr *heapdump*
The same list of files will be passed to ls, which will print information about them.
Bear in mind that it is not recommended to attempt to parse the output of ls.

combine perl command inside the bash script

Perl is excellent language to store command output to variable
As I know bash isn’t support that
for example
my $value = qx(ls -ltr);
print $value;
it will print: ( example )
-r-xr-xr-x 1 root root 761 May 30 2014 ASon.conf
-r-xr-xr-x 1 root root 699 May 30 2014 ascpl
drwxr-xr-x 2 root root 4096 Feb 24 13:22 da
drwx------ 2 root root 4096 Feb 25 08:08 vm-root
drwxr-x--- 2 root root 4096 Feb 25 08:10 hspeoot
-rw-r--r-- 1 root root 2451 Feb 25 16:12 gry.txt
-rw-r--r-- 1 root root 3112 Mar 1 09:49 new.xml
-rw-r--r-- 1 root root 44893 Mar 1 12:13 SMports.xml
-rwxr-xr-x 1 root root 101 Mar 2 09:56 test
-rwxr-xr-x 1 root root 461340404 Mar 2 10:20 audin.log
the problem is that I write my code with bash
and I want to use the Perl add value in my bash script
so is it possible to combine the following line in my bash script?
my $value = qx(ls -ltr);
in way that when I print the value variable it will print the ls -ltr output
You don't need perl for this, your shell can handle it:
value=$(ls -ltr)
echo "$value"
The quotes when you echo are important.
In Perl:
$var = `shell command`;
$var = qx( shell command );
In sh:
var=`shell command`
var=$( shell command )

how to rename multiple prefix filename with one command in linux ubuntu server 12.04

I have a files in a folder like this :
-rwxrwxrwx 1 www-data www-data 25088 Nov 6 2013 108400344_2223_abstraksi.doc
-rwxrwxrwx 1 www-data www-data 15119 Nov 6 2013 108400344_2223_abstraksi.docx
-rwxrwxrwx 1 www-data www-data 146532 Nov 6 2013 108400344_2223_abstraksi.pdf
-rwxrwxrwx 1 www-data www-data 145408 Nov 6 2013 108400344_2223_bab1.doc
-rwxrwxrwx 1 www-data www-data 75399 Nov 6 2013 108400344_2223_bab1.docx
-rwxrwxrwx 1 www-data www-data 193581 Nov 6 2013 108400344_2223_bab1.pdf
-rwxrwxrwx 1 www-data www-data 156672 Nov 6 2013 108400344_2223_bab2.doc
-rwxrwxrwx 1 www-data www-data 101341 Nov 6 2013 108400344_2223_bab2.docx
-rwxrwxrwx 1 www-data www-data 316476 Nov 6 2013 108400344_2223_bab2.pdf
-rwxrwxrwx 1 www-data www-data 952320 Nov 6 2013 108400344_2223_bab3.doc
-rwxrwxrwx 1 www-data www-data 89826 Nov 6 2013 108400344_2223_bab3.docx
-rwxrwxrwx 1 www-data www-data 399203 Nov 6 2013 108400344_2223_bab3.pdf
And then i want to change "108400344_2223_" to be "14.04.329" in the same time. I try almost all tutorial but it fail..so how to do this in linux ubuntu server 12.04?
Thanks
Given your files in some directory 'e':
$ ls -1 e/
108400344_2223_abstraksi.doc
108400344_2223_abstraksi.docx
108400344_2223_abstraksi.pdf
108400344_2223_bab1.doc
108400344_2223_bab1.docx
108400344_2223_bab1.pdf
108400344_2223_bab2.doc
108400344_2223_bab2.docx
108400344_2223_bab2.pdf
108400344_2223_bab3.doc
108400344_2223_bab3.docx
108400344_2223_bab3.pdf
You can easily use string substitution to rename all files from 108400344_2223_file to 14.04.329file
$ for i in e/*; do mv "$i" ${i//108400344_2223_/14.04.329}; done
confirm:
$ ls -11 e
14.04.329abstraksi.doc
14.04.329abstraksi.docx
14.04.329abstraksi.pdf
14.04.329bab1.doc
14.04.329bab1.docx
14.04.329bab1.pdf
14.04.329bab2.doc
14.04.329bab2.docx
14.04.329bab2.pdf
14.04.329bab3.doc
14.04.329bab3.docx
14.04.329bab3.pdf
Personally, I would add an underscore between the date and file:
$ for i in e/*; do mv "$i" ${i//108400344_2223/14.04.329}; done
Giving:
14.04.329_abstraksi.doc
14.04.329_abstraksi.docx
...
Hope that helps.
There are many ways to do this.
Since Ubuntu comes with the perl rename program, you could simply do
rename 's/108400344_2223_/14.04.329_/' *
Shell
for f in *; do mv "$f" "${f/108400344_2223_/14.04.329_}"; done
If the files can be in multiple sub directories, use find:
find . -name '108400344_2223_*' -exec rename 's/108400344_2223_/14.04.329_/' {} +
I hope this will be helpfull
for k in `ls -la | awk '{print $9}'`; do mv $k 14.04.329`ls -la $k | awk -F _ '{print $3}'`; done
output is like this:
-rw-r--r-- 1 root root 0 Sep 23 09:46 14.04.329abstraksi.doc
-rw-r--r-- 1 root root 0 Sep 23 09:46 14.04.329abstraksi.docx
-rw-r--r-- 1 root root 0 Sep 23 09:46 14.04.329abstraksi.pdf
-rw-r--r-- 1 root root 0 Sep 23 09:46 14.04.329bab1.doc
-rw-r--r-- 1 root root 0 Sep 23 09:46 14.04.329bab1.docx
-rw-r--r-- 1 root root 0 Sep 23 09:46 14.04.329bab1.pdf
-rw-r--r-- 1 root root 0 Sep 23 09:46 14.04.329bab2.doc

Resources