How to write a shell script that creates empty files with all possible permissions. File names should be, for example, rwxrw_r__.txt. I know how to do it manually. As an example:
#!/bin/bash
touch task5/rwxrwxrwx.txt | chmod 777 task5/rwxrwxrwx.txt
touch task5/rwxr-xr-x.txt | chmod 755 task5/rwxr-xr-x.txt
touch task5/rwx------.txt | chmod 700 task5/rwx------.txt
touch task5/rw-rw-rw-.txt | chmod 666 task5/rw-rw-rw-.txt
touch task5/rw-r--r--.txt | chmod 644 task5/rw-r--r--.txt
touch task5/rw-------.txt | chmod 600 task5/rw-------.txt
I do not know how to write a script that will create files according to the required template and give them permissions
Do a loop:
for ((i=0; i < 512; i++)); do
mod=$(printf "%03o" "$i");
touch ${mod}.txt; chmod $mod $mod.txt;
done
Rather than trying to construct the names, if you want the names to look like the output of ls -l, just do something like
for ((i=0; i < 512; i++)); do
mod=$(printf "%03o" "$i")
touch ${mod}.txt
chmod $mod $mod.txt
n=$(ls -l $mod.txt | cut -b1-10)
mv -- $mod.txt "$n.txt"
done
It's just a permutations problem.
p=( --- --x -w- -wx r-- r-x rw- rwx ) # the set of permissions
for u in "${p[#]}"; do for g in "${p[#]}"; do for o in "${p[#]}"; do
f="task5/$u$g$o.txt"; touch -- "$f" && chmod "u=${u//-/},g=${g//-/},o=${o//-/}" -- "$f";
done; done; done
NOTE
thanks to #kvantour for pointing out I was passing dashed to chmod, and that it doesn't know what to do with them. I am surprised I wasn't getting errors.
Let's break it down and look at what's happening.
If you have any questions about what permissions sets mean or how chmod works, see here.
So for each of the user, group, or other, there are eight possible symbolic representations (representing the values of one octal digit, 0-7).
We set those into a simple array we can loop over.
p=( --- --x -w- -wx r-- r-x rw- rwx ) # the set of permissions
You can access any element with it's octal digit (technically the decimal equivalent, but that doesn't matter unless you go over 7) so ${p[5]} is r-x. Indexing with # returns the whole array, so the loops walk through them sequentially with ${p[#]}.
To get every possible permutation, we loop over them for each of user/group/other.
for u in "${p[#]}"; do # assign each permission set for the user
for g in "${p[#]}"; do # assign each permission set for the group
for o in "${p[#]}"; do # assign each permission set for the other
This is just simple iterations in nested loops to hit every permutation.
f="task5/$u$g$o.txt" # assign the permissions string AS the filename
By putting the path and filename info into a variable, we can maintain any changes in one place, and it makes the rest of the line shorter and easier to read.
touch -- "$f" && # create the file and test for success
touch will create an empty file. Because the filenames could sometimes begin with a dash (any time the permissions disallow user read), we give touch a first argument of --, which is a *NIX standard idiom meaning "options are done now, anything left is arguments"; otherwise it would try to interpret a leading dash as an invalid option set and fail. This won't be a problem while you are putting "task5/" at the beginning of the filename, but if you end up using the filename bare it would.
The && is a boolean test to see whether touch succeeded. If it did not, then we silently skip trying the chmod (touch should have emitted an error message for your debugging, but if that fails, you probably got a ton of them, and will need to fix whatever ...)
chmod "u=${u//-/},g=${g//-/},o=${o//-/}" -- "$f" # change the permissions
This uses chmod's symbolic mode. We have the permissions of each section from the nexted loops - just apply them. Again, we use the -- to tell chmod when we are done passing options so that leading dashes in filenames won't be a problem later if you refactor just just cd into the directory and create the files locally, though even then you could always prefix ./ or $PWD/ on it.
We have to get rid of thew dashes in the symbolic file modes, though, as (thanks agains #kvantour) chmod doesn't recognize those. An inline string edit works beautirully: in "u=${u//-/},g=${g//-/},o=${o//-/}", the // inside the variable spec is a replacement of all occurrences, replacing - with the nothing between the following / and }.
done; done; done # these just close each of the loops
We could (and probably should) put each of these on separate lines, but the interpreter doesn't care since we used semicolons. It lets us compact the code to put the loop nesting and closures on lines together, so long as you are comfortable with the ONE thing that's changing being obvious enough.
Anything you still have questions about that I didn't cover?
Alternate
Another version, because I like going through a loop once instead of nested shenannigans...
p=( --- --x -w- -wx r-- r-x rw- rwx ) # the set of permissions
for dec in {0..511}; do oct="$(printf "%03o" "$dec")"
u="${p[${oct:0:1}]}"; g="${p[${oct:1:1}]}"; o="${p[${oct:2:1}]}";
f="task5/$u$g$o.txt"; touch "$f"; chmod "u=${u//-/},g=${g//-/},o=${o//-/}" "$f";
done
This walks through the combinations numerically, converts decimal to octal with printf, slices each digit out of the octal permission set with basic substring parsing and uses it to look up the relevant string from the array, assign the segments, assign the resulting filename, touch/create the file, then apply the scrubbed permissions strings with chmod. It's once-through and faster, if a little harder to understand.
u="${p[${oct:0:1}]}" # grabs 1 byte from offset 0 of $oct as index to $p
As suggested, to skip the decimal to octal conversion step, replace:
for dec in {0..511}; do oct="$(printf "%03o" "$dec")"
with
for oct in {0..7}{0..7}{0..7}; do
As per Paul Hodges answer - this is a permutation question - . However, this approach uses bash with SQLite in memory database to generate a data set inspired by this web page, https://towardsdatascience.com/unix-permissions-the-easy-way-98cc19979b3e creating a cartesian product out of eight records - generating the commands and placing the octal alongside the user group all permission set
The octal for the chmod is illustrated in this approach - which more aligns with the original request
#!/bin/bash
WorkingFolder=task5
RunnerFile=_T5Builder.bash
rm -Rf ${WorkingFolder} ${RunnerFile}
mkdir ${WorkingFolder}
sqlite3 << FIN
.headers off
.mode column
.once ${RunnerFile}
with cte_elements as (
select '0' as Id, '000' as Octal, '---' as Permission union
select '1', '001', '--x' union select '2', '010', '-w-' union
select '3', '011', '-wx' union select '4', '100', 'r--' union
select '5', '101', 'r-x' union select '6', '110', 'rw-' union
select '7', '111', 'rwx'
)
, cte_allpermissions as (
select cte_elements.Id || cte_elements1.Id || cte_elements2.Id as Dec
, cte_elements.Octal || cte_elements1.Octal || cte_elements2.Octal as OctFull
, cte_elements.Permission || cte_elements1.Permission || cte_elements2.Permission as Permission
from -- cartesian product
cte_elements cross join cte_elements cte_elements1 cross join cte_elements cte_elements2 --cartesian product
)
, cte_constructor as (
select 'touch ${WorkingFolder}/' || Dec || '_' || Permission || '.txt;' as "#TouchCommand"
, 'chmod ' || Dec || ' ${WorkingFolder}/' || Dec || '_' || Permission || '.txt ;' as ChmodCommand
, 'echo -n "' || Dec || ' "' as Progress
from cte_allpermissions
)
select * from cte_constructor;
FIN
chmod +x ${RunnerFile}
echo Top And Tail of Runner File ${RunnerFile}
head -2 ${RunnerFile}
tail -2 ${RunnerFile}
source ${RunnerFile}
echo
echo Done
ls -l ${WorkingFolder} | awk '{print $1, $9}' | more
Output as follows (takes about 40 seconds - if performance is really an issue
MyNode:~/viasql$ time ./CreateAllPermFiles.bash
Top And Tail of Runner File _T5Builder.bash
touch task5/000_---------.txt; chmod 000 task5/000_---------.txt ; echo -n "000 "
touch task5/001_--------x.txt; chmod 001 task5/001_--------x.txt ; echo -n "001 "
touch task5/776_rwxrwxrw-.txt; chmod 776 task5/776_rwxrwxrw-.txt ; echo -n "776 "
touch task5/777_rwxrwxrwx.txt; chmod 777 task5/777_rwxrwxrwx.txt ; echo -n "777 "
000 001 002 003 004 005 006 007 010 011 012 013 014 015 016 017 020 021 022 023 024 025 026 027 030 031 032 033 034 035 036 037 040 041 042 043 044 045 046 047 050 051 052 053 054 055 056 057 060 061 062 063 064 065 066 067 070 071 072 073 074 075 076 ...
First few files
... 761 762 763 764 765 766 767 770 771 772 773 774 775 776 777
Done
total
---------- 000_---------.txt
---------x 001_--------x.txt
--------w- 002_-------w-.txt
--------wx 003_-------wx.txt
-------r-- 004_------r--.txt
-------r-x 005_------r-x.txt
-------rw- 006_------rw-.txt
-------rwx 007_------rwx.txt
------x--- 010_-----x---.txt
------x--x 011_-----x--x.txt
------x-w- 012_-----x-w-.txt
Related
I want to ask you for help with this problem, I have this script to detect files (I'm learning), but it doesn't detect the files and I can't see the error.
Can you guide me?
Thank you!
Folder where the script is located and executed:
home/jlia/scripts
Folder where the files are located:
home/jlia/data
Contains:
jane_contact_07292018.csv
jane_profile_07272018.doc
list.txt
list.txt
001 jane /data/jane_profile_07272018.doc
002 kwood /data/kwood_profile_04022017.doc
003 pchow /data/pcwow_profile_11042019.doc
004 janez /data/janez_ptofile_11042019.doc
005 jane /data/jane_pic_07282018.jpg
006 kwood /data/kwood_pic_04032017.jpg
007 pchow /data/pcwow_pic_05162019.jpg
008 jane /data/jane_contact_07292018.csv
009 kwood /data/kwood_contact_04042017.csv
010 pchow /data/pchow_contact_05172019.csv
#!/bin/env bash
archivos=$(grep "/jane_" ../data/list.txt | cut -d' ' -f 3);
for file2 in $archivos;
do
if [ -e "..${file2}" ];then
echo "exist ..${file2}"; else
echo "not existe ..${file2}";
fi
done
Result:
not existe ../data/jane_profile_07272018.doc
not existe ../data/jane_pic_07282018.jpg
not existe ../data/jane_contact_07292018.csv
In the home/jlia/data folder there is a text file and others, with the script I extract with grep the location of the files that I must verify if they exist and return "exist" as a result
Problem, it doesn't detect the files that exist in the script but when executing it independently [ -e "Expression" ] it detects it.
Something doesn't seem to jive between your script and your messaging.
If the files weren't where they should be, for the script to report correctly, then the setting of archivos using the grep and relative reference would fail!!!
The fact that archivos was correctly set with valid path values from list.txt, the logic should have worked.
I am unable to reproduce your results, even using the script and creating demo directory and dummy files.
My version of the script is this:
#!/bin/bash
cd ${HOME}/scripts
if [ ! -s ../data/list.txt ]
then
printf "\n WARNING: Creating DEMO Data ...\n"
mkdir ../data
cat >"../data/list.txt" <<"EnDoFiNpUt"
001 jane /data/jane_profile_07272018.doc
002 kwood /data/kwood_profile_04022017.doc
003 pchow /data/pcwow_profile_11042019.doc
004 janez /data/janez_ptofile_11042019.doc
005 jane /data/jane_pic_07282018.jpg
006 kwood /data/kwood_pic_04032017.jpg
007 pchow /data/pcwow_pic_05162019.jpg
008 jane /data/jane_contact_07292018.csv
009 kwood /data/kwood_contact_04042017.csv
010 pchow /data/pchow_contact_05172019.csv
EnDoFiNpUt
if [ ! -s ../data/jane_contact_07292018.csv ]
then
printf "\n WARNING: "
echo "Dummy Content CREATED - jane_contact_07292018.csv" | tee ../data/jane_contact_07292018.csv
fi
if [ ! -s ../data/jane_profile_07272018.doc ]
then
printf "\n WARNING: "
echo "Dummy Content CREATED - jane_profile_07272018.doc" | tee ../data/jane_profile_07272018.doc
fi
echo ""
fi
archivos=$(grep "/jane_" ../data/list.txt | cut -d' ' -f 3) ;
for file2 in $archivos
do
if [ -e "..${file2}" ]
then
echo "exist ..${file2}"
else
echo "not existe ..${file2}"
fi
done
The output generated is this:
WARNING: Creating DEMO Data ...
mkdir: cannot create directory ‘../data’: File exists
WARNING: Dummy Content CREATED - jane_contact_07292018.csv
WARNING: Dummy Content CREATED - jane_profile_07272018.doc
exist ../data/jane_profile_07272018.doc
not existe ../data/jane_pic_07282018.jpg
exist ../data/jane_contact_07292018.csv
How to write a shell script that creates empty files with all possible permissions. File names should be, for example, rwxrw_r__.txt. I know how to do it manually. As an example:
#!/bin/bash
touch task5/rwxrwxrwx.txt | chmod 777 task5/rwxrwxrwx.txt
touch task5/rwxr-xr-x.txt | chmod 755 task5/rwxr-xr-x.txt
touch task5/rwx------.txt | chmod 700 task5/rwx------.txt
touch task5/rw-rw-rw-.txt | chmod 666 task5/rw-rw-rw-.txt
touch task5/rw-r--r--.txt | chmod 644 task5/rw-r--r--.txt
touch task5/rw-------.txt | chmod 600 task5/rw-------.txt
I do not know how to write a script that will create files according to the required template and give them permissions
Do a loop:
for ((i=0; i < 512; i++)); do
mod=$(printf "%03o" "$i");
touch ${mod}.txt; chmod $mod $mod.txt;
done
Rather than trying to construct the names, if you want the names to look like the output of ls -l, just do something like
for ((i=0; i < 512; i++)); do
mod=$(printf "%03o" "$i")
touch ${mod}.txt
chmod $mod $mod.txt
n=$(ls -l $mod.txt | cut -b1-10)
mv -- $mod.txt "$n.txt"
done
It's just a permutations problem.
p=( --- --x -w- -wx r-- r-x rw- rwx ) # the set of permissions
for u in "${p[#]}"; do for g in "${p[#]}"; do for o in "${p[#]}"; do
f="task5/$u$g$o.txt"; touch -- "$f" && chmod "u=${u//-/},g=${g//-/},o=${o//-/}" -- "$f";
done; done; done
NOTE
thanks to #kvantour for pointing out I was passing dashed to chmod, and that it doesn't know what to do with them. I am surprised I wasn't getting errors.
Let's break it down and look at what's happening.
If you have any questions about what permissions sets mean or how chmod works, see here.
So for each of the user, group, or other, there are eight possible symbolic representations (representing the values of one octal digit, 0-7).
We set those into a simple array we can loop over.
p=( --- --x -w- -wx r-- r-x rw- rwx ) # the set of permissions
You can access any element with it's octal digit (technically the decimal equivalent, but that doesn't matter unless you go over 7) so ${p[5]} is r-x. Indexing with # returns the whole array, so the loops walk through them sequentially with ${p[#]}.
To get every possible permutation, we loop over them for each of user/group/other.
for u in "${p[#]}"; do # assign each permission set for the user
for g in "${p[#]}"; do # assign each permission set for the group
for o in "${p[#]}"; do # assign each permission set for the other
This is just simple iterations in nested loops to hit every permutation.
f="task5/$u$g$o.txt" # assign the permissions string AS the filename
By putting the path and filename info into a variable, we can maintain any changes in one place, and it makes the rest of the line shorter and easier to read.
touch -- "$f" && # create the file and test for success
touch will create an empty file. Because the filenames could sometimes begin with a dash (any time the permissions disallow user read), we give touch a first argument of --, which is a *NIX standard idiom meaning "options are done now, anything left is arguments"; otherwise it would try to interpret a leading dash as an invalid option set and fail. This won't be a problem while you are putting "task5/" at the beginning of the filename, but if you end up using the filename bare it would.
The && is a boolean test to see whether touch succeeded. If it did not, then we silently skip trying the chmod (touch should have emitted an error message for your debugging, but if that fails, you probably got a ton of them, and will need to fix whatever ...)
chmod "u=${u//-/},g=${g//-/},o=${o//-/}" -- "$f" # change the permissions
This uses chmod's symbolic mode. We have the permissions of each section from the nexted loops - just apply them. Again, we use the -- to tell chmod when we are done passing options so that leading dashes in filenames won't be a problem later if you refactor just just cd into the directory and create the files locally, though even then you could always prefix ./ or $PWD/ on it.
We have to get rid of thew dashes in the symbolic file modes, though, as (thanks agains #kvantour) chmod doesn't recognize those. An inline string edit works beautirully: in "u=${u//-/},g=${g//-/},o=${o//-/}", the // inside the variable spec is a replacement of all occurrences, replacing - with the nothing between the following / and }.
done; done; done # these just close each of the loops
We could (and probably should) put each of these on separate lines, but the interpreter doesn't care since we used semicolons. It lets us compact the code to put the loop nesting and closures on lines together, so long as you are comfortable with the ONE thing that's changing being obvious enough.
Anything you still have questions about that I didn't cover?
Alternate
Another version, because I like going through a loop once instead of nested shenannigans...
p=( --- --x -w- -wx r-- r-x rw- rwx ) # the set of permissions
for dec in {0..511}; do oct="$(printf "%03o" "$dec")"
u="${p[${oct:0:1}]}"; g="${p[${oct:1:1}]}"; o="${p[${oct:2:1}]}";
f="task5/$u$g$o.txt"; touch "$f"; chmod "u=${u//-/},g=${g//-/},o=${o//-/}" "$f";
done
This walks through the combinations numerically, converts decimal to octal with printf, slices each digit out of the octal permission set with basic substring parsing and uses it to look up the relevant string from the array, assign the segments, assign the resulting filename, touch/create the file, then apply the scrubbed permissions strings with chmod. It's once-through and faster, if a little harder to understand.
u="${p[${oct:0:1}]}" # grabs 1 byte from offset 0 of $oct as index to $p
As suggested, to skip the decimal to octal conversion step, replace:
for dec in {0..511}; do oct="$(printf "%03o" "$dec")"
with
for oct in {0..7}{0..7}{0..7}; do
As per Paul Hodges answer - this is a permutation question - . However, this approach uses bash with SQLite in memory database to generate a data set inspired by this web page, https://towardsdatascience.com/unix-permissions-the-easy-way-98cc19979b3e creating a cartesian product out of eight records - generating the commands and placing the octal alongside the user group all permission set
The octal for the chmod is illustrated in this approach - which more aligns with the original request
#!/bin/bash
WorkingFolder=task5
RunnerFile=_T5Builder.bash
rm -Rf ${WorkingFolder} ${RunnerFile}
mkdir ${WorkingFolder}
sqlite3 << FIN
.headers off
.mode column
.once ${RunnerFile}
with cte_elements as (
select '0' as Id, '000' as Octal, '---' as Permission union
select '1', '001', '--x' union select '2', '010', '-w-' union
select '3', '011', '-wx' union select '4', '100', 'r--' union
select '5', '101', 'r-x' union select '6', '110', 'rw-' union
select '7', '111', 'rwx'
)
, cte_allpermissions as (
select cte_elements.Id || cte_elements1.Id || cte_elements2.Id as Dec
, cte_elements.Octal || cte_elements1.Octal || cte_elements2.Octal as OctFull
, cte_elements.Permission || cte_elements1.Permission || cte_elements2.Permission as Permission
from -- cartesian product
cte_elements cross join cte_elements cte_elements1 cross join cte_elements cte_elements2 --cartesian product
)
, cte_constructor as (
select 'touch ${WorkingFolder}/' || Dec || '_' || Permission || '.txt;' as "#TouchCommand"
, 'chmod ' || Dec || ' ${WorkingFolder}/' || Dec || '_' || Permission || '.txt ;' as ChmodCommand
, 'echo -n "' || Dec || ' "' as Progress
from cte_allpermissions
)
select * from cte_constructor;
FIN
chmod +x ${RunnerFile}
echo Top And Tail of Runner File ${RunnerFile}
head -2 ${RunnerFile}
tail -2 ${RunnerFile}
source ${RunnerFile}
echo
echo Done
ls -l ${WorkingFolder} | awk '{print $1, $9}' | more
Output as follows (takes about 40 seconds - if performance is really an issue
MyNode:~/viasql$ time ./CreateAllPermFiles.bash
Top And Tail of Runner File _T5Builder.bash
touch task5/000_---------.txt; chmod 000 task5/000_---------.txt ; echo -n "000 "
touch task5/001_--------x.txt; chmod 001 task5/001_--------x.txt ; echo -n "001 "
touch task5/776_rwxrwxrw-.txt; chmod 776 task5/776_rwxrwxrw-.txt ; echo -n "776 "
touch task5/777_rwxrwxrwx.txt; chmod 777 task5/777_rwxrwxrwx.txt ; echo -n "777 "
000 001 002 003 004 005 006 007 010 011 012 013 014 015 016 017 020 021 022 023 024 025 026 027 030 031 032 033 034 035 036 037 040 041 042 043 044 045 046 047 050 051 052 053 054 055 056 057 060 061 062 063 064 065 066 067 070 071 072 073 074 075 076 ...
First few files
... 761 762 763 764 765 766 767 770 771 772 773 774 775 776 777
Done
total
---------- 000_---------.txt
---------x 001_--------x.txt
--------w- 002_-------w-.txt
--------wx 003_-------wx.txt
-------r-- 004_------r--.txt
-------r-x 005_------r-x.txt
-------rw- 006_------rw-.txt
-------rwx 007_------rwx.txt
------x--- 010_-----x---.txt
------x--x 011_-----x--x.txt
------x-w- 012_-----x-w-.txt
I have this .sh script that goes through every folder in a parent folder and runs program in each. The code I used was the following:
for d in ./*/
do cp program "$d"
(cd "$d" ; ./program)
done
program, among other things, gets the name of each folder and writes it to a file data.dat, so that all folder names are listed there. These folders' names are numbers (decimal) that identify their contents. program writes the folder name to data.dat when it enters each folder, so that they will appear in the order that Bash goes through the folders.
I want them to be sorted, in data.dat, in alphabetical order, putting lower numbers before higher, regardless of being a 1-digit or 2-digit number. For example, I want 2.32 to come before 10.43 and not the other way around.
The problem, it seems, is that for Bash the . comes after numbers in the order.
How can I change it to come before numbers?
Thanks in advance!
EDIT:
program is in Fortran 77 and goes like this:
` program getData
implicit none
character counter*20, ac*4, bash*270, Xname*4, fname*15
double precision Qwallloss, Qrad, Nrad, Qth, QreacSUM
double precision Xch4in, Ych4in, length, porosity, Uin, RHOin
double precision MFLR, Area, Xvalue
integer I
bash="printf '%s\n'"//' "${PWD##*/}" > RunNumber.txt'
call system(bash) !this gets the folder name and writes
!to RunNumber.txt
open(21, form="FORMATTED", STATUS="OLD", FILE="RunNumber.txt")
rewind(21)
read(21,*) counter !brings the folder name into the program
close(21)
`
(...)
`
call system(' cp -rf ../PowerData.dat . ')
open(27, form="FORMATTED", STATUS="OLD", ACCESS="APPEND", !the new row is appended to the existing file
1 FILE="PowerData.dat")
write(27,600) Counter, Xvalue, Nrad, Qrad, Qth, !writes a row of variables,
1 Area, MFLR, Uin, RHOin, Xch4in, Ych4in !starting with the folder name,
!to the Data file
close(27)
call system('cp -rf PowerData.dat ../')
end program`
I expect that your program will in the future do perhaps a bit more and therefore I made the second loop.
for d in ./*/ ; do
echo "$d"a >> /tmp/tmpfile
done
for d in $(sort -n /tmp/tmpfile) ; do
cp program "$d"
(cd "$d" ; ./program)
done
There are more ways to do this; for example:
for d in $(ls | sort -n) ; do
(some will castigate me for parsing the output of ls) etcetera.
So if you do:
mkdir test
cd test
touch 100
touch 2.00
touch 50.1
ls will give you
100 2.00 50.1
ls | sort -n will give you
2.00
50.1
100
and, as a bonus, ls -v will give you
2.00 50.1 100
I have log files for year's data , I want to read inside the log file and take the unix timestamp column and convert to human readable timestamp(not literally, only logically) . Depends on the timestamp .
I want to create a Time based directory structure Something like this ( 2017/05/20/13 ) (YYYY/MM/DD/HH) and copy those files to respective directory structure
LANG=C
if [[ -z "$1" ]]
then
if [[ -p /dev/stdin ]] # input from a pipe
then
read -r p
else
echo "No timestamp given." >&2
exit
fi
else
p=$1
fi
awk 'BEGIN {FS=OFS=":"}{$7=strftime("%Y-%m-%d %H:%M:%S",$7)} {print}'
sample.txt
this is the Log file sample output after the awk command , Now I want to create a folder depends on the date and hour . Something like this
Input
counter.txt
28:-58:31147:28:0:0:2017-05-24 15:12:50:3064:0:0:0:103
28:-58:31147:28:0:0:2017-05-24 15:12:50:3064:0:0:0:102
21:-60:17086:28:0:0:2017-05-24 15:12:50:1384:0:0:0:102
10:-64:16651:28:0:0:2017-05-24 16:12:50:656:0:0:0:103
folder structure be like
[root#master 24-05-17]# ls -ltr | grep counter_
-rw-rw-r-- 1 user user 11974276 May 23 12:32 counter_25.txt
-rw-rw-r-- 1 user user 17271141 May 23 13:32 counter_24.txt
-rw-rw-r-- 1 user user 16819152 May 23 14:32 counter_23.txt
output is like :
/2017/05/24/15 - in this folder , we have to seperate the input file and write
counter_2017_05_24_15.txt
28:-58:31147:28:0:0:2017-05-24 15:12:50:3064:0:0:0:103
28:-58:31147:28:0:0:2017-05-24 15:12:50:3064:0:0:0:102
21:-60:17086:28:0:0:2017-05-24 15:12:50:1384:0:0:0:102
/2017/05/24/16 in this folder , I want to write the remaining data in another
new file and store seperately under the folder
counter_2017_05_24_16.txt
10:-64:16651:28:0:0:2017-05-24 16:12:50:656:0:0:0:103
23:-66:33444:28:0:0:2017-05-24 16:12:50:656:0:0:0:103
10:-64:16651:28:0:0:2017-05-24 16:12:50:656:0:0:0:102
Instead of this:
strftime("%Y-%m-%d %H:%M:%S")
Do this:
strftime("%d/%m/%Y")
Now you have a DD/MM/YYYY string, and can use it as part of a path.
I have been trying to understand what exactly is happening in the below mentioned code. But i am not able to understand it.
$mode = (stat($filename))[2];
printf "Permissions are %04o\n", $mode & 07777;
Lets say my $mode value is 33188
$mode & 07777 yields a value = 420
is the $mode value a decimal number ?
why we are choosing 07777 and why we are doing a bitwise and operation. I am not able to underand the logic in here.
The mode from your question corresponds to a regular file with 644 permissions (read-write for the owner and read-only for everyone else), but don’t take my word for it.
$ touch foo
$ chmod 644 foo
$ perl -le 'print +(stat "foo")[2]'
33188
The value of $mode can be viewed as a decimal integer, but doing so is not particularly enlightening. Seeing the octal representation gives something a bit more familiar.
$ perl -e 'printf "%o\n", (stat "foo")[2]'
100644
Bitwise AND with 07777 gives the last twelve bits of a number’s binary representation. With a Unix mode, this operation gives the permission or mode bits and discards any type information.
$ perl -e 'printf "%d\n", (stat "foo")[2] & 07777' # decimal, not useful
420
$ perl -e 'printf "%o\n", (stat "foo")[2] & 07777' # octal, eureka!
644
A nicer way to do this is below. Read on for all the details.
Mode Bits
The third element returned from stat (which corresponds to st_mode in struct stat) is a bit field where the different bit positions are binary flags.
For example, one bit in st_mode POSIX names S_IWUSR. A file or directory whose mode has this bit set is writable by its owner. A related bit is S_IROTH that when set means other users (i.e., neither the owner nor in the group) may read that particular file or directory.
The perlfunc documentation for stat gives the names of commonly available mode bits. We can examine their values.
#! /usr/bin/env perl
use strict;
use warnings;
use Fcntl ':mode';
my $perldoc_f_stat = q(
# Permissions: read, write, execute, for user, group, others.
S_IRWXU S_IRUSR S_IWUSR S_IXUSR
S_IRWXG S_IRGRP S_IWGRP S_IXGRP
S_IRWXO S_IROTH S_IWOTH S_IXOTH
# Setuid/Setgid/Stickiness/SaveText.
# Note that the exact meaning of these is system dependent.
S_ISUID S_ISGID S_ISVTX S_ISTXT
# File types. Not necessarily all are available on your system.
S_IFREG S_IFDIR S_IFLNK S_IFBLK S_IFCHR S_IFIFO S_IFSOCK S_IFWHT S_ENFMT
);
my %mask;
foreach my $sym ($perldoc_f_stat =~ /\b(S_I\w+)\b/g) {
my $val = eval { no strict 'refs'; &$sym() };
if (defined $val) {
$mask{$sym} = $val;
}
else {
printf "%-10s - undefined\n", $sym;
}
}
my #descending = sort { $mask{$b} <=> $mask{$a} } keys %mask;
printf "%-10s - %9o\n", $_, $mask{$_} for #descending;
On Red Hat Enterprise Linux and other operating systems in the System V family, the output of the above program will be
S_ISTXT - undefined
S_IFWHT - undefined
S_IFSOCK - 140000
S_IFLNK - 120000
S_IFREG - 100000
S_IFBLK - 60000
S_IFDIR - 40000
S_IFCHR - 20000
S_IFIFO - 10000
S_ISUID - 4000
S_ISGID - 2000
S_ISVTX - 1000
S_IRWXU - 700
S_IRUSR - 400
S_IWUSR - 200
S_IXUSR - 100
S_IRWXG - 70
S_IRGRP - 40
S_IWGRP - 20
S_IXGRP - 10
S_IRWXO - 7
S_IROTH - 4
S_IWOTH - 2
S_IXOTH - 1
Bit twiddling
The numbers above are octal (base 8), so any given digit must be 0-7 and has place value 8n, where n is the zero-based number of places to the left of the radix point. To see how they map to bits, octal has the convenient property that each digit corresponds to three bits. Four, two, and 1 are all exact powers of two, so in binary, they are 100, 10, and 1 respectively. Seven (= 4 + 2 + 1) in binary is 111, so then 708 is 1110002. The latter example shows how converting back and forth is straightforward.
With a bit field, you don’t care exactly what the value of a bit in that position is but whether it is zero or non-zero, so
if ($mode & $mask) {
tests whether any bit in $mode corresponding to $mask is set. For a simple example, given the 4-bit integer 1011 and a mask 0100, their bitwise AND is
1011
& 0100
------
0000
So the bit in that position is clear—as opposed to a mask of, say, 0010 or 1100.
Clearing the most significant bit of 1011 looks like
1011 1011
& ~(1000) = & 0111
------
0011
Recall that ~ in Perl is bitwise complement.
For completeness, set a bit with bitwise OR as in
$bits |= $mask;
Octal and file permissions
An octal digit’s direct mapping to three bits is convenient for Unix permissions because they come in groups of three. For example, the permissions for the program that produced the output above are
-rwxr-xr-x 1 gbacon users 1096 Feb 24 20:34 modebits
That is, the owner may read, write, and execute; but everyone else may read and execute. In octal, this is 755—a compact shorthand. In terms of the table above, the set bits in the mode are
S_IRUSR
S_IWUSR
S_IXUSR
S_IRGRP
S_IXGRP
S_IROTH
S_IXOTH
We can decompose the mode from your question by adding a few lines to the program above.
my $mode = 33188;
print "\nBits set in mode $mode:\n";
foreach my $sym (#descending) {
if (($mode & $mask{$sym}) == $mask{$sym}) {
print " - $sym\n";
$mode &= ~$mask{$sym};
}
}
printf "extra bits: %o\n", $mode if $mode;
The mode test has to be more careful because some of the masks are shorthand for multiple bits. Testing that we get the exact mask back avoids false positives when some of the bits are set but not all.
The loop also clears the bits from all detected hits so at the end we can check that we have accounted for each bit. The output is
Bits set in mode 33188:
- S_IFREG
- S_IRUSR
- S_IWUSR
- S_IRGRP
- S_IROTH
No extra warning, so we got everything.
That magic 07777
Converting 77778 to binary gives 0b111_111_111_111. Recall that 78 is 1112, and four 7s correspond to 4×3 ones. This mask is useful for selecting the set bits in the last twelve. Looking back at the bit masks we generated earlier
S_ISUID - 4000
S_ISGID - 2000
S_ISVTX - 1000
S_IRWXU - 700
S_IRWXG - 70
S_IRWXO - 7
we see that the last 9 bits are the permissions for user, group, and other. The three bits preceding those are the setuid, setgroupid, and what is sometimes called the sticky bit. For example, the full mode of sendmail on my system is -rwxr-sr-x or 3428510. The bitwise AND works out to be
(dec) (oct) (bin)
34285 102755 1000010111101101
& 4095 = & 7777 = & 111111111111
------- -------- ------------------
1517 = 2755 = 10111101101
The high bit in the mode that gets discarded is S_IFREG, the indicator that it is a regular file. Notice how much clearer the mode expressed in octal is when compared with the same information in decimal or binary.
The stat documentation mentions a helpful function.
… and the S_IF* functions are
S_IMODE($mode)
the part of $mode containing the permission bits and the setuid/setgid/sticky bits
In ext/Fcntl/Fcntl.xs, we find its implementation and a familiar constant on the last line.
void
S_IMODE(...)
PREINIT:
dXSTARG;
SV *mode;
PPCODE:
if (items > 0)
mode = ST(0);
else {
mode = &PL_sv_undef;
EXTEND(SP, 1);
}
PUSHu(SvUV(mode) & 07777);
To avoid the bad practice of magic numbers in source code, write
my $permissions = S_IMODE $mode;
Using S_IMODE and other functions available from the Fcntl module also hides the low-level bit twiddling and focuses on the domain-level information the program wants. The documentation continues
S_IFMT($mode)
the part of $mode containing the file type which can be bit-anded with (for example) S_IFREG or with the following functions
# The operators -f, -d, -l, -b, -c, -p, and -S.
S_ISREG($mode) S_ISDIR($mode) S_ISLNK($mode)
S_ISBLK($mode) S_ISCHR($mode) S_ISFIFO($mode) S_ISSOCK($mode)
# No direct -X operator counterpart, but for the first one
# the -g operator is often equivalent. The ENFMT stands for
# record flocking enforcement, a platform-dependent feature.
S_ISENFMT($mode) S_ISWHT($mode)
Using these constants and functions will make your programs clearer by more directly expressing your intent.
It is explained in perldoc -f stat, which is where I assume you found this example:
Because the mode contains both the file type and its
permissions, you should mask off the file type portion and
(s)printf using a "%o" if you want to see the real permissions.
The output of printf "%04o", 420 is 0644 which is the permissions on your file. 420 is just the decimal representation of the octal number 0644.
If you try and print the numbers in binary form, it is easier to see:
perl -lwe 'printf "%016b\n", 33188'
1000000110100100
perl -lwe 'printf "%016b\n", 33188 & 07777'
0000000110100100
As you'll notice, bitwise and removes the leftmost bit in the number above, which presumably represents file type, leaving you with only the file permissions. This number 07777 is the binary number:
perl -lwe 'printf "%016b\n", 07777'
0000111111111111
Which acts as a "mask" in the bitwise and. Since 1 & 1 = 1, and 0 & 1 = 0, it means that any bit that is not matched by a 1 in 07777 is set to 0.