This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
File names with spaces in BASH
(6 answers)
Closed 4 years ago.
Im trying to run trough a directory showing file sizes, but when i try to print the size of a file that has spaces stat command fails. How can i fix this?
#!/bin/bash
for file in /home/user/Desktop/*; do
fileSize=$(stat -c%s $file)
echo $fileSize
done
Use quotes.
#!/bin/bash
for file in /home/user/Desktop/*; do
fileSize=$(stat -c%s "$file")
echo $fileSize
done
Adjusted for my desktop -
$: for file in *
do case "$f" in
*\ *) printf "'$f': %d\n" $(stat -c%s "$file")
esac
done
'BPS Stuff': 0
'New TWC Account - Hodges Paul A.msg': 37888
'Nov FTOTD calendar.JPG': 138769
'OCA Web Client - ASAP.lnk': 2406
'Paul - Copy.png': 64915
'Solstice Client.lnk': 2165
'VIP Access.lnk': 2079
You should add quotes around variables to make sure if there are spaces in there that they get picked up correctly:
#!/bin/bash
for file in /home/user/Desktop/*; do
fileSize=$(stat -c%s "$file")
echo $fileSize
done
What bash does is it simply replaces $var with the thing in $var. If that contains spaces it becomes something else then you intended, because spaces are used in bash to separate command options.
Consider the following example:
file="-l -h -a -F"
ls $file
This gets parsed as:
ls -l -h -a -F
The output will not be the just the file "-l -h -a -F" but it will get parsed as options for ls and it will show the current directory listing. If you had put quotes around $file like so:
file="-l -h -a -F"
ls "$file"
It will get parsed like:
ls "-l -h -a -F"
ls will search for the file "-l -h -a -F" and show only that one file (assuming it exists, it errors otherwise).
Related
I am attempting to curl multiple urls in a bash command. Eventually I will be curling a large number of Urls so I am using xargs to use multiple processes to speed up the process.
My file consists of x number of URLs:
https://someurl.com
https://someotherurl.com
My issue comes when attempting to output the results to separate files named after the URLs I curl.
The bash command I have is:
xargs -P 5 -n 1 -I% curl -k -L % -0 % < urls.txt
When I run this I get 'Failed to create file https://someotherurl.com'
You cannot create a file with / in the filename. You could do it this way:
#!/bin/bash
while IFS= read -r line
do
echo "LINE: $line"
if [[ "$line" != "" ]]
then
filename="${line#https://}"
echo "FILENAME: $filename"
# YOUR CURL COMMAND HERE, USING $filename
fi
done < url.txt
it ignores empty lines
variable substitution is used to remove the https:// part of each URL
this will allow you to create the file
Note: if your URLs containt sub-directories, they must be removed as well.
Ex: you want to do https://www.exemple.com/some/sub/dir
The script I suggested here would try to create a file named "www.exemple.com/some/sub/dir". In this case, you could replace the / with _ using tr.
The script would become:
#!/bin/bash
while IFS= read -r line
do
echo "LINE: $line"
if [[ "$line" != "" ]]
then
filename=$(echo "$line" | tr '/' '_')
filename2=${filename#https:__}
echo "FILENAME: $filename2"
# YOUR CURL COMMAND HERE, USING $filename2
fi
done < url.txt
Because your question is ambiguous, I would assume:
You have a file urls.txt that contains URLs separated by LF.
You want to download all URLs by curl and use each URL as its filename.
Unfortunately, that's not possible because URL contains invalid characters like slash /. Alternatively, for this case, I would suggest you use Bsse64 safe mode to decode URL before saving to file based on RFC 3548.
After applying this requirement, your script would become like:
seq 100 | xargs -I# echo 'https://example.com?#' > urls.txt
xargs -P0 -L1 sh -c 'curl -SskL0 -o $(printf %s "$1" | uuencode -m /dev/stdout | sed "1d;\$d" | tr +/ -_) "$1"' sh < urls.txt
I have a folder structure like the following:
2020-123-1
2020-123-2
2020-123-3
2020-124-1
2020-124-2
...
I need to create folders from the first 2 numbers and omit whatever follows the second dash (-). Then I need to put the prior folders under the newly created ones with the correct name.
2020-123
->2020-123-1
->2020-123-2
->2020-123-3
2020-124
->2020-124-1
->2020-124-2
I tried to write a script in bash like this:
ls -d */ > folder.txt
cut -f1,2 -d"-" folder.txt |cut -f1 -d"/" |sort|uniq > mainfolder.txt
while read line; do mkdir $line ; done < mainfolder.txt
while read line; do mv $(cut -f1,2 -d"-" $line) $line/ ; done < folder.txt
I couldn't make the last line work, I know it has issues.
Actually, you don't have to parse the directory names and build the hierarchy. You can make use of the -p option of mkdir, thus, an awk one-liner will do the job:
awk -F'-' '{top=$1 FS $2;printf "mkdir -p %s; mv %s %s\n",top, $0, top}' dir.txt
The output with your example:
mkdir -p 2020-123; mv 2020-123-1 2020-123
mkdir -p 2020-123; mv 2020-123-2 2020-123
mkdir -p 2020-123; mv 2020-123-3 2020-123
mkdir -p 2020-124; mv 2020-124-1 2020-124
mkdir -p 2020-124; mv 2020-124-2 2020-124
Note
This one-liner just print the commands without executing them, you just pipe the output to |sh if everything looks fine. Examine the output commands, change the printf format/values for adjustment.
I didn't quote the filenames, since your example doesn't contain any special chars. Do it if it is in the case.
So the final script is as follows:
ls -d */ | cut -f1 -d"/" > folder.txt
awk -F'-' '{top=$1 FS $2;printf "mkdir -p %s; mv %s %s\n",top, $0, top}' folder.txt |sh
In pure bash:
#!/bin/bash
for src in *-*-*; do
destdir=${src%-*}
[[ -d $destdir ]] || mkdir "$destdir" || exit
# This just prints out the command that will be called.
# Remove the "echo" in actual script after making sure it will run as intented
echo mv "$src" "$destdir"
done
In the script above it is assumed that each file name to be moved contains exactly two dashes. If it can contain two or more dashes then the destdir=${src%-*} line should be replaced with these two lines:
suffix=${src#*-*-}
destdir=${src%"-$suffix"}
For detailed information read the "shell parameter expansion" section in bash reference.
Additionally, a good read article is: Why you shouldn't parse the output of ls
I want to list the files recursively in the HOME directory. I'm trying to write my own script , so I should not use the command find or ls. My script is:
#!/bin/bash
minSize=102400;
printFiles() {
for x in "$1/"*; do
if [ -d "$x" ]; then
printFiles "$x";
else
size=$(wc -c "$x");
if [[ "$size" -gt "$minSize" ]]; then
echo "$size";
fi
fi
done
}
printFiles "/~";
So, the problem here is that when I run this script, the terminal throws Line 11: division by 0 and /home/gandalf/Videos/*: No such file or directory. I have not divided by any number, why I'm getting this error?. And the second one?
Alternatively, I can't use find or ls because I have to display the files one by one asking to the user if he want to see the next file or not. This is possible using the command find or ls or only can be done writing my own function?
Thanks.
size=$(wc -c "$x");
That's the line that is failing. When you run that wc command manually you should be able to see why:
$ wc -c /tmp/out
5 /tmp/out
The output contains not only the file size but also the file name. So you can't use $size with the -gt comparator on the next line. One way to fix that is to change the wc line to use cut (or awk, or sed, etc) to keep just the file size.
size=$(wc -c "$x" | cut -f1 -d " ")
A simpler alternative suggested by #mklement0:
size=$(wc -c < "$x")
I am trying to write a bash script to scan for authorized_keys files and remove the keys of a couple previous employees if found. I am having one heck of a time figuring out the escaping for the sed command at the end. I am using commas instead of / since / can show up in the ssh-key. Any help would be appreciated
#!/bin/bash
declare -A keys
keys["employee1"]='AAAAB3NzaC1yc2EAAAABJQAAAIEAxoZ7ZdpJkL98n8cSTkFBwaAeSNK0m/tOWtF1mu5NAzMM/+1SDO6rJH/ruyyqBJo9s+AHWZLGRHfXT2XBg2SRaUnubAKp0w6qNIbej0MsA/ifAs8AfVGdj0pUPLtKpo6XVZkB8vEZSIQ+xNk1n5HJrGJnFGWKWeY3z1/KOLxcLHU='
keys["employee2"]='AAAAB3NzaC1yc2EAAAABIwAAAQEAwHYNAVhb319OBVXPhYF8cSTkFBwaAekr7UcKjfLPCHMpz19W0L/C0g+75Hn8COxOQILDUhIPhYHXOduQjGD/6NXgJDWxgyT00Azg5BREUnBd58WqZPlEvTZYlAgmdMIbnWPPGdJwzqKH/k7/STK6vTKxL6rxBo4lSNK0m/tOWtF1mu5NAzMM/+1SDO6rJH/ruyyqBJo9s+NIbej0MsA/ifAs8AfAkfO2JjgeQpJMyZ7B02XVN5iSLAyC3Cb5FXIjJuk4LPhcApuVyszH2lgve0r5bt/nFgVujJTvJTHPlGrqkYDcDJVUtfbjoLqGPrnpijp6rGIC7aFDDe7bk0ygHYMXDFWcjJBerfLGUWTYWFFLY3bfiO/h/9oEycmQHyB2co4a0IyyDnaYn9OY6xsRRATVlk4Q=='
files=`find / -name authorized_keys`
echo "Checking Authorized_Keys files on: " `hostname`
echo ""
echo "Located files: "
for file in $files; do
echo " $file"
done
echo""
for file in $files; do
for key in "${!keys[#]}"; do
if grep -q ${keys[$key]} $file; then
echo " *** Removing $key from $file"
sed "s,${keys[$key]},d" $file
fi
done
done
You've made it a bit complicated I think.
You can do this using grep -vf and process substitution:
# array to hold the value you want to remove
keys=(
'AAAAB3NzaC1yc2EAAAABJQAAAIEAxoZ7ZdpJkL98n8cSTkFBwaAeSNK0m/tOWtF1mu5NAzMM/+1SDO6rJH/ruyyqBJo9s+AHWZLGRHfXT2XBg2SRaUnubAKp0w6qNIbej0MsA/ifAs8AfVGdj0pUPLtKpo6XVZkB8vEZSIQ+xNk1n5HJrGJnFGWKWeY3z1/KOLxcLHU=',
'AAAAB3NzaC1yc2EAAAABIwAAAQEAwHYNAVhb319OBVXPhYF8cSTkFBwaAekr7UcKjfLPCHMpz19W0L/C0g+75Hn8COxOQILDUhIPhYHXOduQjGD/6NXgJDWxgyT00Azg5BREUnBd58WqZPlEvTZYlAgmdMIbnWPPGdJwzqKH/k7/STK6vTKxL6rxBo4lSNK0m/tOWtF1mu5NAzMM/+1SDO6rJH/ruyyqBJo9s+NIbej0MsA/ifAs8AfAkfO2JjgeQpJMyZ7B02XVN5iSLAyC3Cb5FXIjJuk4LPhcApuVyszH2lgve0r5bt/nFgVujJTvJTHPlGrqkYDcDJVUtfbjoLqGPrnpijp6rGIC7aFDDe7bk0ygHYMXDFWcjJBerfLGUWTYWFFLY3bfiO/h/9oEycmQHyB2co4a0IyyDnaYn9OY6xsRRATVlk4Q=='
)
while IFS= read -d '' -r file; do
grep -vf <(printf "%s\n" "${keys[#]}") "$file" > "$file.tmp"
mv "$file.tmp" "$file"
done < <(find / -name authorized_keys -print0)
In your case, it's easy, just need to use a sign which not contained in base64 code as the delimiter, eg |:
sed "\|${keys[$key]}|d" $file
Explanation in the sed manual:
\%regexp%
(The % may be replaced by any other single character.)
This also matches the regular expression regexp, but allows one to use a different delimiter than /.
I apologize if this is a trivial question. I am learning how to use linux bash and this little task is giving me a headache...
So I need to write a script, let's call it count.sh. I want that: for each file in the working directory, prints the filename, the number of lines, and the number of words to the console:
test.txt 100 1023
someOtherfiles 10 233
So far, I know that the following gives me all the files names in the directory. And thanks for all who helped me, I get this working version:
for f in *; do
echo -n "$f"
cat "$f" | wc -wl
done
I would really appreciate your help! Thanks ahead!
P.s. If you know great resources (links for tutorials) for learning about script and you are willing to share it with me. I think I really need to know these basics. Thanks again!
If you must have the file name as the first field in your output, try this:
for f in *; do
if [ -f "$f" ]; then
echo -n "$f"
cat "$f" | wc -wl
fi
done
for f in *; do
if [[ -f $f ]]; then
echo "$f $(wc -wl < "$f")"
fi
done
[[ -f $f ]] processes only files (excludes subdirectories) and also handles the case where the directory is empty (in which case * is (by default) left unexpanded, i.e. assigned to $f as is).
echo "$f $(wc -wl < "$f")" uses command substitution ($( ... )) to directly include the output from the enclosed command in the output string passed to echo.
Note that the reason that < is used to direct the content of file $f to wc via stdin is that wc would otherwise append the name of the input file to its output (thanks, #R Sahu).