I have the follwoing bash code i
declare -A matrix
num_rows=6
num_columns=1
for ((i=1;i<=num_rows;i++)) do
for ((j=1;j<=num_columns;j++)) do
if [[ i -eq 1 ]]; then
matrix[$i,$j]= echo $i
else
matrix[$i,$j]= echo $j
fi
done
done >> out.dat
This code directs the outputs to dat file
but the problem when I rerun the code, the old outputs are removed and replaced by the new outputs, How can I keep the old outputs and save the new too?
> "out.dat"
truncates the file to zero size, essentially removing all its content.
If you want to preserve the previous old file you can move it:
mv out.dat out.dat.old
> "out.dat"
If you want to preserve all the old files, you can save them with a reasonably unique names such as using the date command:
mv out.dat out.dat.old.$(date +%s%2N)
> "out.dat"
The truncation may or may not be necessary depending whether your rest of code expects an empty or creates one if it doesn't exist.
Related
I need to have multiple temporary files. I decided that an array is best for that because I indeed to create 10 temporary files, use them and them remove the 10 files. From researching I've come up with this:
declare -A my_array
my_array=()
for i in `seq -w 1 10`
do
my_array[$i]= $(mktemp /tmp/$i.XXXX)
done
#Do stuff with the files in the array
for i in `seq -w 1 10`
do
rm my_array[$i]
done
However, this gives me the error:
./plot.sh: line 7: /tmp/01.PkUG: Permission denied
./plot.sh: line 7: /tmp/02.eFNZ: Permission denied And so on...
I'm confused because when I create the 10 files without the loop it works fine, but is obviously very messy:
tmpfile1=$(mktemp /tmp/data1.XXX)
tmpfile2=$(mktemp /tmp/data2.XXX)
And so on...
#And then remove
rm $tmpfile1
rm $tmpfile2
And so on....
You have a couple of syntax errors which I've marked below:
declare -A my_array
my_array=()
for i in `seq -w 1 10`
do
my_array[$i]=$(mktemp /tmp/$i.XXXX)
# ^^
# | no space
done
#Do stuff with the files in the array
for i in `seq -w 1 10`
do
rm "${my_array[$i]}"
# ^^^ ^^
# | | dollar sign and curly braces required, quotes recommended
done
Try using ShellCheck to check your scripts for errors. It has better diagnostics than the shell's built-in ones. It can be downloaded as a CLI tool, or you can just paste your script into the web site. Pretty convenient!
Some additional improvements:
There's no need to use declare -A when you've got a regular non-associative array.
for ((i = 0; i < n; i++)) avoids an unnecessary call to an external process like seq.
You can append to an array with array+=(items...).
You can often avoid explicitly looping over arrays. Many commands, rm included, take lists of file names, which you can use to your advantage. "${array[#]}" expands to all of the items in the array.
There's no real need to micromanage mktemp's file name generation. Letting it use the default algorithm is nice because it'll respect the user's $TMPDIR setting in case they want to use a directory other than /tmp. (If you do want to control the file name use --tmpdir to get the same behavior.)
files=()
for ((i = 1; i <= 10; i++)); do
files+=("$(mktemp)") # or: files+=("$(mktemp --tmpdir "$i".XXXX)")
done
# Do stuff with files in the array.
rm "${files[#]}"
I am new to scripting with bash, and I am trying to make a user friendly script to download videos using youtube-dl.
After getting the available streams from a video link, i want to write to a file the available resolutions for the given video. This is my code:
#!/bin/bash
youtube-dl -F https://www.youtube.com/watch?v=05X0RRmUtE0 > info.txt
filename='info.txt'
echo Start
cat > res.txt
echo "" > res.txt
while read p; do
for i in '240p' '360p' '480p' '720p' '1080p' '2160p'; do
while read q; do
if [[$i == $q]]; then
break
else
echo $i >> res.txt
fi
done < res.txt
done
done < $filename
With the current script, '240p' endlessly gets appended to 'res.txt'. I am not able to figure out where i am going wrong
UPDATE
i have edited the code as follows:
while read p ; do
for i in '240p' '360p' '480p' '720p' '1080p' '2160p' ; do
if [[ "$p" == *"$i"* ]]; then
while read q; do
if [[ "$q" == "$i" ]]; then
break
else
echo $i >> res.txt
fi
done < res.txt
fi
done
done < info.txt
but i still do not get the ouput I want
also, for those who do not understand what the script is supposed to do:
1) input a link and extract the available streams to a text file ( youtube-dl -F https://www.youtube.com/watch?v=05X0RRmUtE0 > info.txt)
2) append recurring resolutions from the streams in info.txt to another file res.txt only once
I'll ignore the outer loop (read p ... < $filename) since it does not impact the issue you're asking about, but you should probably check why it's there because it doesn't do anything right now.
The first inner loop (for i in '240p' '360p' ...) loops over that list, and performs the second inner loop for every value.
The second inner loop (read q ... < res.txt) reads res.txt, and compares every line to the value from the first inner loop. If the values match, the second inner loop stops, and the first inner loop continues with the next value, restarting the second inner loop at the beginning of the file.
If the values do not match, you add '240p' to the res.txt file. If it never finds a matching value, it will keep going endlessly, because it does not simply use the original contents of the file, it also reads lines added to this file during the process.
This means that, if the res.txt file does not contain one of the values of the first inner loop, it will get stuck at that value, and keep adding '240p' endlessly, except if the original file does not contain '240p', in which case it will read the entire file, adding a '240p' line for every value it already contains, then break when it encounters the first '240p' it added.
The only way this code will not get stuck is if res.txt contains all of the values from the first inner loop, or all except '240p' (which it will add, then finish).
EDIT:
As for a solution, I'd throw out the entire inner while loop, replacing it with grep:
#!/bin/bash
youtube-dl -F https://www.youtube.com/watch?v=05X0RRmUtE0 > info.txt
echo Start
rm res.txt >/dev/null 2>&1 # remove res.txt if it exists, don't care about warnings if it doesn't exist
touch res.txt # create empty file
while read p ; do
for i in '240p' '360p' '480p' '720p' '1080p' '2160p' ; do
if [[ "$p" =~ "$i" ]]; then # regex matching ensures foo480pbar matches '480p'
if ! grep "$p" res.txt >/dev/null 2>&1; then # if grep fails, res.txt does not contain $p yet
echo $i >> res.txt
fi
fi
done
done < info.txt
You could even throw out the for loop and put all the resolutions in one regex match, and combine both if statements into one, but this way it's easier to read what's going on, so I'll leave that as an exercise to the reader 😉
I am still a newbie in shell scripting and trying to come up with a simple code. Could anyone give me some direction here. Here is what I need.
Files in path 1: /tmp
100abcd
200efgh
300ijkl
Files in path2: /home/storage
backupfile_100abcd_str1
backupfile_100abcd_str2
backupfile_200efgh_str1
backupfile_200efgh_str2
backupfile_200efgh_str3
Now I need to delete file 300ijkl in /tmp as the corresponding backup file is not present in /home/storage. The /tmp file contains more than 300 files. I need to delete the files in /tmp for which the corresponding backup files are not present and the file names in /tmp will match file names in /home/storage or directories under /home/storage.
Appreciate your time and response.
You can also approach the deletion using grep as well. You can loop though the files in /tmp checking with ls piped to grep, and deleting if there is not a match:
#!/bin/bash
[ -z "$1" -o -z "$2" ] && { ## validate input
printf "error: insufficient input. Usage: %s tmpfiles storage\n" ${0//*\//}
exit 1
}
for i in "$1"/*; do
fn=${i##*/} ## strip path, leaving filename only
## if file in backup matches filename, skip rest of loop
ls "${2}"* | grep -q "$fn" &>/dev/null && continue
printf "removing %s\n" "$i"
# rm "$i" ## remove file
done
Note: the actual removal is commented out above, test and insure there are no unintended consequences before preforming the actual delete. Call it passing the path to tmp (without trailing /) as the first argument and with /home/storage as the second argument:
$ bash scriptname /path/to/tmp /home/storage
You can solve this by
making a list of the files in /home/storage
testing each filename in /tmp to see if it is in the list from /home/storage
Given the linux+shell tags, one might use bash:
make the list of files from /home/storage an associative array
make the subscript of the array the filename
Here is a sample script to illustrate ($1 and $2 are the parameters to pass to the script, i.e., /home/storage and /tmp):
#!/bin/bash
declare -A InTarget
while read path
do
name=${path##*/}
InTarget[$name]=$path
done < <(find $1 -type f)
while read path
do
name=${path##*/}
[[ -z ${InTarget[$name]} ]] && rm -f $path
done < <(find $2 -type f)
It uses two interesting shell features:
name=${path##*/} is a POSIX shell feature which allows the script to perform the basename function without an extra process (per filename). That makes the script faster.
done < <(find $2 -type f) is a bash feature which lets the script read the list of filenames from find without making the assignments to the array run in a subprocess. Here the reason for using the feature is that if the array is updated in a subprocess, it would have no effect on the array value in the script which is passed to the second loop.
For related discussion:
Extract File Basename Without Path and Extension in Bash
Bash Script: While-Loop Subshell Dilemma
I spent some really nice time on this today because I needed to delete files which have same name but different extensions, so if anyone is looking for a quick implementation, here you go:
#!/bin/bash
# We need some reference to files which we want to keep and not delete,
# let's assume you want to keep files in first folder with jpeg, so you
# need to map it into the desired file extension first.
FILES_TO_KEEP=`ls -1 ${2} | sed 's/\.pdf$/.jpeg/g'`
#iterate through files in first argument path
for file in ${1}/*; do
# In my case, I did not want to do anything with directories, so let's continue cycle when hitting one.
if [[ -d $file ]]; then
continue
fi
# let's omit path from the iterated file with baseline so we can compare it to the files we want to keep
NAME_WITHOUT_PATH=`basename $file`
# I use mac which is equal to having poor quality clts
# when it comes to operating with strings,
# this should be safe check to see if FILES_TO_KEEP contain NAME_WITHOUT_PATH
if [[ $FILES_TO_KEEP == *"$NAME_WITHOUT_PATH"* ]];then
echo "Not deleting: $NAME_WITHOUT_PATH"
else
# If it does not contain file from the other directory, remove it.
echo "deleting: $NAME_WITHOUT_PATH"
rm -rf $file
fi
done
Usage: sh deleteDifferentFiles.sh path/from/where path/source/of/truth
I am very new in programing scripts-.
I have a lot of zip files in a directory. I want to extract them replacing the name of the inside file by the zip file, with the correct extension. Error reporting if there is more than one file, excep if is "remora.txt" inside.
The file "remora.txt" was an ini file for the zip, and I wont use it any more, but is in a lot of my zip files.
Example 1.
ZIp file: maths.zip,
Inside it has:
- "tutorial in maths.doc"
- "remora.txt"
Action:
So the script should erase or deprease "remora.txt" and extract "tutorial in maths.doc" under the name maths.doc
Example 2.
ZIp file: geo.zip,
Inside it has:
- "excersices for geometry.doc"
- "geometry.doc"
- "remora.txt"item
Action:
It should out put "I found more than a file in geo.zip"
I am
Using linux, ubuntu 12
I have done this script, but is not working.
#!/bin/bash
#
# Linux Shell Scripting Tutorial 1.05r3, Summer-2002
#
for archive in *.zip # First I read the zip file
do
((i++))
unzip -Z1 $archive | while read line; # I read all the files in the ZIP
do
line=( ${line//,/ } )
inside[$a]=("${line[#]}") # Here I assigne the name of the file to an array
((a++))
done
If ( $a > 2) then
echo " Too much files in file $archive "
fi
If ($a <= 2)
then
if (inside[0]!= "remora.txt")
then unzip -p $archive > $(printf "%s" $archive).doc
fi
if (inside[1]!= "remora.txt")
then unzip -p $archive > $(printf "%s" $archive).doc
fi
fi
done
Try writing scripts incrementally. Instead of writing 20 statements and then trying to debug them all at once, write one statement at a time and test to make sure it works before writing the next one.
If you run e.g.
If ( $a > 2) then
echo " Too much files in file $archive "
fi
by itself, you'll see that it doesn't work. You then know more specifically what the problem is, and you can look up something like "bash if variable greater than" on Google or Stackoverflow.
Check out the bash tag wiki for more helpful tips on debugging and asking about code.
Things you'll find includes:
if has to be lower case
You need line feed or semicolon before then
To see if a variable is greater than, use [[ $a -gt 2 ]].
To see if an array element does not equal, use [[ ${inside[0]} != "remora.txt" ]]
Pipelines cause subshells. Use while read ...; do ...; done < <(somecommand) instead.
I noticed many files in my directory, called "sedAbCdEf" or such.
Why does it create these files?
Do these have any value after a script has run?
Can I send these files to another location , e.g. /tmp/?
Update:
I checked the scripts until I found one which makes the files. Here is some sample code:
#!/bin/bash
a=1
b=`wc -l < ./file1.txt`
while [ $a -le $b ]; do
for i in `sed -n "$a"p ./file1.txt`; do
for j in `sed -n "$a"p ./file2.txt`; do
sed -i "s/$i/\nZZ$jZZ\n/g" ./file3.txt
c=`grep -c $j file3.txt`
if [ "$c" -ge 1 ]
then
echo $j >> file4.txt
echo "Replaced "$i" with "$j" "$c" times ("$a"/"$b")."
fi
echo $i" not found ("$a"/"$b")."
a=`expr $a + 1`
done
done
done
Why does it create these files?
sed -i "s/$i/\nZZ$jZZ\n/g" ./file3.txt
the -i option makes sed stores the stdout output into a temporary file.
After sed is done, it will rename this temp file to replace your original file3.txt.
If something is wrong when sed is running, these sedAbCdE temp files will be left there.
Do these have any value after a script has run?
Your old file is untouched. Usually no.
Can I send these files to another location , e.g. /tmp/?
Yes you can, see above.
Edit: see this for further reading.
If you use -i option (it means make changes inplace) sed writes to a temporary file and then renames it to your file. Thus if operation is aborted your file is left unchanged.
You can see which files are opened, renamed with strace:
$ strace -e open,rename sed -i 's/a/b/g' somefile
Note: somefile is opened as readonly.
It seems there is no way to override the backup directory. GNU sed always writes in the file's directory (±symlinks). From sed/execute.c:
if (follow_symlinks)
input->in_file_name = follow_symlink (name);
else
input->in_file_name = name;
/* get the base name */
tmpdir = ck_strdup(input->in_file_name);
if ((p = strrchr(tmpdir, '/')))
*(p + 1) = 0;
else
strcpy(tmpdir, ".");
Prefix sed is hardcoded:
output_file.fp = ck_mkstemp (&input->out_file_name, tmpdir, "sed");
This may be that, since you have used too much sed actions, and in a looped pattern, sed may be making tmp files which are not removed properly.
Sed creates un-deleteable files in Windows
Take a look at this post, sed have such an issue to be reported before. The better way is to make a script that removes the files, or create a function that remove all files that deletes all files with name starting with sed, (^sed* )like thing.