I am new to scripting with bash, and I am trying to make a user friendly script to download videos using youtube-dl.
After getting the available streams from a video link, i want to write to a file the available resolutions for the given video. This is my code:
#!/bin/bash
youtube-dl -F https://www.youtube.com/watch?v=05X0RRmUtE0 > info.txt
filename='info.txt'
echo Start
cat > res.txt
echo "" > res.txt
while read p; do
for i in '240p' '360p' '480p' '720p' '1080p' '2160p'; do
while read q; do
if [[$i == $q]]; then
break
else
echo $i >> res.txt
fi
done < res.txt
done
done < $filename
With the current script, '240p' endlessly gets appended to 'res.txt'. I am not able to figure out where i am going wrong
UPDATE
i have edited the code as follows:
while read p ; do
for i in '240p' '360p' '480p' '720p' '1080p' '2160p' ; do
if [[ "$p" == *"$i"* ]]; then
while read q; do
if [[ "$q" == "$i" ]]; then
break
else
echo $i >> res.txt
fi
done < res.txt
fi
done
done < info.txt
but i still do not get the ouput I want
also, for those who do not understand what the script is supposed to do:
1) input a link and extract the available streams to a text file ( youtube-dl -F https://www.youtube.com/watch?v=05X0RRmUtE0 > info.txt)
2) append recurring resolutions from the streams in info.txt to another file res.txt only once
I'll ignore the outer loop (read p ... < $filename) since it does not impact the issue you're asking about, but you should probably check why it's there because it doesn't do anything right now.
The first inner loop (for i in '240p' '360p' ...) loops over that list, and performs the second inner loop for every value.
The second inner loop (read q ... < res.txt) reads res.txt, and compares every line to the value from the first inner loop. If the values match, the second inner loop stops, and the first inner loop continues with the next value, restarting the second inner loop at the beginning of the file.
If the values do not match, you add '240p' to the res.txt file. If it never finds a matching value, it will keep going endlessly, because it does not simply use the original contents of the file, it also reads lines added to this file during the process.
This means that, if the res.txt file does not contain one of the values of the first inner loop, it will get stuck at that value, and keep adding '240p' endlessly, except if the original file does not contain '240p', in which case it will read the entire file, adding a '240p' line for every value it already contains, then break when it encounters the first '240p' it added.
The only way this code will not get stuck is if res.txt contains all of the values from the first inner loop, or all except '240p' (which it will add, then finish).
EDIT:
As for a solution, I'd throw out the entire inner while loop, replacing it with grep:
#!/bin/bash
youtube-dl -F https://www.youtube.com/watch?v=05X0RRmUtE0 > info.txt
echo Start
rm res.txt >/dev/null 2>&1 # remove res.txt if it exists, don't care about warnings if it doesn't exist
touch res.txt # create empty file
while read p ; do
for i in '240p' '360p' '480p' '720p' '1080p' '2160p' ; do
if [[ "$p" =~ "$i" ]]; then # regex matching ensures foo480pbar matches '480p'
if ! grep "$p" res.txt >/dev/null 2>&1; then # if grep fails, res.txt does not contain $p yet
echo $i >> res.txt
fi
fi
done
done < info.txt
You could even throw out the for loop and put all the resolutions in one regex match, and combine both if statements into one, but this way it's easier to read what's going on, so I'll leave that as an exercise to the reader 😉
Related
For this problem in shell script, I need to accept an unknown multiple number of texts files and concatenate them. But I need to put a divider such as (-----) after the contents of each file. There also needs to be an exit code for the script where if it succeeds then it prints out the correct output. But if it fails it needs to print out this help message, "Usage..." (It's long and the help message itself isn't that important.)
I already figured out how to accept multiple number of text files to concatenate them but the problem is the divider really.
I also figured out how to show the help message using cat but the problem is that I don't know what conditions I need to put for the if and else statement because the conditions itself isn't working as intended.
For example, if it can find txt files and print the output it works as intended, however if there is no txt file it doesn't print the help message.
I am pretty sure it has to be some kind of loop statement for the divider to work but the problem is I have no idea how to do it. Not even sure of the conditions.
# Don't know what condition to put for the if here
if [ $? -eq 0 ]
then
# Don't know what condition to put for the for loop here
for j in "$?"
do
cat *.txt
echo "-----"
done
exit 0
else
cat <<< "Usage..." >&2
exit 1
fi
Let's say we have 2 text files.
one.txt has:
Hello
two.txt has:
Happy
Then if the code runs as intended the output should be:
Hello
-----
Happy
-----
Something amongst those lines ?
#!/bin/bash
for param in $#; do
if [[ -f $param ]]; then
cat $param
echo "-----"
else
echo "wrong parameter passed! usage:..."
exit 1
fi
done
exit 0
To explain quickly :
for param in $#; do
Iterate over every parameter passed to the script
[[ -f $param ]]
This checks whether the parameter is an existing file
I want to write a script that takes a word as an argument and searches the current and sub directories' files for the word. if it is found in any of the files it should echo out a message containing the file name and the line the word is found on.
this is what I have so far, but I can't find a way to actually store the file name of the file being read or the line number..
word=$1
for var in $(grep -R "$word *")
do
filename=$(find . -type f -name "*") ------- //this doesnt work
linenmbr=$(grep -n "$ord" file) ----------- //this doesnt work
echo found $word in $filename on line number $linenmbr
done
In bash, any time you are looping, you want to avoid calling utilities (e.g. grep and find) within the loop. That is horribly inefficient because it will spawn a separate subshell for every utility every iteration. (which for 10 iterations -- that is 20 additional subshells, it adds up quick) So in your case, you call grep to feed the loop, and then spawn a separate subshell calling grep again within the loop as well as spawning a separate subshell for find.
You should think of a way to only call grep (or a utility that will provide the needed information) only once, and then parse the output.
If you did want to use grep, then calling grep -rn within a process substitution which is used to feed a while loop is probably as good as you are going to get. You can then use the bash builtin parameter expansions to isolate the filename and line-numbers which will be about as efficient as bash could get, e.g.
#!/bin/bash
[ -z "$1" ] && { ## validate at least 1 input given
printf "error: insufficient input.\nusage: %s srch_term\n" "${0##*/}"
exit 1
}
while read -r line; do ## read each line of grep output
fn="${line%%:*}" ## isolate filename
no="${line#*:}" ## remove filename
no="${no%%:*}" ## isolate number
printf "found %s in %s on line number %d\n" "$1" "$fn" "$no"
done < <(grep -rn "$1") ## grep in process substitution
Choosing A More Efficient Method
If you can accomplish what you are attempting with one of the stream editing tools, e.g. awk or sed, you are likely to be able to isolate the wanted information an order of magnitude faster. For example, using awk and setting globstar you could do something similar to the following:
#!/bin/bash
shopt -s globstar ## set globstar
[ -z "$1" ] && { ## validate at least 1 input given
printf "error: insufficient input.\nusage: %s srch_term\n" "${0##*/}"
exit 1
}
## find all matching files and line numbers
awk -v word="$1" '/'$1'/ {
print "found",word,"in",FILENAME,"on line number",FNR; next
}' **/* 2>/dev/null
Give both a try and let me know if you have further questions.
If you want to compare and ensure both are producing the same output, you can use diff to confirm, e.g.
$ diff <(grepscript.sh | sort) <(awkscript.sh | sort)
(if no difference is reported, the output is the same)
#! /bin/bash
for i in $(ls);
do
j=1
echo "$i"
not expected Output:-
autodeploy
bin
config
console-ext
edit.lok
need Output like below if give input 2 it should print "bin" based on below condition, but I want out put like Directory list
1.)autodeploy
2.)bin
3.)config
4.)console-ext
5.)edit.lok
and if i like as input:- 2 then it should print "bin"
Per BashFAQ #1, a while read loop is the correct way to read content line-by-line:
#!/usr/bin/env bash
enumerate() {
local line i
i=0
while IFS= read -r line; do
((++i))
printf '%d.) %s\n' "$i" "$line"
done
}
ls | enumerate
However, ls is not an appropriate tool for programmatic use; the above is acceptable if the results of ls are only for human consumption, but not if they're going to be parsed by a machine -- see Why you shouldn't parse the output of ls(1).
If you want to list files and let the user choose among them by number, pass the results of a glob expression to select:
select filename in *; do
echo "$filename" && break
done
I don't understand what you mean in your question by like Directory list, but following your example, you do not need to write a loop:
ls|nl -s '.)' -w 1
If you want to avoid ls, you can do the following (but be careful - this only works if the directory entries do not contain white spaces (because this would make fmt to break them into two lines):
echo *|fmt -w 1 |nl -s '.)' -w 1
I have the follwoing bash code i
declare -A matrix
num_rows=6
num_columns=1
for ((i=1;i<=num_rows;i++)) do
for ((j=1;j<=num_columns;j++)) do
if [[ i -eq 1 ]]; then
matrix[$i,$j]= echo $i
else
matrix[$i,$j]= echo $j
fi
done
done >> out.dat
This code directs the outputs to dat file
but the problem when I rerun the code, the old outputs are removed and replaced by the new outputs, How can I keep the old outputs and save the new too?
> "out.dat"
truncates the file to zero size, essentially removing all its content.
If you want to preserve the previous old file you can move it:
mv out.dat out.dat.old
> "out.dat"
If you want to preserve all the old files, you can save them with a reasonably unique names such as using the date command:
mv out.dat out.dat.old.$(date +%s%2N)
> "out.dat"
The truncation may or may not be necessary depending whether your rest of code expects an empty or creates one if it doesn't exist.
I am foxed by the following situation.
I have a file list.txt that I want to run through line by line, in a loop, in bash. A typical line in list.txt has spaces in. The problem is that the loop contains a "read" command. I want to write this loop in bash rather than something like perl. I can't do it :-(
Here's how I would usually write a loop to read from a file line by line:
while read p; do
echo $p
echo "Hit enter for the next one."
read x
done < list.txt
This doesn't work though, because of course "read x" will be reading from list.txt rather than the keyboard.
And this doesn't work either:
for i in `cat list.txt`; do
echo $i
echo "Hit enter for the next one."
read x
done
because the lines in list.txt have spaces in.
I have two proposed solutions, both of which stink:
1) I could edit list.txt, and globally replace all spaces with "THERE_SHOULD_BE_A_SPACE_HERE" . I could then use something like sed, within my loop, to replace THERE_SHOULD_BE_A_SPACE_HERE with a space and I'd be all set. I don't like this for the stupid reason that it will fail if any of the lines in list.txt contain the phrase THERE_SHOULD_BE_A_SPACE_HERE (so malicious users can mess me up).
2) I could use the while loop with stdin and then in each loop I could actually launch e.g. a new terminal, which would be unaffected by the goings-on involving stdin in the original shell. I tried this and I did get it to work, but it was ugly: I want to wrap all this up in a shell script and I don't want that shell script to be randomly opening new windows. What would be nice, and what might somehow be the answer to this question, would be if I could figure out how to somehow invoke a new shell in the command and feed commands to it without feeding stdin to it, but I can't get it to work. For example this doesn't work and I don't really know why:
while read p; do
bash -c "echo $p; echo ""Press enter for the next one.""; read x;";
done < list.txt
This attempt seems to fail because "read x", despite being in a different shell somehow, is still seemingly reading from list.txt. But I feel like I might be close with this one -- who knows.
Help!
You must open as a different file descriptor
while read p <&3; do
echo "$p"
echo 'Hit enter for the next one'
read x
done 3< list.txt
Update: Just ignore the lengthy discussion in the comments below. It has nothing to do with the question or this answer.
I would probably count lines in a file and iterate each of those using eg. sed. It is also possible to read infinitely from stdin by changing while condition to: while true; and exit reading with ctrl+c.
line=0 lines=$(sed -n '$=' in.file)
while [ $line -lt $lines ]
do
let line++
sed -n "${line}p" in.file
echo "Hit enter for the next ${line} of ${lines}."
read -s x
done
AWK is also great tool for this. Simple way to iterate through input would be like:
awk '{ print $0; printf "%s", "Hit enter for the next"; getline < "-" }' file
As an alternative, you can read from stderr, which by default is connected to the tty as well. The following then also includes a test for that assumption:
(
tty -s <& 2|| exit 1
while read -r line; do
echo "$line"
echo 'Hit enter'
read x <& 2
done < file
)