Shell file parsing to automate youtube uploads - linux

I was writing a small script to mass upload videos to youtube.. It looks like this:
#! /bin/sh
python --version
for file in ./*.mp4 ; do
export title=$(basename "$file" ".mp4")
echo $title "for" $file
youtube-upload -m mail#mailer.com -p pass -c Category -t "$title" -d "description" "$file"
done
Where youtube-upload is a cli based python uploader..
My question is as you can see i'm taking the title from the file name path, and the description is always the same :(. I want to write the description into a text or xml file parse it, and then upload with a proper description for each file..
How to do this using shell commands in this context?
Thanks for your suggestions

Have one file containing all of your filename->description mappings:
my file.mp4 this is a description
file2.mp4 this is another description
my last file.mp4 this is a description
ilied.mp4 description
And then just grab the line beginning with the filename and use the rest of the line:
#! /bin/sh
python --version
for file in ./*.mp4 ; do
export title=$(basename "$file" ".mp4")
echo $title "for" $file
youtube-upload -m mail#mailer.com -p pass -c Category -t "$title" -d "$(grep "^$(basename "$file")" desc | sed 's/.*.mp4 //')" "$file"
done
Looking at what's in the $(...):
grep "^$(basename "$file")" desc | sed 's/.*.mp4 //'
The grep finds the line that starts with the basename($file) in the "desc" file and then uses sed to get rid of the filename, leaving the rest of the line (which would be the description).
Note that this doesn't need to be a part of the youtube-upload comand line. You can also toss it into a variable:
#! /bin/sh
python --version
for file in ./*.mp4 ; do
export title=$(basename "$file" ".mp4")
echo $title "for" $file
description=$(grep "^$(basename "$file")" desc | sed 's/.*.mp4 //')
youtube-upload -m mail#mailer.com -p pass -c Category -t "$title" -d "$description" "$file"
done

Related

Multiple process curl command for urls to output to individual files

I am attempting to curl multiple urls in a bash command. Eventually I will be curling a large number of Urls so I am using xargs to use multiple processes to speed up the process.
My file consists of x number of URLs:
https://someurl.com
https://someotherurl.com
My issue comes when attempting to output the results to separate files named after the URLs I curl.
The bash command I have is:
xargs -P 5 -n 1 -I% curl -k -L % -0 % < urls.txt
When I run this I get 'Failed to create file https://someotherurl.com'
You cannot create a file with / in the filename. You could do it this way:
#!/bin/bash
while IFS= read -r line
do
echo "LINE: $line"
if [[ "$line" != "" ]]
then
filename="${line#https://}"
echo "FILENAME: $filename"
# YOUR CURL COMMAND HERE, USING $filename
fi
done < url.txt
it ignores empty lines
variable substitution is used to remove the https:// part of each URL
this will allow you to create the file
Note: if your URLs containt sub-directories, they must be removed as well.
Ex: you want to do https://www.exemple.com/some/sub/dir
The script I suggested here would try to create a file named "www.exemple.com/some/sub/dir". In this case, you could replace the / with _ using tr.
The script would become:
#!/bin/bash
while IFS= read -r line
do
echo "LINE: $line"
if [[ "$line" != "" ]]
then
filename=$(echo "$line" | tr '/' '_')
filename2=${filename#https:__}
echo "FILENAME: $filename2"
# YOUR CURL COMMAND HERE, USING $filename2
fi
done < url.txt
Because your question is ambiguous, I would assume:
You have a file urls.txt that contains URLs separated by LF.
You want to download all URLs by curl and use each URL as its filename.
Unfortunately, that's not possible because URL contains invalid characters like slash /. Alternatively, for this case, I would suggest you use Bsse64 safe mode to decode URL before saving to file based on RFC 3548.
After applying this requirement, your script would become like:
seq 100 | xargs -I# echo 'https://example.com?#' > urls.txt
xargs -P0 -L1 sh -c 'curl -SskL0 -o $(printf %s "$1" | uuencode -m /dev/stdout | sed "1d;\$d" | tr +/ -_) "$1"' sh < urls.txt

How can i move/group specific folders in bash?

I have a folder structure like the following:
2020-123-1
2020-123-2
2020-123-3
2020-124-1
2020-124-2
...
I need to create folders from the first 2 numbers and omit whatever follows the second dash (-). Then I need to put the prior folders under the newly created ones with the correct name.
2020-123
->2020-123-1
->2020-123-2
->2020-123-3
2020-124
->2020-124-1
->2020-124-2
I tried to write a script in bash like this:
ls -d */ > folder.txt
cut -f1,2 -d"-" folder.txt |cut -f1 -d"/" |sort|uniq > mainfolder.txt
while read line; do mkdir $line ; done < mainfolder.txt
while read line; do mv $(cut -f1,2 -d"-" $line) $line/ ; done < folder.txt
I couldn't make the last line work, I know it has issues.
Actually, you don't have to parse the directory names and build the hierarchy. You can make use of the -p option of mkdir, thus, an awk one-liner will do the job:
awk -F'-' '{top=$1 FS $2;printf "mkdir -p %s; mv %s %s\n",top, $0, top}' dir.txt
The output with your example:
mkdir -p 2020-123; mv 2020-123-1 2020-123
mkdir -p 2020-123; mv 2020-123-2 2020-123
mkdir -p 2020-123; mv 2020-123-3 2020-123
mkdir -p 2020-124; mv 2020-124-1 2020-124
mkdir -p 2020-124; mv 2020-124-2 2020-124
Note
This one-liner just print the commands without executing them, you just pipe the output to |sh if everything looks fine. Examine the output commands, change the printf format/values for adjustment.
I didn't quote the filenames, since your example doesn't contain any special chars. Do it if it is in the case.
So the final script is as follows:
ls -d */ | cut -f1 -d"/" > folder.txt
awk -F'-' '{top=$1 FS $2;printf "mkdir -p %s; mv %s %s\n",top, $0, top}' folder.txt |sh
In pure bash:
#!/bin/bash
for src in *-*-*; do
destdir=${src%-*}
[[ -d $destdir ]] || mkdir "$destdir" || exit
# This just prints out the command that will be called.
# Remove the "echo" in actual script after making sure it will run as intented
echo mv "$src" "$destdir"
done
In the script above it is assumed that each file name to be moved contains exactly two dashes. If it can contain two or more dashes then the destdir=${src%-*} line should be replaced with these two lines:
suffix=${src#*-*-}
destdir=${src%"-$suffix"}
For detailed information read the "shell parameter expansion" section in bash reference.
Additionally, a good read article is: Why you shouldn't parse the output of ls

Bash script to download graphic files from website

I'm trying to write bash script in Linux (Debian), that will be used for downloading graphic files from website given by user during start-up. I'm not sure if my code is correct but first problem is when i try to run my script with website e.g. http://www.bbc.com/ an error shows: http://www.bbc.com/ : invalid identifier. I even tried a simple website that has only a few JPG files. My next problem is to find out how to download files from .txt file where the images Internet adresses are included.
#!/bin/bash
# $1 - URL $2 - new catalog name
read $1 $2
url=$1
fold=$2
mkdir -p $fold
if [$# -ne 3];
then
echo "Wrong command"
exit -1
fi
curl $url | grep -o -e "<img src=\".*\"+>" > img_list.txt |wc -l img_list.txt | lin=${% *}
baseurl=$(echo $url | grep -o "https?://[a-z.]*"")
curl -s $url | egrep -o "<img src\=[^>]*>" | sed 's/<img src=\"\([^"]*\).*/\1/.*/\1/g' > url_list.txt
sed -i "s|^/|$baseurl/|" url_list.txt
cd $fold;
what can I do next?
For download every image from the webpage I would to use:
mech-dump --absolute --images http://example.com | xargs -n1 curl -O
but this need to be installed the mech-dump command from the WWW::Mechanize package.
Using the list file
while read -r url folder
do
mkdir -p "$folder" || exit 1
(cd "$folder" && mech-dump --absolute --images "$url" | xargs -n1 curl -O)
done < list.txt
(assuming than no url nor folder containing a space).
an error shows: http://www.bbc.com/ : invalid identifier
Your use of read is wrong; change
read $1 $2
url=$1
fold=$2
to
read url fold
or decide to specify the arguments on the command line and omit only read $1 $2.
Also, each operand in [ ] must be separated from the brackets; change
if [$# -ne 3];
to
if [ -z "$fold" ]

Find and highlight text in linux command line

I am looking for a linux command that searches a string in a text file,
and highlights (colors) it on every occurence in the file, WITHOUT omitting text lines (like grep does).
I wrote this handy little script. It could probably be expanded to handle args better
#!/bin/bash
if [ "$1" == "" ]; then
echo "Usage: hl PATTERN [FILE]..."
elif [ "$2" == "" ]; then
grep -E --color "$1|$" /dev/stdin
else
grep -E --color "$1|$" $2
fi
it's useful for stuff like highlighting users running processes:
ps -ef | hl "alice|bob"
Try
tail -f yourfile.log | egrep --color 'DEBUG|'
where DEBUG is the text you want to highlight.
command | grep -iz -e "keyword1" -e "keyword2" (ignore -e switch if just searching for a single word, -i for ignore case, -z for treating as a single file)
Alternatively,while reading files
grep -iz -e "keyword1" -e "keyword2" 'filename'
OR
command | grep -A 99999 -B 99999 -i -e "keyword1" "keyword2" (ignore -e switch if just searching for a single word, -i for ignore case,-A and -B for no of lines before/after the keyword to be displayed)
Alternatively,while reading files
grep -A 99999 -B 99999 -i -e "keyword1" "keyword2" 'filename'
command ack with --passthru switch:
ack --passthru pattern path/to/file
I take it you meant "without omitting text lines" (instead of emitting)...
I know of no such command, but you can use a script such as this (this one is a simple solution that takes the filename (without spaces) as the first argument and the search string (also without spaces) as the second):
#!/usr/bin/env bash
ifs_store=$IFS;
IFS=$'\n';
for line in $(cat $1);
do if [ $(echo $line | grep -c $2) -eq 0 ]; then
echo $line;
else
echo $line | grep --color=always $2;
fi
done
IFS=$ifs_store
save as, for instance colorcat.sh, set permissions appropriately (to be able to execute it) and call it as
colorcat.sh filename searchstring
I had a requirement like this recently and hacked up a small program to do exactly this. Link
Usage: ./highlight test.txt '^foo' 'bar$'
Note that this is very rough, but could be made into a general tool with some polishing.
Using dwdiff, output differences with colors and line numbers.
echo "Hello world # $(date)" > file1.txt
echo "Hello world # $(date)" > file2.txt
dwdiff -c -C 0 -L file1.txt file2.txt

Validate script's argument by file extension?

I am writing a script which you can pass a file name into as an argument and it'll only run if it's a certain file extension.
flac2mp3 "01 Song.flac"
or
flac2mp3 "01 Song.FLAC"
I know there a lot of scripts out there showing you how to convert flac to mp3, but this is my script and I want to learn how to write the script using this method.
It's so I can learn arguments and for when I feel like converting only 1 individual file. (for multiple files I just wrote a for loop with *.flac inside the script)
I just want to learn how to check if the $1 argument contains *.[Ff][Ll][Aa][Cc]
Here's what I cobbled up together from the internet so far (which I know is embarrassingly wrong but I wanted to show what I was going for) :
#!/bin/bash
#flac2mp3
if [ -z $1 ] && [[$1 !=~ *.[Ff][Ll][Aa][Cc]]];then echo "Give FLAC File Name"; exit 0;fi
OUTF=${1%.flac}.mp3
ARTIST=$(metaflac "$1" --show-tag=ARTIST | sed s/.*=//g)
TITLE=$(metaflac "$1" --show-tag=TITLE | sed s/.*=//g)
ALBUM=$(metaflac "$1" --show-tag=ALBUM | sed s/.*=//g)
GENRE=$(metaflac "$1" --show-tag=GENRE | sed s/.*=//g)
TRACKNUMBER=$(metaflac "$1" --show-tag=TRACKNUMBER | sed s/.*=//g)
DATE=$(metaflac "$1" --show-tag=DATE | sed s/.*=//g)
flac -c -d "$1" | lame -m j -q 0 --vbr-new -V 0 -s 44.1 - "$OUTF"
id3 -t "$TITLE" -T "${TRACKNUMBER:-0}" -a "$ARTIST" -A "$ALBUM" -y "$DATE" -g "${GENRE:-12}" "$OUTF"
done
Please and Thank Your for the help.
Try the following code:
shopt -s nocasematch
if [[ $1 == *flac ]]; then
echo "ok"
fi
This is case insensitive.
EDIT
$ LANG=C help shopt
shopt: shopt [-pqsu] [-o] [optname ...]
Set and unset shell options.
Change the setting of each shell option OPTNAME. Without any option
arguments, list all shell options with an indication of whether or not each
is set.
Options:
-o restrict OPTNAMEs to those defined for use with `set -o'
-p print each shell option with an indication of its status
-q suppress output
-s enable (set) each OPTNAME
-u disable (unset) each OPTNAME
Exit Status:
Returns success if OPTNAME is enabled; fails if an invalid option is
given or OPTNAME is disabled.
If you run shopt alone in a shell, you will see al options available :
$ shopt
autocd on
cdable_vars on
cdspell off
checkhash off
checkjobs off
checkwinsize off
cmdhist on
compat31 off
compat32 off
compat40 off
compat41 off
direxpand off
dirspell off
dotglob on
execfail off
expand_aliases on
extdebug off
extglob on
extquote on
failglob off
force_fignore on
globstar on
gnu_errfmt off
histappend on
histreedit off
histverify off
hostcomplete off
huponexit off
interactive_comments on
lastpipe off
lithist off
login_shell off
mailwarn off
no_empty_cmd_completion off
nocaseglob off
nocasematch off
nullglob off
progcomp on
promptvars on
restricted_shell off
shift_verbose off
sourcepath on
xpg_echo off
To know what does all these options :
man bash | less +/'^SHELL BUILTIN COMMANDS'
then search `shopt from within this section.

Resources