I have a bash script that removes the first five lines and last nine lines on a user specified file. I need it to run on ~100 files a day, is there any way to run the script against the entire directory and then output every file to a separate directory with the same file name? Here is the script I am using.
read -e file
$file | sed '1,7d' | head -n -9 > ~/Documents/Databases/db3/$file
rm $file
I set it up to loop because of so many files but that is the core of the script.
You can do:
for file in ~/Documents/Databases/db3/*; do
[ -f "$file" ] && sed '1,7d' "$file" | head -n -9 > /out/dir/"${file##*/}"
done
Assuming the input directory is ~/Documents/Databases/db3/ and the output directory is /out/dir/.
Related
I am trying to make a script what looks at a folder and will automatically encode files that go into that folder using hand brake. I want to do this doing monitoring the folder using inotify putting the new additions to the folder into a list then using a cron job to encode them overnight. However when using a while loop to loop over the list handbrake only encodes the first file exists then the scripts carrys on to after the loop without doing every file in the list. Here is the script that is calling handbrake:
#!/bin/bash
while IFS= read -r line
do
echo "$(basename "$line")"
HandBrakeCLI -Z "Very Fast 1080p30" -i "$line" -o "$line.m4v"
rm "$line"
done < list.txt
> list.txt
When testing the loop with a simple echo instead of the HandBrakeCLI it works fine and prints out every file so I have no idea what is wrong.
Here is the scripts that is monitoring the folder incase that is the problem:
#!/bin/bash
if ! [ -f list.txt ]
then
touch list.txt
fi
inotifywait -m -e create --format "%w%f" tv-shows | while read FILE
do
echo "$FILE" >> list.txt
done
Any help would be great, thanks
EDIT:
Just to be more specific, the script works fine for the first file in the list.txt, it encodes it no problem and removes the old version, but then it doesn't do any of the others in the list
Taken from here
To solve the problem simply
echo "" | HandBrakeCLI ......
or
HandBrakeCLI ...... < /dev/null
I have a list of files that I need to pipe into a shell script. I can list the files within a directory by using the following:
ls ~/data/2121/*SOMEFILE*
resulting in:
2121.SOMEFILEaa
2121.SOMEFILEab
2121.SOMEFILEac
and so on...
I have another script that performs some processing on a single file (2121.SOMEFILEaa) which I run by using the following command:
bash runscript ../data/2121/2121.SOMEFILEaa
However, I need to make this more efficient by piping individual files from the list of files generated via ls into the script. How can I pipe the results from the ls ~/data/2121/*SOMEFILES* command--file by file--into the runscript script?
Another option
ls ~/data/2121/*SOMEFILE* | xargs -L1 bash runscript
I think you are looking for this:
for file in ~/data/2121/*SOMEFILE*; do
bash runscript "$file"
done
In this way, you're calling bash runscript for each file.
$ cat pipe.sh
#!/bin/bash
## Store data from pipe to variable $PIPE ------#
_read_pipe(){ #
while read -t 10 pipe; do
if [ -n "$pipe" ] ;then
PIPE="$PIPE $pipe" ;fi ;done ;}
## your code -----------------------------------#
_read_pipe #
for kung_foo in $PIPE ;do
echo $kung_foo ;done
$ ls 2121.SOMEFILE* | ./pipe.sh
2121.SOMEFILEaa
2121.SOMEFILEab
2121.SOMEFILEac
and so on...
[ -t ] is for timeout
I hope this helps,
cheers Karim
for file in $*
head -n 1 $file | while read folder
do
mkdir $directory $folder
done
Hello guys, I'm having problem with my script. What I want to do is: read first line from my specifed file and create new directories in my specifed directory from words that i have taken from that file.
I'm getting errors like this:
./scriptas: line 2: syntax error near unexpected token `head'
./scriptas: line 2: `head -n 1 $file | while read folder'
And my second question: how do I add a second variable from command line (putty) $directory ?
Example i have file with text:
one two three
five seven nine eleven
okey
i need script to take the first line and create directories "one" "two" "three"
You have to put do before the command in a for/while cycle.
Your code should look like something like this:
#!/bin/bash
files=$*
for file in $files
do
head -n1 "$file" | while read dname
do
mkdir $dname
done
done
as for other variables, the simple syntax is a number behind the $ sign.
so you could do
files="$1"
directory="$2"
and then run the script as
./script.sh "file1.txt file2.txt file3.txt" dir2
More complex solutions include getopts and such....
Updated the script. You can use it in this way:
script.sh "one.txt two.txt three.txt" destdir
#! /bin/bash
for files in $1
do
for i in $(head -n 1 $files)
do
if [ -z $2 ]
then
mkdir $i
else
mkdir $2/$i -p
fi
done
done
There is an input like:
folder1
folder2
folder3
...
foldern
I would like to iterate over taking multiple lines at once and processes each line, remove the first / (and more but for now this is enough) and echo the. Iterating over in bash with a single thread can be slow sometimes. The alternative way of doing this would be splitting up the input file to N pieces and run the same script with different input and output N times, at the end you can merge the results.
I was wondering if this is possible with xargs.
Update 1:
Input:
/a/b/c
/d/f/e
/h/i/j
Output:
mkdir a/b/c
mkdir d/f/e
mkdir h/i/j
Script:
for i in $(<test); do
echo mkdir $(echo $i | sed 's/\///') ;
done
Doing it with xargs does not work as I would expect:
xargs -a test -I line --max-procs=2 echo mkdir $(echo $line | sed 's/\///')
Obviously I need a way to execute the sed on the input for each line, but using $() does not work.
You probably want:
--max-procs=max-procs, -P max-procs
Run up to max-procs processes at a time; the default is 1. If
max-procs is 0, xargs will run as many processes as possible at
a time. Use the -n option with -P; otherwise chances are that
only one exec will be done.
http://unixhelp.ed.ac.uk/CGI/man-cgi?xargs
With GNU Parallel you can do:
cat file | perl -pe s:/:: | parallel mkdir -p
or:
cat file | parallel mkdir -p {= s:/:: =}
I have multiple files with an insanely long list of commands. I can't run them all in one go, so I need a smart way to read and execute from file as well as delete the command after completion.
So far I have tried
for i in filename.txt ; do ; execute $i ; sed -s 's/$i//' ; done ;
but it doesn't work. Before I introduced sed, $i was executing. Now even that is not working.
I thought of a workaround where I will read first line and delete first line till file is empty.
Any better ideas or commands?
This should work for you, list.txt is your file containing commands.
Make sure you backup the command file before running.
while read line; do $line;sed -i '1d' list.txt;done < "list.txt"
sed -i edits in-place so list.txt will be changed along the loop and you will end up with a empty file.
I think what you want to do is something like this:
while read -r -- i; do $i; sed -i "0,/$i/s/$i//;/^$/d" filename.txt; done < filename.txt
The file is read into the loop. Each line is executed, and the sed command will delete only the first entry it finds, then delete the empty line.
I think that one way to do it is to have the source file of all the commands to be executed, and the script that executes the commands also writes a second log file that lists the files as they are executed.
If you need to resume the process, you work on the lines in the source file that are not present in the log file.
logfile=commands.log
srcfile=commands.src
oldfile=commands.old
trap "mv $oldfile $logfile; exit 1" 0 1 2 3 13 15
[ -f $logfile ] || cp /dev/null $logfile
cp $logfile $oldfile
comm -23 $srcfile $logfile |
while read -r line
do
echo "$line" >> $oldfile
($line) < /dev/null
done
mv $oldfile $logfile
trap 0