I use nohup to run a bash script to read in each line of a file (and extract info I need). I've used it on multiple files with different line sizes, mostly between 50k and 100k. But no matter how many lines my file is, nohup always stops outputting info around 6k before the last line.
my script called: fetchStuff.sh
#!/bin/bash
urlFile=$1
myHost='http://example.com'
useragent='me'
count=0
total_lines=$(wc -l < $urlFile)
while read url; do
if [[ "$url" == *html ]]; then continue; fi
reqURL=${myHost}${url}
stuffInfo=$(curl -s -XGET -A "$useragent" "$reqURL" | jq -r '.stuff')
[ "$stuffInfo" != "null" ] && echo ${stuffInfo/unwanted_garbage/} >> newversion-${urlFile}
((count++))
if [ $(( $count%20 )) -eq 0 ]
then
sleep 1
fi
if [ $(( $count%100 )) -eq 0 ]; then echo "$urlFile read ${count} of $total_lines"; fi
done < $urlFile
I call it like so: nohup ./fetchStuff.sh file1.txt &
I get count info in nohup.out, e.g. "file1 read 100 of 60000", "file1 read 200 of 60000", etc.
But it always stops around 6k before end of file.
When I do tail nohup.out each time after running the script on the file, I get these as the last line in nohup.out:
file1.txt read 90000 of 96317
file2.txt read 68000 of 73376
file3.txt read 85000 of 91722
file4.txt read 93000 of 99757
I can't figure out why it always stops around 6k before end of file. (I put the sleep timer in to avoid flooding the api w/a lot of requests).
The loops skips lines that end with html, and they're not counted in $count. So I'll bet there are 6317 lines in file1.txt that end with html, 5376 in file2.txt, and so on.
If you want $count to include them, put ((count++)) before the if statement that checks the suffix.
while read url; do
((count++))
if [[ "$url" == *html ]]; then continue; fi
reqURL=${myHost}${url}
stuffInfo=$(curl -s -XGET -A "$useragent" "$reqURL" | jq -r '.stuff')
[ "$stuffInfo" != "null" ] && echo ${stuffInfo/unwanted_garbage/} >> newversion-${urlFile}
if [ $(( $count%20 )) -eq 0 ]
then
sleep 1
fi
if [ $(( $count%100 )) -eq 0 ]; then echo "$urlFile read ${count} of $total_lines"; fi
done < $urlFile
Alternatively you could leave them out of total_lines with:
total_lines=$(grep -c -v 'html$' "$urlFile")
And you could do away with the if statement by using
grep -v 'html$' "$urlFile" | while read url; do
...
done
Am trying to make a script to unhide files in the directory with parameter --unhide but it is not working. May you help me? Here is my part of the code.
for i in `ls -1`
do
if [ -f $i ] || [ -d $i ]
then
if [ `echo $i | cut -c1` == "." ]
then
mv $i ${i#.}
fi
fi
done
Thanks!
And how about unhide then i write --unhide file1 file2?
elif [ $1 = --unhide ] && [ ! $2 = --all ]
then
for i in $#
do
if [ -f $i ] || [ -d $i ]
then
if [ `echo $i | cut -c1` == "." ]
then
mv $i ${i#.}
fi
fi
done
Its easy then we have not hidden files, then parameter name is the same name with the file name. But how to check hidden files? I tried to write --unhide .file1 .file2 but script only unhides .file1 and .file2 not.
So basically what you are asking for is a script that rename files within a directory, where the files start with a '.'?
Something like the following should work
GLOBIGNORE=".:.."
for file in .*; do
mv -n "$file" "${file#.}"
done
this is what I come up with
ls -a | awk '{ if ($1 ~/^\..*txt$/) print $1,substr($1,2)}'| xargs -n2 mv
I have a small crash detect script for a teamspeak server. The only issue is that I keep getting a syntaxt error about an unexpected end of file. I am not sure if I am missing something in this from all the guides I have been following. I have also ran dos2unix since I do alot of the coding in sublime text 2 on windows.
#!/bin/bash
TEAMSPEAK=`ps ax | grep ts3server_linux_amd64 | grep -v grep | wc -l`
if [ $TEAMSPEAK -eq 1 ]; then
exit
else
cd /home/ryahn/ts3
if [ -f ts3server.pid ]; then
rm -f ts3server.pid
echo "File here"
. ./home/minecraft/ts3/ts3server_startscript.sh start
fi
You are missing a fi for outer if condition.
Keep your code indented to understand it better:
if [ $TEAMSPEAK -eq 1 ]; then
exit
else
cd /home/ryahn/ts3
if [ -f ts3server.pid ]; then
rm -f ts3server.pid
echo "File here"
. ./home/minecraft/ts3/ts3server_startscript.sh start
fi
fi
btw you can shorten your piped commands using pgrep:
TEAMSPEAK=$(pgrep -f ts3server_linux_amd64|wc -l)
I am using Linux ksh to remove some old directories that I don't want.
What I use is this:
#! /bin/ksh
OLD=/opt/backup
DIR_PREFIX="active"
DIRS=$(ls ${OLD} -t | grep ${DIR_PREFIX})
i=0
while [[ $i -lt ${#DIRS[*]} ]]; do
if [ $i -gt 4 ];
then
echo ${DIRS[$i]}
((i++))
else
((i++))
fi
done
what I am trying to do is: to store a list all the directories sorted by time into a variable-I assume it would be an array but somehow the size of it is 1... ..., then in the while loop, if the position of the directory is greater than 4, then I print out the directory name.
Any idea of how to
If all you want is to print all but the first four entries, just pipe it to head or sed:
#!/bin/sh
OLD=/opt/backup
DIR_PREFIX=active
ls $OLD -t | grep $DIR_PREFIX | sed 1,4d | while read DIR; do
echo $DIR;
done
If you are just using echo, the while loop is redundant, but presumably you will have more commands in the loop.
OLD=/opt/backup
DIR_PREFIX="active"
DIRS_RESULT=$(ls ${OLD} -t | grep ${DIR_PREFIX})
i=0
for DIR in ${DIRS_RESULT}
do
if [ $i -gt 4 ];
then
echo ${DIR}
rm -rf ${DIR}
((i++))
else
((i++))
fi
done
this one works for me
This question already has answers here:
Test whether a glob has any matches in Bash
(22 answers)
Closed 1 year ago.
How would you go about telling whether files of a specific extension are present in a directory, with bash?
Something like
if [ -e *.flac ]; then
echo true;
fi
#!/bin/bash
count=`ls -1 *.flac 2>/dev/null | wc -l`
if [ $count != 0 ]
then
echo true
fi
#/bin/bash
myarray=(`find ./ -maxdepth 1 -name "*.py"`)
if [ ${#myarray[#]} -gt 0 ]; then
echo true
else
echo false
fi
This uses ls(1), if no flac files exist, ls reports error and the script exits; othewise the script continues and the files may be be processed
#! /bin/sh
ls *.flac >/dev/null || exit
## Do something with flac files here
shopt -s nullglob
if [[ -n $(echo *.flac) ]] # or [ -n "$(echo *.flac)" ]
then
echo true
fi
#!/bin/bash
files=$(ls /home/somedir/*.flac 2> /dev/null | wc -l)
if [ "$files" != "0" ]
then
echo "Some files exists."
else
echo "No files with that extension."
fi
You need to be carful which flag you throw into your if statement, and how it relates to the outcome you want.
If you want to check for only regular files and not other types of file system entries then you'll want to change your code skeleton to:
if [ -f file ]; then
echo true;
fi
The use of the -f restricts the if to regular files, whereas -e is more expansive and will match all types of filesystem entries. There are of course other options like -d for directories, etc. See http://tldp.org/LDP/abs/html/fto.html for a good listing.
As pointed out by #msw, test (i.e. [) will choke if you try and feed it more than one argument. This might happen in your case if the glob for *.flac returned more than one file. In that case try wrapping your if test in a loop like:
for file in ./*.pdf
do
if [ -f "${file}" ]; then
echo 'true';
break
fi
done
This way you break on the first instance of the file extension you want and can keep on going with the rest of the script.
The top solution (if [ -e *.flac ];) did not work for me, giving: [: too many arguments
if ls *.flac >/dev/null 2>&1; then it will work.
You can use -f to check whether files of a specific type exist:
#!/bin/bash
if [ -f *.flac ] ; then
echo true
fi
bash only:
any_with_ext () (
ext="$1"
any=false
shopt -s nullglob
for f in *."$ext"; do
any=true
break
done
echo $any
)
if $( any_with_ext flac ); then
echo "have some flac"
else
echo "dir is flac-free"
fi
I use parentheses instead of braces to ensure a subshell is used (don't want to clobber your current nullglob setting).
shopt -s nullglob
set -- $(echo *.ext)
if [ "${#}" -gt 0 ];then
echo "got file"
fi
For completion, with zsh:
if [[ -n *.flac(#qN) ]]; then
echo true
fi
This is listed at the end of the Conditional Expressions section in the zsh manual. Since [[ disables filename globbing, we need to force filename generation using (#q) at the end of the globbing string, then the N flag (NULL_GLOB option) to force the generated string to be empty in case there’s no match.
Here is a solution using no external commands (i.e. no ls), but a shell function instead. Tested in bash:
shopt -s nullglob
function have_any() {
[ $# -gt 0 ]
}
if have_any ./*.flac; then
echo true
fi
The function have_any uses $# to count its arguments, and [ $# -gt 0 ] then tests whether there is at least one argument. The use of ./*.flac instead of just *.flac in the call to have_any is to avoid problems caused by files with names like --help.
Here's a fairly simple solution:
if [ "$(ls -A | grep -i \\.flac\$)" ]; then echo true; fi
As you can see, this is only one line of code, but it works well enough. It should work with both bash, and a posix-compliant shell like dash. It's also case-insensitive, and doesn't care what type of files (regular, symlink, directory, etc.) are present, which could be useful if you have some symlinks, or something.
I tried this:
if [ -f *.html ]; then
echo "html files exist"
else
echo "html files dont exist"
fi
I used this piece of code without any problem for other files, but for html files I received an error:
[: too many arguments
I then tried #JeremyWeir's count solution, which worked for me:
count=`ls -1 *.flac 2>/dev/null | wc -l`
if [ $count != 0 ]
then
echo true
fi
Keep in mind you'll have to reset the count if you're doing this in a loop:
count=$((0))
This should work in any borne-like shell out there:
if [ "$(find . -maxdepth 1 -type f | grep -i '.*\.flac$')" ]; then
echo true
fi
This also works with the GNU find, but IDK if this is compatible with other implementations of find:
if [ "$(find . -maxdepth 1 -type f -iname \*.flac)" ]; then
echo true
fi