I have the following lines to prompt a user to enter a date/time that I use further down in my script:
printf "Please insert start of time window in format YYYY-MM-DD HH:MM:SS: \n"
read startDatetime
Is there a way I can have the "read" prompt insert in the dashes for the date, the space between the date and time, and the colons for the time dynamically as a user types? I know this is possible using JavaScript on the web, but I am specifically looking for a command line version of that functionality.
I searched the web for an answer to this and cannot find one. I see an option for delimiter in the read Man Page but cannot see how it would achieve what I am trying to do.
You can cobble it together by limiting the number of characters you read at a time using the -N flag for read:
echo "Please enter start time in YYYY-MM-DD HH:MM:SS format"
read -rN4 year
printf '-'
read -rN2 month
printf '-'
read -rN2 day
printf ' '
read -rN2 hour
printf ':'
read -rN2 minute
printf ':'
read -rN2 second
echo
printf -v datetime '%s-%s-%s %s:%s:%s' \
"$year" "$month" "$day" "$hour" "$minute" "$second"
echo "You entered $datetime"
Looks like this:
Alternatively, with the option to move the cursor back past the automatically added -, blank, and :: we can use tput to move the cursor, clear the line, and then use read -e (use readline for input) and -i (prefill prompt):
moveup=$(tput cuu1)
clearln=$(tput el)
echo "Please enter start time in YYYY-MM-DD HH:MM:SS format"
read -rN4 datetime
printf '%b' '\r' "$clearln"
read -erN7 -i "$datetime-" datetime
printf '%b' '\r' "$moveup" "$clearln"
read -erN10 -i "$datetime-" datetime
printf '%b' '\r' "$moveup" "$clearln"
read -erN13 -i "$datetime " datetime
printf '%b' '\r' "$moveup" "$clearln"
read -erN16 -i "$datetime:" datetime
printf '%b' '\r' "$moveup" "$clearln"
read -erN19 -i "$datetime:" datetime
echo "You entered $datetime"
This isn't perfect, either: the extra characters are only inserted once, but not when you go back to fix something; and there's a newline inserted after the prompt that results in a blank line, which would be noticeable at the bottom of the terminal emulator:
At this point, you might consider using something like charmbracelet/gum for a very user-friendly interactive input experience at the terminal.
No, it is not possible. But note that read offers the -p switch:
read -p "Please insert start of time window in format YYYY-MM-DD HH:MM:SS: \n" startDatetime
I have to extract particular date's log from any application log. what we generally do
vi app.log
/date (searching)
.= (position where date found)
G(last line)
first_line,Last_line w filname.log (to generate log file with particular date)
I made small script but I think My logic is not correct:
#!/bin/bash
read -p " enter the App name" : app
read =p " enter the Date in this Format 10 Jan 20:01:00" : date
FILE=/logs/$app.log
# Check if $app exists or not
if test ! -f "$app" /logs
then
echo "Error - $app not found."
else
First_line=$(cat $FILE | grep $date | .=)
Second_line=$(tail -1 $FILE | .=)
vi $FILE | $First_line, $Second_line w $app.txt
fi
There is an error in the line
if test ! -f "$app" /logs
You seperated "$app" and /logs with a space; test -f expects one parameter, so it should be:
if test ! -f "$app"/logs
To get the log starting from the date given to the current end of the file, You should use sed (meaning "stream editor"), which is designed for noninteractive editing of files:
sed -n '/'$date'/,$ p' $app.log >$app.txt
The parameter -n is to only output, what matched and not the whole file.
Here I am telling sed to extract everything between two regular expressions, the first one, being the $date parameter given, the second one meaning "end of file". You could change this easily to extract all logs for one specific day or whatever.
As something seems to be unclear I am adding a complete example:
I have an apache log file, where the date logged looks like this [01/Apr/2015:22:31:21 +0200].
So I can do:
export date="01\/Apr\/2015"
It is important to note that I am escaping the slashes here, as they are going into a regex. This has to be done with everything that has some special meaning inside regexes too.
Then I can do:
sed -n '/'$date'/,$ p' myaccess.log
and I get everything logged for the 1st of April until the end of the file.
You could pass commands to vi from your script with a here document, if your task is as mechanical as you describe.
I'm talking about something like this
#!/bin/bash
read -p " enter the App name" : app
read =p " enter the Date in this Format 10 Jan 20:01:00" : date
FILE=/logs/$app.log
# Check if $app exists or not
if test ! -f "$app" /logs
then
echo "Error - $app not found."
else
vi $FILE <<-_EOF_
/date
.=
G
first_line,Last_line w filname.log
etc, etc
_EOF_
fi
I just wrote this as an idea that you mighr find fun to explore. I know the code will not work.
I'd use awk:
awk -v date="$date" '$0 ~ date {p=1} p' file.in > file.out
That will start printing lines when the date is first seen, until the end of the line.
I have a folder with lots of files having a pattern, which is some string followed by a date and time:
BOS_CRM_SUS_20130101_10-00-10.csv (3 strings before date)
SEL_DMD_20141224_10-00-11.csv (2 strings before date)
SEL_DMD_SOUS_20141224_10-00-10.csv (3 strings before date)
I want to loop through the folder and extract only the part before the date and output into a file.
Output
BOS_CRM_SUS_
SEL_DMD_
SEL_DMD_SOUS_
This is my script but it is not working
#!/bin/bash
# script variables
FOLDER=/app/list/l088app5304d1/socles/Data/LEMREC/infa_shared/Shell/Check_Header_T24/
LOG_FILE=/app/list/l088app5304d1/socles/Data/LEMREC/infa_shared/Shell/Check_Header_T24/log
echo "Starting the programme at: $(date)" >> $LOG_FILE
# Getting part of the file name from FOLDER
for file in `ls $FOLDER/*.csv`
do
mv "${file}" "${file/date +%Y%m%d HH:MM:SS}" 2>&1 | tee -a $LOG_FILE
done #> $LOG_FILE
Use sed with extended-regex and groups to achieve this.
cat filelist | sed -r 's/(.*)[0-9]{8}_[0-9][0-9]-[0-9][0-9].[0-9][0-9].csv/\1/'
where filelist is a file with all the names you care about. Of course, this is just a placeholder because I don't know how you are going to list all eligible files. If a glob will do, for example, you can do
ls mydir/*.csv | sed -r 's/(.*)[0-9]{8}_[0-9][0-9]-[0-9][0-9].[0-9][0-9].csv/\1/'
Assuming you wont have numbers in the first part, you could use:
$ for i in *csv;do str=$(echo $i|sed -r 's/[0-9]+.*//'); echo $str; done
BOS_CRM_SUS_
SEL_DMD_
SEL_DMD_SOUS_
Or with parameter substitution:
$ for i in *csv;do echo ${i%_*_*}_; done
BOS_CRM_SUS_
SEL_DMD_
SEL_DMD_SOUS_
When you use ${var/pattern/replace}, the pattern must be a filename glob, not command to execute.
Instead of using the substitution operator, use the pattern removal operator
mv "${file}" "${file%_*-*-*.csv}.csv"
% finds the shortest match of the pattern at the end of the variable, so this pattern will just match the date and time part of the filename.
The substitution:
"${file/date +%Y%m%d HH:MM:SS}"
is unlikely to do anything, because it doesn't execute date +%Y%m%d HH:MM:SS. It just treats it as a pattern to search for, and it's not going to be found.
If you did execute the command, though, you would get the current date and time, which is also (apparently) not what you find in the filename.
If that pattern is precise, then you can do the following:
echo "${file%????????_??-??-??.csv}" >> "$LOG_FILE"
using grep:
ls *.csv | grep -Po "\K^([A-Za-z]+_)+"
output:
BOS_CRM_SUS_
SEL_DMD_
SEL_DMD_SOUS_
I have a text file that contains references to variables and lets a user set up the formatting they want around variables, say something like
The date is $DATE
The time is $TIME
I then want to read this text file in, replace the variables, and print the result to stdout using a bash script. The closest thing I've gotten is using "echo" to output it,
DATE="1/1/2010"
TIME="12:00"
TMP=`cat file.txt`
echo $TMP
However, the output ends up all on one line, and I don't want to have \n at the end of every line in the text file. I tried using "cat << $TMP", but then there are no newlines and the variables inside the text aren't getting replaced with values.
You can use eval to ensure that variables are expanded in your data file:
DATE="1/1/2010"
TIME="12:00"
while read line
do
eval echo ${line}
done < file.txt
Note that this allows the user with control over the file content to execute arbitrary commands.
If the input file is outside your control, a much safer solution would be:
DATE="1/1/2010"
TIME="12:00"
sed -e "s#\$DATE#$DATE#" -e "s#\$TIME#$TIME#" < file.txt
It assumes that neither $DATE nor $TIME contain the # character and that no other variables should be expanded.
Slightly more compact than Adam's response:
DATE="1/1/2010"
TIME="12:00"
TMP=`cat file.txt`
eval echo \""$TMP"\"
The downside to all of this is that you end up squashing quotes. Better is to use a real template. I'm not sure how to do that in shell, so I'd use another language if at all possible. The plus side to templating is that you can eliminate that "arbitrary command" hole that Adam mentions, without writing code quite as ugly as sed.
Just quote your variable to preserve newline characters.
DATE="1/1/2010"
TIME="12:00"
TMP=$(cat file.txt)
echo "$TMP"
For your modification of values in file with variables you can do -
while read -r line
do
sed -e "s#\$DATE#$DATE#" -e "s#\$TIME#$TIME#" <<< "$line"
done < file.txt
Test:
[jaypal:~/Temp] cat file.txt
The date is $DATE
The time is $TIME
[jaypal:~/Temp] DATE="1/1/2010"
[jaypal:~/Temp] TIME="12:00"
[jaypal:~/Temp] while read -r line; do sed -e "s#\$DATE#$DATE#" -e "s#\$TIME#$TIME#" <<< "$line"; done < file.txt
The date is 1/1/2010
The time is 12:00
How do I iterate through each line of a text file with Bash?
With this script:
echo "Start!"
for p in (peptides.txt)
do
echo "${p}"
done
I get this output on the screen:
Start!
./runPep.sh: line 3: syntax error near unexpected token `('
./runPep.sh: line 3: `for p in (peptides.txt)'
(Later I want to do something more complicated with $p than just output to the screen.)
The environment variable SHELL is (from env):
SHELL=/bin/bash
/bin/bash --version output:
GNU bash, version 3.1.17(1)-release (x86_64-suse-linux-gnu)
Copyright (C) 2005 Free Software Foundation, Inc.
cat /proc/version output:
Linux version 2.6.18.2-34-default (geeko#buildhost) (gcc version 4.1.2 20061115 (prerelease) (SUSE Linux)) #1 SMP Mon Nov 27 11:46:27 UTC 2006
The file peptides.txt contains:
RKEKNVQ
IPKKLLQK
QYFHQLEKMNVK
IPKKLLQK
GDLSTALEVAIDCYEK
QYFHQLEKMNVKIPENIYR
RKEKNVQ
VLAKHGKLQDAIN
ILGFMK
LEDVALQILL
One way to do it is:
while read p; do
echo "$p"
done <peptides.txt
As pointed out in the comments, this has the side effects of trimming leading whitespace, interpreting backslash sequences, and skipping the last line if it's missing a terminating linefeed. If these are concerns, you can do:
while IFS="" read -r p || [ -n "$p" ]
do
printf '%s\n' "$p"
done < peptides.txt
Exceptionally, if the loop body may read from standard input, you can open the file using a different file descriptor:
while read -u 10 p; do
...
done 10<peptides.txt
Here, 10 is just an arbitrary number (different from 0, 1, 2).
cat peptides.txt | while read line
do
# do something with $line here
done
and the one-liner variant:
cat peptides.txt | while read line; do something_with_$line_here; done
These options will skip the last line of the file if there is no trailing line feed.
You can avoid this by the following:
cat peptides.txt | while read line || [[ -n $line ]];
do
# do something with $line here
done
Option 1a: While loop: Single line at a time: Input redirection
#!/bin/bash
filename='peptides.txt'
echo Start
while read p; do
echo "$p"
done < "$filename"
Option 1b: While loop: Single line at a time:
Open the file, read from a file descriptor (in this case file descriptor #4).
#!/bin/bash
filename='peptides.txt'
exec 4<"$filename"
echo Start
while read -u4 p ; do
echo "$p"
done
This is no better than other answers, but is one more way to get the job done in a file without spaces (see comments). I find that I often need one-liners to dig through lists in text files without the extra step of using separate script files.
for word in $(cat peptides.txt); do echo $word; done
This format allows me to put it all in one command-line. Change the "echo $word" portion to whatever you want and you can issue multiple commands separated by semicolons. The following example uses the file's contents as arguments into two other scripts you may have written.
for word in $(cat peptides.txt); do cmd_a.sh $word; cmd_b.py $word; done
Or if you intend to use this like a stream editor (learn sed) you can dump the output to another file as follows.
for word in $(cat peptides.txt); do cmd_a.sh $word; cmd_b.py $word; done > outfile.txt
I've used these as written above because I have used text files where I've created them with one word per line. (See comments) If you have spaces that you don't want splitting your words/lines, it gets a little uglier, but the same command still works as follows:
OLDIFS=$IFS; IFS=$'\n'; for line in $(cat peptides.txt); do cmd_a.sh $line; cmd_b.py $line; done > outfile.txt; IFS=$OLDIFS
This just tells the shell to split on newlines only, not spaces, then returns the environment back to what it was previously. At this point, you may want to consider putting it all into a shell script rather than squeezing it all into a single line, though.
Best of luck!
A few more things not covered by other answers:
Reading from a delimited file
# ':' is the delimiter here, and there are three fields on each line in the file
# IFS set below is restricted to the context of `read`, it doesn't affect any other code
while IFS=: read -r field1 field2 field3; do
# process the fields
# if the line has less than three fields, the missing fields will be set to an empty string
# if the line has more than three fields, `field3` will get all the values, including the third field plus the delimiter(s)
done < input.txt
Reading from the output of another command, using process substitution
while read -r line; do
# process the line
done < <(command ...)
This approach is better than command ... | while read -r line; do ... because the while loop here runs in the current shell rather than a subshell as in the case of the latter. See the related post A variable modified inside a while loop is not remembered.
Reading from a null delimited input, for example find ... -print0
while read -r -d '' line; do
# logic
# use a second 'read ... <<< "$line"' if we need to tokenize the line
done < <(find /path/to/dir -print0)
Related read: BashFAQ/020 - How can I find and safely handle file names containing newlines, spaces or both?
Reading from more than one file at a time
while read -u 3 -r line1 && read -u 4 -r line2; do
# process the lines
# note that the loop will end when we reach EOF on either of the files, because of the `&&`
done 3< input1.txt 4< input2.txt
Based on #chepner's answer here:
-u is a bash extension. For POSIX compatibility, each call would look something like read -r X <&3.
Reading a whole file into an array (Bash versions earlier to 4)
while read -r line; do
my_array+=("$line")
done < my_file
If the file ends with an incomplete line (newline missing at the end), then:
while read -r line || [[ $line ]]; do
my_array+=("$line")
done < my_file
Reading a whole file into an array (Bash versions 4x and later)
readarray -t my_array < my_file
or
mapfile -t my_array < my_file
And then
for line in "${my_array[#]}"; do
# process the lines
done
More about the shell builtins read and readarray commands - GNU
More about IFS - Wikipedia
BashFAQ/001 - How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?
Related posts:
Creating an array from a text file in Bash
What is the difference between thee approaches to reading a file that has just one line?
Bash while read loop extremely slow compared to cat, why?
Use a while loop, like this:
while IFS= read -r line; do
echo "$line"
done <file
Notes:
If you don't set the IFS properly, you will lose indentation.
You should almost always use the -r option with read.
Don't read lines with for
If you don't want your read to be broken by newline character, use -
#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "$line"
done < "$1"
Then run the script with file name as parameter.
Suppose you have this file:
$ cat /tmp/test.txt
Line 1
Line 2 has leading space
Line 3 followed by blank line
Line 5 (follows a blank line) and has trailing space
Line 6 has no ending CR
There are four elements that will alter the meaning of the file output read by many Bash solutions:
The blank line 4;
Leading or trailing spaces on two lines;
Maintaining the meaning of individual lines (i.e., each line is a record);
The line 6 not terminated with a CR.
If you want the text file line by line including blank lines and terminating lines without CR, you must use a while loop and you must have an alternate test for the final line.
Here are the methods that may change the file (in comparison to what cat returns):
1) Lose the last line and leading and trailing spaces:
$ while read -r p; do printf "%s\n" "'$p'"; done </tmp/test.txt
'Line 1'
'Line 2 has leading space'
'Line 3 followed by blank line'
''
'Line 5 (follows a blank line) and has trailing space'
(If you do while IFS= read -r p; do printf "%s\n" "'$p'"; done </tmp/test.txt instead, you preserve the leading and trailing spaces but still lose the last line if it is not terminated with CR)
2) Using process substitution with cat will reads the entire file in one gulp and loses the meaning of individual lines:
$ for p in "$(cat /tmp/test.txt)"; do printf "%s\n" "'$p'"; done
'Line 1
Line 2 has leading space
Line 3 followed by blank line
Line 5 (follows a blank line) and has trailing space
Line 6 has no ending CR'
(If you remove the " from $(cat /tmp/test.txt) you read the file word by word rather than one gulp. Also probably not what is intended...)
The most robust and simplest way to read a file line-by-line and preserve all spacing is:
$ while IFS= read -r line || [[ -n $line ]]; do printf "'%s'\n" "$line"; done </tmp/test.txt
'Line 1'
' Line 2 has leading space'
'Line 3 followed by blank line'
''
'Line 5 (follows a blank line) and has trailing space '
'Line 6 has no ending CR'
If you want to strip leading and trading spaces, remove the IFS= part:
$ while read -r line || [[ -n $line ]]; do printf "'%s'\n" "$line"; done </tmp/test.txt
'Line 1'
'Line 2 has leading space'
'Line 3 followed by blank line'
''
'Line 5 (follows a blank line) and has trailing space'
'Line 6 has no ending CR'
(A text file without a terminating \n, while fairly common, is considered broken under POSIX. If you can count on the trailing \n you do not need || [[ -n $line ]] in the while loop.)
More at the BASH FAQ
I like to use xargs instead of while. xargs is powerful and command line friendly
cat peptides.txt | xargs -I % sh -c "echo %"
With xargs, you can also add verbosity with -t and validation with -p
This might be the simplest answer and maybe it don't work in all cases, but it is working great for me:
while read line;do echo "$line";done<peptides.txt
if you need to enclose in parenthesis for spaces:
while read line;do echo \"$line\";done<peptides.txt
Ahhh this is pretty much the same as the answer that got upvoted most, but its all on one line.
#!/bin/bash
#
# Change the file name from "test" to desired input file
# (The comments in bash are prefixed with #'s)
for x in $(cat test.txt)
do
echo $x
done
Here is my real life example how to loop lines of another program output, check for substrings, drop double quotes from variable, use that variable outside of the loop. I guess quite many is asking these questions sooner or later.
##Parse FPS from first video stream, drop quotes from fps variable
## streams.stream.0.codec_type="video"
## streams.stream.0.r_frame_rate="24000/1001"
## streams.stream.0.avg_frame_rate="24000/1001"
FPS=unknown
while read -r line; do
if [[ $FPS == "unknown" ]] && [[ $line == *".codec_type=\"video\""* ]]; then
echo ParseFPS $line
FPS=parse
fi
if [[ $FPS == "parse" ]] && [[ $line == *".r_frame_rate="* ]]; then
echo ParseFPS $line
FPS=${line##*=}
FPS="${FPS%\"}"
FPS="${FPS#\"}"
fi
done <<< "$(ffprobe -v quiet -print_format flat -show_format -show_streams -i "$input")"
if [ "$FPS" == "unknown" ] || [ "$FPS" == "parse" ]; then
echo ParseFPS Unknown frame rate
fi
echo Found $FPS
Declare variable outside of the loop, set value and use it outside of loop requires done <<< "$(...)" syntax. Application need to be run within a context of current console. Quotes around the command keeps newlines of output stream.
Loop match for substrings then reads name=value pair, splits right-side part of last = character, drops first quote, drops last quote, we have a clean value to be used elsewhere.
This is coming rather very late, but with the thought that it may help someone, i am adding the answer. Also this may not be the best way. head command can be used with -n argument to read n lines from start of file and likewise tail command can be used to read from bottom. Now, to fetch nth line from file, we head n lines, pipe the data to tail only 1 line from the piped data.
TOTAL_LINES=`wc -l $USER_FILE | cut -d " " -f1 `
echo $TOTAL_LINES # To validate total lines in the file
for (( i=1 ; i <= $TOTAL_LINES; i++ ))
do
LINE=`head -n$i $USER_FILE | tail -n1`
echo $LINE
done
#Peter: This could work out for you-
echo "Start!";for p in $(cat ./pep); do
echo $p
done
This would return the output-
Start!
RKEKNVQ
IPKKLLQK
QYFHQLEKMNVK
IPKKLLQK
GDLSTALEVAIDCYEK
QYFHQLEKMNVKIPENIYR
RKEKNVQ
VLAKHGKLQDAIN
ILGFMK
LEDVALQILL
Another way to go about using xargs
<file_name | xargs -I {} echo {}
echo can be replaced with other commands or piped further.
for p in `cat peptides.txt`
do
echo "${p}"
done