I'm trying to use this command:
for i in $(cat file); do echo "$whatever_text.$i">$i; done
which is for making each line a new file, I will get straight to the point here!
I want for bash to ignore the expressions such as "$" because for example if I have a line like this:
$a = 'string';
or multiple lines like that, it won't be printed like that, bash leaves only 'string'.
The correct way to iterate over the lines of a text file is to use the read command from a while loop.
while IFS= read -r i; do
echo "$whatever_text.$i" > "$i"
done < file
See Bash FAQ 001 for more details.
Related
I have a bash script which asks for two arguments with a space between them. Now I would like to automate filling out the prompt in the command line with reading from a text file. The text file contains a list with the argument combinations.
So something like this in the command line I think;
for line in 'cat text.file' ; do script.sh ; done
Can this be done? What am I missing/doing wrong?
Thanks for the help.
A while loop is probably what you need. Put the space separated strings in the file text.file :
cat text.file
bingo yankee
bravo delta
Then write the script in question like below.
#!/bin/bash
while read -r arg1 arg2
do
/path/to/your/script.sh "$arg1" "$arg2"
done<text.file
Don't use for to read files line by line
Try something like this:
#!/bin/bash
ARGS=
while IFS= read -r line; do
ARGS="${ARGS} ${line}"
done < ./text.file
script.sh "$ARGS"
This would add each line to a variable which then is used as the arguments of your script.
'cat text.file' is a string literal, $(cat text.file) would expand to output of command however cat is useless because bash can read file using redirection, also with quotes it will be treated as a single argument and without it will split at space tab and newlines.
Bash syntax to read a file line by line, but will be slow for big files
while IFS= read -r line; do ... "$line"; done < text.file
unsetting IFS for read command preserves leading spaces
-r option preserves \
another way, to read whole file is content=$(<file), note the < inside the command substitution. so a creative way to read a file to array, each element a non-empty line:
read_to_array () {
local oldsetf=${-//[^f]} oldifs=$IFS
set -f
IFS=$'\n' array_content=($(<"$1")) IFS=$oldifs
[[ $oldsetf ]]||set +f
}
read_to_array "file"
for element in "${array_content[#]}"; do ...; done
oldsetf used to store current set -f or set +f setting
oldifs used to store current IFS
IFS=$'\n' to split on newlines (multiple newlines will be treated as one)
set -f avoid glob expansion for example in case line contains single *
note () around $() to store the result of splitting to an array
If I were to create a solution determined by the literal of what you ask for (using a for loop and parsing lines from a file) I would use iterations determined by the number of lines in the file (if it isn't too large).
Assuming each line has two strings separated by a single space (to be used as positional parameters in your script:
file="$1"
f_count="$(wc -l < $file)"
for line in $(seq 1 $f_count)
do
script.sh $(head -n $line $file | tail -n1) && wait
done
You may have a much better time using sjsam's solution however.
I want to run a program (when executed it produces logdata) out of a shell script and write the output into a text file. I failed to do so :/
$prog is the executed prog -> socat /dev/ttyUSB0,b9600 STDOUT
$log/$FILE is just path to a .txt file
I had a Perl script to do this:
open (S,$prog) ||die "Cannot open $prog ($!)\n";
open (R,">>","$log") ||die "Cannot open logfile $log!\n";
while (<S>) {
my $date = localtime->strftime('%d.%m.%Y;%H:%M:%S;');
print "$date$_";
}
I tried to do this in a shell script like this
#!/bin/sh
FILE=/var/log/mylogfile.log
SOCAT=/usr/bin/socat
DEV=/dev/ttyUSB0
BAUD=,b9600
PROG=$SOCAT $DEV$BAUD STDOUT
exec 3<&0
exec 0<$PROG
while read -r line
do
DATE=`date +%d.%m.%Y;%H:%M:%S;`
echo $DATE$line >> $FILE
done
exec 0<&3
Doesn't work at all...
How do I read the output of that prog and pipe it into my text file using a shell script? What did I do wrong (if I didn't do everything wrong)?
Final code:
#!/bin/sh
FILE=/var/log/mylogfile.log
SOCAT=/usr/bin/socat
DEV=/dev/ttyUSB0
BAUD=,b9600
CMD="$SOCAT $DEV$BAUD STDOUT"
$CMD |
while read -r line
do
echo "$(date +'%d.%m.%Y;%H:%M:%S;')$line" >> $FILE
done
To read from a process, use process substitution
exec 0< <( $PROG )
/bin/sh doesn't support it, so use /bin/bash instead.
To assign several words to a variable, quote or backslash whitespace:
PROG="$SOCAT $DEV$BAUD STDOUT"
Semicolon is special in shell, quote it or backslash it:
DATE=$(date '+%d.%m.%Y;%H:%M:%S;')
Moreover, no exec's are needed:
while ...
...
done < <( $PROG )
You might even add > $FILE after done instead of adding each line separately to the file.
Original answer
You haven't shown the error messages — which would have been helpful.
Your problem, though, is probably this line:
DATE=`date +%d.%m.%Y;%H:%M:%S;`
where the semicolons mark the end of a command, and there likely isn't a command %H that does anything useful, etc.
You need quotes around the format argument to date, and I'd use single quotes for this job:
DATE=$(date +'%d.%m.%Y;%H:%M:%S;')
or even replace the two lines in the body of the loop with:
echo "$(date +'%d.%m.%Y;%H:%M:%S;')$line" >> $FILE
The double quotes prevent a variety of problems.
That assumes you fix a bunch of other problems, such as the setting of the variables FILE and prog. Also, I'd probably use:
exec > $FILE
to initially zap the output file and then all subsequent standard output would go to that file, so the echo line becomes:
echo "$(date +'%d.%m.%Y;%H:%M:%S;')$line"
Amended answer
The question was originally missing lots of key information. It eventually got updated to include the complete code.
The problem I identified originally remains an issue, but you weren't running into it because the input redirection was not working. If you want the input to come from a process, use a pipe, or possibly process substitution. However, note that you have #!/bin/sh as your shebang line, and /bin/sh won't recognized process substitution; either change the shebang or use the pipe notation. Note that process substitution has advantages if the loop is setting variables that need to be accessed after the loop is complete.
$SOCAT $DEV$BAUD STDOUT |
while read -r line
do
…
done
or
while read -r line
do
…
done < <($SOCAT $DEV$BAUD STDOUT)
Note that your code contains the line:
PROG=$SOCAT $DEV$BAUD STDOUT
This runs the command identified by $DEV$BAUD with the argument STDOUT and the environment variable PROG set to the value of $SOCAT. That is not what you wanted.
You could use an array:
PROG=($SOCAT $DEV$BAUD STDOUT)
and then run:
"${PROG[#]}"
either in the pipe line:
"${PROG[#]}" |
while read -r line
do
…
done
or with process substitution:
while read -r line
do
…
done < <("${PROG[#]}")
Note that unless there is code after the final exec 0<&3, there was no particular virtue in the redirections involving file descriptor 3. You should also close 3 when you're done with it:
exec 0<&3 3>&-
The 'final' code includes the lines:
CMD="$SOCAT $DEV$BAUD STDOUT"
$CMD |
while read -r line
This works OK because there are no spaces in the arguments to the command. That's a common case, but beware of spaces in arguments and file paths.
I am foxed by the following situation.
I have a file list.txt that I want to run through line by line, in a loop, in bash. A typical line in list.txt has spaces in. The problem is that the loop contains a "read" command. I want to write this loop in bash rather than something like perl. I can't do it :-(
Here's how I would usually write a loop to read from a file line by line:
while read p; do
echo $p
echo "Hit enter for the next one."
read x
done < list.txt
This doesn't work though, because of course "read x" will be reading from list.txt rather than the keyboard.
And this doesn't work either:
for i in `cat list.txt`; do
echo $i
echo "Hit enter for the next one."
read x
done
because the lines in list.txt have spaces in.
I have two proposed solutions, both of which stink:
1) I could edit list.txt, and globally replace all spaces with "THERE_SHOULD_BE_A_SPACE_HERE" . I could then use something like sed, within my loop, to replace THERE_SHOULD_BE_A_SPACE_HERE with a space and I'd be all set. I don't like this for the stupid reason that it will fail if any of the lines in list.txt contain the phrase THERE_SHOULD_BE_A_SPACE_HERE (so malicious users can mess me up).
2) I could use the while loop with stdin and then in each loop I could actually launch e.g. a new terminal, which would be unaffected by the goings-on involving stdin in the original shell. I tried this and I did get it to work, but it was ugly: I want to wrap all this up in a shell script and I don't want that shell script to be randomly opening new windows. What would be nice, and what might somehow be the answer to this question, would be if I could figure out how to somehow invoke a new shell in the command and feed commands to it without feeding stdin to it, but I can't get it to work. For example this doesn't work and I don't really know why:
while read p; do
bash -c "echo $p; echo ""Press enter for the next one.""; read x;";
done < list.txt
This attempt seems to fail because "read x", despite being in a different shell somehow, is still seemingly reading from list.txt. But I feel like I might be close with this one -- who knows.
Help!
You must open as a different file descriptor
while read p <&3; do
echo "$p"
echo 'Hit enter for the next one'
read x
done 3< list.txt
Update: Just ignore the lengthy discussion in the comments below. It has nothing to do with the question or this answer.
I would probably count lines in a file and iterate each of those using eg. sed. It is also possible to read infinitely from stdin by changing while condition to: while true; and exit reading with ctrl+c.
line=0 lines=$(sed -n '$=' in.file)
while [ $line -lt $lines ]
do
let line++
sed -n "${line}p" in.file
echo "Hit enter for the next ${line} of ${lines}."
read -s x
done
AWK is also great tool for this. Simple way to iterate through input would be like:
awk '{ print $0; printf "%s", "Hit enter for the next"; getline < "-" }' file
As an alternative, you can read from stderr, which by default is connected to the tty as well. The following then also includes a test for that assumption:
(
tty -s <& 2|| exit 1
while read -r line; do
echo "$line"
echo 'Hit enter'
read x <& 2
done < file
)
I have a text file named sqlfile, with the following content:
a.sql
b.sql
c.sql
d.sql
What I want is that to store them in variables and then print using for loop.
But here I get only d.sql in the output of the script.
The script:
#!/bin/bash
while read line
do
files=`echo $line`
done < /home/abdul_old/Desktop/My_Shell_Script/sqlfile
for file in $files
do
echo $file
done
A variable can only hold one element, what you want is an array
#!/bin/bash
while read line
do
files+=( "$line" )
done < /home/abdul_old/Desktop/My_Shell_Script/sqlfile
for file in "${files[#]}"
do
echo "$file"
done
while read line
do files="$files $line"
done < /home/abdul_old/Desktop/My_Shell_Script/sqlfile
or
files=$(</home/abdul_old/Desktop/My_Shell_Script/sqlfile)
or
files=$(cat /home/abdul_old/Desktop/My_Shell_Script/sqlfile)
You're doing way too much work in your loop.
The middle alternative works with bash; the other two work with most shells. Prefer $(...) to back-quotes.
This code assumes there are no spaces in file names to mess things up. If you do use blanks in file names, you have to work marginally harder - see the array-based solution by SiegeX
I think you need to make the "files" as array. otherwise, as soon as the while finishes, "files" stores the latest "line".
try:
files=( "${files[#]}" $line )
That's right, you assifn last value to "files"
You must use for instance += instead of =
#!/bin/bash
while read line
do
files+=`echo " $line"`
done < /home/abdul_old/Desktop/My_Shell_Script/sqlfile
for file in $files
do
echo $file
done
Using read is fine but you have to set the IFS environment variable first else leading and trailing white space are removed from each line: Preserving leading white space while reading>>writing a file line by line in bash.
All you have to do is:
readarray myData < sqlfile
This will put file lines into an array called myData
Now you can access any of these lines like this:
printf "%s\n" "${myData[0]}" #outputs first line
printf "%s\n" "${myData[2]}" #outputs third line
And you can iterate over it:
for curLine in "${myData[#]}"; do
echo "$curLine"
done
Note that these lines would contain \n character as well. To remove trailing newlines you can use -t flag like this:
readarray -t myData < sqlfile
readarray is a synonym to mapfile. You can read about it in man bash
Say, I have a file foo.txt specifying N arguments
arg1
arg2
...
argN
which I need to pass to the command my_command
How do I use the lines of a file as arguments of a command?
If your shell is bash (amongst others), a shortcut for $(cat afile) is $(< afile), so you'd write:
mycommand "$(< file.txt)"
Documented in the bash man page in the 'Command Substitution' section.
Alterately, have your command read from stdin, so: mycommand < file.txt
As already mentioned, you can use the backticks or $(cat filename).
What was not mentioned, and I think is important to note, is that you must remember that the shell will break apart the contents of that file according to whitespace, giving each "word" it finds to your command as an argument. And while you may be able to enclose a command-line argument in quotes so that it can contain whitespace, escape sequences, etc., reading from the file will not do the same thing. For example, if your file contains:
a "b c" d
the arguments you will get are:
a
"b
c"
d
If you want to pull each line as an argument, use the while/read/do construct:
while read i ; do command_name $i ; done < filename
command `< file`
will pass file contents to the command on stdin, but will strip newlines, meaning you couldn't iterate over each line individually. For that you could write a script with a 'for' loop:
for line in `cat input_file`; do some_command "$line"; done
Or (the multi-line variant):
for line in `cat input_file`
do
some_command "$line"
done
Or (multi-line variant with $() instead of ``):
for line in $(cat input_file)
do
some_command "$line"
done
References:
For loop syntax: https://www.cyberciti.biz/faq/bash-for-loop/
You do that using backticks:
echo World > file.txt
echo Hello `cat file.txt`
If you want to do this in a robust way that works for every possible command line argument (values with spaces, values with newlines, values with literal quote characters, non-printable values, values with glob characters, etc), it gets a bit more interesting.
To write to a file, given an array of arguments:
printf '%s\0' "${arguments[#]}" >file
...replace with "argument one", "argument two", etc. as appropriate.
To read from that file and use its contents (in bash, ksh93, or another recent shell with arrays):
declare -a args=()
while IFS='' read -r -d '' item; do
args+=( "$item" )
done <file
run_your_command "${args[#]}"
To read from that file and use its contents (in a shell without arrays; note that this will overwrite your local command-line argument list, and is thus best done inside of a function, such that you're overwriting the function's arguments and not the global list):
set --
while IFS='' read -r -d '' item; do
set -- "$#" "$item"
done <file
run_your_command "$#"
Note that -d (allowing a different end-of-line delimiter to be used) is a non-POSIX extension, and a shell without arrays may also not support it. Should that be the case, you may need to use a non-shell language to transform the NUL-delimited content into an eval-safe form:
quoted_list() {
## Works with either Python 2.x or 3.x
python -c '
import sys, pipes, shlex
quote = pipes.quote if hasattr(pipes, "quote") else shlex.quote
print(" ".join([quote(s) for s in sys.stdin.read().split("\0")][:-1]))
'
}
eval "set -- $(quoted_list <file)"
run_your_command "$#"
If all you need to do is to turn file arguments.txt with contents
arg1
arg2
argN
into my_command arg1 arg2 argN then you can simply use xargs:
xargs -a arguments.txt my_command
You can put additional static arguments in the xargs call, like xargs -a arguments.txt my_command staticArg which will call my_command staticArg arg1 arg2 argN
Here's how I pass contents of a file as an argument to a command:
./foo --bar "$(cat ./bar.txt)"
None of the answers seemed to work for me or were too complicated. Luckily, it's not complicated with xargs (Tested on Ubuntu 20.04).
This works with each arg on a separate line in the file as the OP mentions and was what I needed as well.
cat foo.txt | xargs my_command
One thing to note is that it doesn't seem to work with aliased commands.
The accepted answer works if the command accepts multiple args wrapped in a string. In my case using (Neo)Vim it does not and the args are all stuck together.
xargs does it properly and actually gives you separate arguments supplied to the command.
I suggest using:
command $(echo $(tr '\n' ' ' < parameters.cfg))
Simply trim the end-line characters and replace them with spaces, and then push the resulting string as possible separate arguments with echo.
In my bash shell the following worked like a charm:
cat input_file | xargs -I % sh -c 'command1 %; command2 %; command3 %;'
where input_file is
arg1
arg2
arg3
As evident, this allows you to execute multiple commands with each line from input_file, a nice little trick I learned here.
Both solutions work even when lines have spaces:
readarray -t my_args < foo.txt
my_command "${my_args[#]}"
if readarray doesn't work, replace it with mapfile, they're synonyms.
I formerly tried this one below, but had problems when my_command was a script:
xargs -d '\n' -a foo.txt my_command
After editing #Wesley Rice's answer a couple times, I decided my changes were just getting too big to continue changing his answer instead of writing my own. So, I decided I need to write my own!
Read each line of a file in and operate on it line-by-line like this:
#!/bin/bash
input="/path/to/txt/file"
while IFS= read -r line
do
echo "$line"
done < "$input"
This comes directly from author Vivek Gite here: https://www.cyberciti.biz/faq/unix-howto-read-line-by-line-from-file/. He gets the credit!
Syntax: Read file line by line on a Bash Unix & Linux shell:
1. The syntax is as follows for bash, ksh, zsh, and all other shells to read a file line by line
2. while read -r line; do COMMAND; done < input.file
3. The -r option passed to read command prevents backslash escapes from being interpreted.
4. Add IFS= option before read command to prevent leading/trailing whitespace from being trimmed -
5. while IFS= read -r line; do COMMAND_on $line; done < input.file
And now to answer this now-closed question which I also had: Is it possible to `git add` a list of files from a file? - here's my answer:
Note that FILES_STAGED is a variable containing the absolute path to a file which contains a bunch of lines where each line is a relative path to a file I'd like to do git add on. This code snippet is about to become part of the "eRCaGuy_dotfiles/useful_scripts/sync_git_repo_to_build_machine.sh" file in this project, to enable easy syncing of files in development from one PC (ex: a computer I code on) to another (ex: a more powerful computer I build on): https://github.com/ElectricRCAircraftGuy/eRCaGuy_dotfiles.
while IFS= read -r line
do
echo " git add \"$line\""
git add "$line"
done < "$FILES_STAGED"
References:
Where I copied my answer from: https://www.cyberciti.biz/faq/unix-howto-read-line-by-line-from-file/
For loop syntax: https://www.cyberciti.biz/faq/bash-for-loop/
Related:
How to read contents of file line-by-line and do git add on it: Is it possible to `git add` a list of files from a file?