I have to go through 2 files stored as variables and delete the lines which contain a string stored in another variable:
file1="./file1"
file2="./file2"
text="searched text"
for i in $file1,$file2; do
sed -i.txt '/$text/d' $i
done
The files to exist in the same folder as the script.
I get "No such file or directory". I have been stuck for the past 3 hours on this and honestly I'm pretty much about to quit Linux.
You have a several issues in your script. The right way to do is:
file1="./file1"
file2="./file2"
text="searched text"
for i in "$file1" "$file2"; do
sed -i.txt "/$text/d" "$i"
done
Issues:
for expects a space delimited list of arguments, not comma separated
it is important to enclose your variable expansions in double quotes to prevent word splitting
you need double quotes to enclose the sed expression since single quotes won't expand the variable inside
You could catch these issues through shellcheck and debug mode (bash -x script) as suggested by Charles.
Sorry to say that your shell script is not nicely design. In a shell scripts multi files should not be stored in multiple variables. Suppose you need to do the same operation on 100 different files what will you do? So follow the below style of code. Put all your file names in a file for example filelist.dat now see:-
First put all the file names in filelist.dat and save it
text="searched text"
while read file; do
sed -i.txt '/$text/d' $i
done < filelist.dat
Also not sure whether sed command will work like that. If not working make it like below:-
sed -i.txt 's|'"$text"'|d' $i
Related
$1 is DirectoryName
$2 is txt (file extension)
$3 is 500 (or any other positive integer)
I don't understand the syntax of single quotes. I think what its supposed to do is to find all text files in a directory name passed in as parameter one, then do a "character count" of the txt files that come up in the search and if the character count is over a specified amount passed in parameter 3, then change the file permissions.
however it doesn't actually work. it says "expects an integer". Now it could be that the question is trying to trick me. But I can't get it to work either by changing it slightly. I've tried removing the single quotes (error says "too much data or something), using double quotes instead (something about syntax), I tried using a pipe or >. I read somewhere that single quotes was supposed to make everything inside literal, so that a $asdf would be taken as $asdf literal characters, but then the commands wc -c should have also failed, instead I am told it is expectinng a non=existent integer.
I even tried to play around with substituting variables like
a = wc -c $i
echo $a
which failed with token / syntax errors.
could someone please help with any concepts here that I've totally misunderstood? I have an exam tomorrow and this is past papers, so it's totally for revision only.
Those are not single-quotes but rather backticks (`). You might want to search for 'Command Substitution' in bash's man.
I need to generate filename from three parts, two strings, and one variable.
for f in `cat files.csv`; do echo fastq/$f\_1.fastq.gze; done
files.csv has the following lines:
Sample_11
Sample_12
I need to generate the following:
fastq/Sample_11_1.fastq.gze
fastq/Sample_12_1.fastq.gze
My problem is that I got the below files:
_1.fastq.gze_11
_1.fastq.gze_12
the string after the variable deletes the string before it.
I appreciate any help
Regards
By the way your idiom: for f in cat files.csv should be avoid. Refer: Dangerous Backticks
while read f
do
echo "fastq/${f}/_1.fastq.gze"
done < files.csv
You can make it a one-liner with xargs and printf.
xargs printf 'fastq/%s_1.fastq.gze\n' <files.csv
The function of printf is to apply the first argument (the format string) to each argument in turn.
xargs says to run this command on as many files as it can fit onto the command line (splitting it up into multiple invocations if the input file is too large to fit all the arguments onto a single command line, subject to the ARG_MAX constant in your kernel).
Your best bet, generally, is to wrap the variable name in braces. So, in this case:
echo fastq/${f}_1.fastq.gz
See this answer for some details about the general concept, as well.
Edit: An additional thought looking at the now-provided output makes me think that this isn't a coding problem at all, but rather a conflict between line-endings and the terminal/console program.
Specifically, if the CSV file ends its lines with just a carriage return (ASCII/Unicode 13), the end of Sample_11 might "rewind" the line to the start and overwrite.
In that case, based loosely on this article, I'd recommend replacing cat (if you understandably don't want to re-architect the actual script with something like while) with something that will strip the carriage returns, such as:
for f in $(tr -cd '\011\012\040-\176' < temp.csv)
do
echo fastq/${f}_1.fastq.gze
done
As the cited article explains, Octal 11 is a tab, 12 a line feed, and 40-176 are typeable characters (Unicode will require more thinking). If there aren't any line feeds in the file, for some reason, you probably want to replace that with tr '\015' '\012', which will convert the carriage returns to line feeds.
Of course, at that point, better is to find whatever produces the file and ask them to put reasonable line-endings into their file...
my goal is to sed the 100th line and convert it to a string, then separate the data of the sentence to word
#!/bin/bash
fid=log.txt;
sentence=`expr sed -n '100p' ${fid}`;
for word in $sentence
do
echo $word
done
but apparently this has failed.
expr: syntax error
would somebody please let me know what have I done wrong? previously for number it worked.
The expr does not seem to serve a useful purpose here, and if it did, a sed command would certainly not be a valid or useful thing to pass to it, under most circumstances. You should probably just take it out.
However, the following loop is also problematic. Unquoted variables in shell script are very frequently an error. In this case, you can't quote the thing you pass to the for loop (that would cause the loop to only run once, with the loop variable set to the quoted string) but you also cannot prevent the shell from performing wildcard expansion on the unquoted string. So if the string happened to contain *, the shell will expand that to a list of files in the current directory, for example.
Fortunately, this can all be done in an only slightly more complicated sed script.
sed '100!d;s/[ \t]\+/\n/g;q' "$fid"
That is, if the line number is not 100, delete this line and start over with the next line. Otherwise, we are at line 100; replace runs of whitespace with newlines, (print) and quit.
(The backslash escape codes \t and \n are not universally portable; and \+ for repetition is also an optional extension. I believe there are also sed variants which dislike semicolon as a command separator. Consult your sed manual page, experiment, and if everything else fails, maybe switch to Awk or Perl. Just in case, here is a version which works even on Mac OSX:
sed '100!d
s/[ ][ ]*/\
/g;q' log.txt
The stuff inside the square brackets are a space and a literal tab; in Bash, with default keybindings, type ctrl-V, tab to produce a literal tab.)
Incidentally, this also gets rid of the variable capture antipattern. There are good reasons to capture output to a variable, but if it can be avoided, you often end up with a simpler, more robust and efficient, as well as more idiomatic and elegant script. (I see no reason to put the log file name in a variable, either, in this isolated case; but in a larger script, it might make sense.)
I don't think you need expr command in this case.
expr is used to do calculations. Something like:
expr 1 + 1
Just this one is fine:
sentence=`sed -n '100p' ${fid}`;
#!/bin/bash
fid=log.txt;
sentence=$(sed -n '100p' ${fid});
for word in $sentence
do
echo $word
done
put a dollar sign and parenthesis solve the problem
I have several files named scale1.dat, scale2.dat scale3.dat ... up to scale9.dat.
I want to read these files in do loop one by one and with each file I want to do some manipulation (I want to write the 1st column of each scale*.dat file to scale*.txt).
So my question is, is there a way to read files with similar names. Thanks.
The regular syntax for this is
for file in scale*.dat; do
awk '{print $1}' "$file" >"${file%.dat}.txt"
done
The asterisk * matches any text or no text; if you want to constrain to just single non-zero digits, you could say for file in scale[1-9].dat instead.
In Bash, there is a non-standard additional glob syntax scale{1..9}.dat but this is Bash-only, and so will not work in #!/bin/sh scripts. (Your question has both sh and bash so it's not clear which you require. Your comment that the Bash syntax is not working for you suggests that you may need a POSIX portable solution.) Furthermore, Bash has something called extended globbing, which allows for quite elaborate pattern matching. See also http://mywiki.wooledge.org/glob
For a simple task like this, you don't really need the shell at all, though.
awk 'FNR==1 { if (f) close (f); f=FILENAME; sub(/\.dat/, ".txt", f); }
{ print $1 >f }' scale[1-9]*.dat
(Okay, maybe that's slightly intimidating for a first-timer. But the basic point is that you will often find that the commands you want to use will happily work on multiple files, and so you don't need shell loops at all in those cases.)
I don't think so. Similar names or not, you will have to iterate through all your files (perhaps with a for loop) and use a nested loop to iterate through lines or words or whatever you plan to read from those files.
Alternatively, you can copy your files into one (say, scale-all.dat) and read that single file.
I'm having an extremely hard time figuring out how to echo this:
I keep getting this error:
bash: ![alt: event not found
Using double quotes around it does not work. The using single quotes around it does work, however, I also need to echo strings that have single quotes within them. I wouldn't be able to wrap the string with single quotes then.
Is there a way to echo a string of ANY content?
Thanks.
EDIT: Here is some context. I am making a Markdown renderer that grabs the content of a code editor, then appends every line of the code individually into a text file. I am doing this by doing this:
echo TheLineOfMarkdown > textfile.txt
Unlike in many programing languages, '...' and "..." in Bash do not represent "strings" per se; they quote/escape whatever they contain, but they do not create boundaries that separate arguments. So, for example, these two commands are equivalent:
echo foobar
echo "fo"ob'ar'
So if you need to quote some of an argument with single-quotes, and a different part of the argument has to contain single-quotes — no problem.
For example:
echo '![alt text](https://... "What'"'"'s up, Doc?")'
Another option is to use \, which is similar to '...' except that it only quotes a single character. It can even be used inside double-quotes:
echo "\![alt text](https://... \"What's up, Doc?\")"
For more information, see §3.1.2 "Quoting" in the Bash Reference Manual.
! is annoying. My advice: Use \!.
! invokes history completion, which is also performed inside double-quotes. So you need to single-quote the exclamation mark, but as you say that conflicts with the need to not single-quote other single-quotes.
Remember that you can mix quotes:
$ echo '!'"'"'"'
!'"
(That's just one argument.) But in this case, the backslash is easier to type and quite possibly more readable.