I generated 1000 output files containing a single line with (mistakenly) no line break at the end, so that
cat filnename_* > outfile
generates a file with a single line. I attempted to remedy this using
cat filename_* | tr '\n' ' ' > outfile
but I get exactly the same result - a file with a single line of output. Why doesn't the latter code (which ought to add a line break for each filename_* file) accomplish what I'm trying to do?
I think you could manually append a line break to your 1000 out files, and then cat them all later:
echo | tee -a filename_*
cat filnename_* > outfile
Edit:
Change the first step to echo | tee -a filename_* as #rowboat suggested
If all your files are missing the final linefeed then you can use sed for adding it on the fly:
# with GNU sed
sed '$s/$/\n/' filnename_* > outfile
# with standard sed and bash, zsh, etc...
sed $'$s/$/\\\n/' filnename_* > outfile
# with standard sed and a POSIX shell
sed '$s/$/\
/' filnename_* > outfile
tr '\n' ' ' says to replace each \n with a space; you've already stated the inputs do not contain any \n so the tr does nothing and the final output is just a copy of the input
Setup:
for ((i=1;i<=5;i++))
do
printf 'abcd' > out${i}
done
$ cat out*
abcdabcdabcdabcdabcd
Many commands can process a file and add a \n, it just depends on how much typing you want to do, eg:
$ sed 's/$/&/' out* # or: sed -n '/$/p' out*
abcd
abcd
abcd
abcd
abcd
$ awk '1' out*
abcd
abcd
abcd
abcd
abcd
I'm not coming up with any ideas on how to use cat to append a \n but one idea would be to use a user-defined function; assume we want to name our new function catn (cat and add \n on end):
$ type -a catn # verify name "catn" not currently in use
-bash: type: catn: not found
$ catn() { awk '1' "${#:--}"; } # wrap function definition around the awk solution
$ catn out*
abcd
abcd
abcd
abcd
abcd
Related
I have two files; the first includes patterns ( file.txt) that I want to search for in the second file (file.cfg).
Once the patter is found in "file.cfg" I want to remove it + whatever comes after it until next Hello that comes at the beginning of the line.
I have created the below script but its not working :
#! /bin/bash
cat file.txt | while read LINE; do
echo $LINE
sed -i "/^$LINE$/,/^Hello/{//p;d;}" "file.cfg"
sed -i "/^$LINE$/d" "file.cfg"
done
It was working fine yesterday on test files, Today I have modified the file's name, and It stopped working.
I am not sure If I changed something by mistake, but if I will use the below from Ubuntu command line, it works :
sed -i "/^Hello World$/,/^Hello/{//p;d;}" "file.cfg"
Also, I added echo in the loop, and I can see each line in "file.txt"
To provide further information, I will give an example of what I need to achieve with this code :
"file.txt" contains patterns I need to find a match in "file.cfg" once the pattern is found, I need to remove it, and anything comes after it until next Hello.
sed -i "/^$LINE$/,/^Hello/{//p;d;}" "file.cfg" -- > this line should remove anything in between.
sed -i "/^$LINE$/d" "file.cfg" --- > remove the pattern itself.
+++++++++
See the below example :
the File.cfg is divided into sections; each section starts with Hello
File.txt contains random sections name; I need a script to read the section's name from File.txt and see if it's available in file.cfg , then remove the section name and all of its contents
File.txt :
Hello World
Hello Mohammad
Hello Scripting
File.cfg :
Hellow xyz
a
b
c
Hello World
v
b
n
Hello stack
q
w
e
The final results should be :
Hellow xyz
a
b
c
Hello stack
q
w
e
Once the section name is found, I need to remove everything until the next "Hello" that comes at the beginning of a line ( new section ).
None of the lines start with Hello except the sections name.
$ awk 'NR==FNR{names[$0]; next} $1=="Hello"{f=($0 in names)} !f' File.txt File.cfg
Hellow xyz
a
b
c
Hello stack
q
w
e
If you want to do "inplace" editing then just like GNU sed, which you're currently using, has -i, GNU awk has -i inplace but note that you're working with 2 input files so you need to write to both of them:
awk -i inplace 'NR==FNR{names[$0]; print; next} $1=="Hello"{f=($0 in names)} !f' File.txt File.cfg
or only activate inplace editing for the 2nd one, see the gawk man page for how to control that. IMHO just using a temp output file is simpler:
tmp=$(mktemp) &&
awk 'NR==FNR{names[$0]; next} $1=="Hello"{f=($0 in names)} !f' File.txt File.cfg > "$tmp" &&
mv -- "$tmp" File.cfg
I like #tripleee's suggestion to create a sed script from the patterns file. It results in a single pass and sed making sed appeals to my sense of humor :)
The first step is to generate the sed script:
sed 's|.*|/^&$/, /^Hello/ {\n\t/^&$/ d\n\t/^Hello/! d\n}|' file.txt
/^Hello World$/, /^Hello/ {
/^Hello World$/ d
/^Hello/! d
}
/^Hello Mohammad$/, /^Hello/ {
/^Hello Mohammad$/ d
/^Hello/! d
}
/^Hello Scripting$/, /^Hello/ {
/^Hello Scripting$/ d
/^Hello/! d
}
In a nut shell, for each address range we want to delete everything except the ending pattern.
I'll generate the above sed using bash process substitution and treat it like a sed program file (or it could be put in a temp file):
#!/bin/bash
sed -f <(
sed 's|.*|/^&$/, /^Hello/ {\n\t/^&$/ d\n\t/^Hello/! d\n}|' file.txt
) file.cfg
I left out the -i in place edit option for testing.
For non-destructive testing, compare the expected results with the output of the script:
diff expect <(./remove.sh) && echo ok
I would like to match all lines from a file containing a word, and take all lines under until coming two two newline characters in a row.
I have the following sed code to cut and paste specific lines, but not subsequent lines:
sed 's|.*|/\\<&\\>/{w results\nd}|' teststring | sed -file.bak -f - testfile
How could I modify this to take all subsequent lines?
For example, say I wanted to match lines with 'dog', the following should take the first 3 lines of the 5:
The best kind of an animal is a dog, for sure
-man's best friend
-related to wolves
Racoons are not cute
Is there a way to do this?
This should do:
awk '/dog/ {f=1} /^$/ {f=0} f {print > "new"} !f {print > "tmp"}' file && mv tmp file
It will set f to true if word dog is found, then if a blank line is found set f to false.
If f is true, print to new file.
If f is false, print to tmp file.
Copy tmp file to original file
Edit: Can be shorten some:
awk '/dog/ {f=1} /^$/ {f=0} {print > (f?"new":"tmp")}' file && mv tmp file
Edit2: as requested add space for every section in the new file:
awk '/dog/ {f=1;print ""> "new"} /^$/ {f=0} {print > (f?"new":"tmp")}' file && mv tmp file
If the original files does contains tabs or spaces instead of just a blank line after each dog section, change from /^$/ to /^[ \t]*$/
This might work for you (GNU sed):
sed 's|.*|/\\<&\\>/ba|' stringFile |
sed -f - -e 'b;:a;w resultFile' -e 'n;/^$/!ba' file
Build a set of regexps from the stringFile and send matches to :a. Then write the matched line and any further lines until an empty line (or end of file) to the resultFile.
N.B. The results could be sent directly to resultFile,using:
sed 's#.*#/\\<&\\>/ba#' stringFile |
sed -nf - -e 'b;:a;p;n;/^$/!ba' file > resultFile
To cut the matches from the original file use:
sed 's|.*|/\\<&\\>/ba|' stringFile |
sed -f - -e 'b;:a;N;/\n\s*$/!ba;w resultFile' -e 's/.*//p;d' file
Is this what you're trying to do?
$ awk -v RS= '/dog/' file
The best kind of an animal is a dog, for sure
-man's best friend
-related to wolves
Could you please try following.
awk '/dog/{count="";found=1} found && ++count<4' Input_file > temp && mv temp Input_file
I have the text file with the column of the numbers, that I need to transform to the line with the numbers separated by ', '
For example:
$ cat file.txt
1034008
1034043
10340431
1034051
Then I use tr:
tr "\n" "', '" < file.txt > file2.txt
But, result is:
$ cat file2.txt
1034008'1034043'10340431'1034051
So, what I need to do to get the correct result?
tr can only to one-to-one mapping, not one-to-many
$ # convert all input lines to one line
$ # using , as separator, cannot give multiple character separator
$ paste -sd, ip.txt
1034008,1034043,10340431,1034051
$ # post process it
$ paste -sd, ip.txt | sed 's/,/, /g'
1034008, 1034043, 10340431, 1034051
$ # or use a tool that allows input record separator manipulation
$ perl -pe 's/\n/, / unless eof' ip.txt
1034008, 1034043, 10340431, 1034051
1.We can do this by sed.
The command N of sed can reads the next line into pattern space.So we use N to merge 2 lines into 1.But how to merge all lines into one?
We can set a lebel at the beginning and use t label to jump to the lebel to make a loop.
$ sed ':myLebel;N;s/\n/, /; t myLebel; ' file.txt > file2.txt
$ cat file2.txt
1034008, 1034043, 10340431, 1034051
2.In your question, we can use xargs to read all content into one line which is delimited by space,and then use sed to replace space to the strings you want.
$ cat file.txt | xargs |sed 's/ /, /g' > file2.txt
$ cat file2.txt
1034008, 1034043, 10340431, 1034051
Refer to:
How the 'N' command works in sed?
https://www.thegeekstuff.com/2009/12/unix-sed-tutorial-6-examples-for-sed-branching-operation/
pure bash, to avoid external commands (faster)
tk="$(< file.txt)"
echo "${tk//$'\n'/, }" > file2.txt
I want to add a text to the end of the first line of a file using a bash script.
The file is /etc/cmdline.txt which does not allow line breaks and needs new commands seperated by a blank, so text i want to add realy needs to be in first line.
What i got so far is:
line=' bcm2708.w1_gpio_pin=20'
file=/boot/cmdline.txt
if ! grep -q -x -F -e "$line" <"$file"; then
printf '%s' "$line\n" >>"$file"
fi
But that appends the text after the line break of the first line, so the result is wrong.
I either need to trim the file contend, add my text and a line feed or somehow just add it to first line of file not touching the rest somehow, but my knowledge of bash scripts is not good enough to find a solution here, and all the examples i find online add beginning/end of every line in a file, not just the first line.
This sed command will add 123 to end of first line of your file.
sed ' 1 s/.*/&123/' yourfile.txt
also
sed '1 s/$/ 123/' yourfile.txt
For appending result to the same file you have to use -i switch :
sed -i ' 1 s/.*/&123/' yourfile.txt
This is a solution to add "ok" at the first line on /etc/passwd, I think you can use this in your script with a little bit of 'tuning' :
$ awk 'NR==1{printf "%s %s\n", $0, "ok"}' /etc/passwd
root:x:0:0:root:/root:/bin/bash ok
To edit a file, you can use ed, the standard editor:
line=' bcm2708.w1_gpio_pin=20'
file=/boot/cmdline.txt
if ! grep -q -x -F -e "$line" <"$file"; then
ed -s "$file" < <(printf '%s\n' 1 a "$line" . 1,2j w q)
fi
ed's commands:
1: go to line 1
a: append (this will insert after the current line)
We're in insert mode and we're inserting the expansion of $line
.: stop insert mode
1,2j join lines 1 and 2
w: write
q: quit
This can be used to append a variable to the first line of input:
awk -v suffix="$suffix" '{print NR==1 ? $0 suffix : $0}'
This will work even if the variable could potentially contain regex formatting characters.
Example:
suffix=' [first line]'
cat input.txt | awk -v suffix="$suffix" '{print NR==1 ? $0 suffix : $0}' > output.txt
input.txt:
Line 1
Line 2
Line 3
output.txt:
Line 1 [first line]
Line 2
Line 3
I am in need of converting the below in multiple files. Text need not be same, but will be in the same format and length
File 1:
XXXxx81511
XXX is Present
abcdefg
07/09/2014
YES
1
XXX
XXX-XXXX
File 2:
XXXxx81511
XXX is Present
abcdefg
07/09/2014
YES
1
XXX
XXX-XXXX
TO
XXXxx81511,XXX is Present,abcdefg,07/09/2014,YES,1,XXXXXX-XXXX
XXXxx81511,XXX is Present,abcdefg,07/09/2014,YES,1,XXXXXX-XXXX
Basically converting row to column and appending to a new file while adding commas to separate them.
I am trying cat filename | tr '\n' ',' but the results do get added in the same line. like this
XXXxx81511,XXX is Present,abcdefg,07/09/2014,YES,1,XXXXXX-XXXX,XXXxx81511,XXX is Present,abcdefg,07/09/2014,YES,1,XXXXXX-XXXX
Use:
paste -sd, file1 file2 .... fileN
#e.g.
paste -sd, *.txt file*
prints
XXXxx81511,XXX is Present,abcdefg,07/09/2014,YES,1,XXX,XXX-XXXX
XXXxx81511,XXX is Present,abcdefg,07/09/2014,YES,1,XXX,XXX-XXXX
and if you need the empty line after each one
paste -sd, file* | sed G
prints
XXXxx81511,XXX is Present,abcdefg,07/09/2014,YES,1,XXX,XXX-XXXX
XXXxx81511,XXX is Present,abcdefg,07/09/2014,YES,1,XXX,XXX-XXXX
Short perl variant:
perl -pe 'eof||s|$/|,|' files....
You need to insert an echo after tr. Use a script like this:
for f in file1 file2; do
tr '\n' ',' < "$f"; echo
done > files.output
Use a for loop:
for f in file*; do sed ':a;N;$!ba;s/\n/,/g' < $f; done
The sed code was taken from sed: How can I replace a newline (\n)?. tr '\n' ',' didn't work on my limited test setup.
perl -ne 'chomp; print $_ . (($. % 8) ? "," : "\n")' f*
where:
-n reads the file line by line but doesn't print each line
-e executes the code from the command line
8 number of lines in each file
f* glob for files (replace with something that will select all
your files). If you need a specific order, you will probably need
something more complicated here.