I'm sure this question has been answered somewhere, but while searching for it, I can't find my exact scenario. I have two files that I am concatenating into a single file, but I am also adding user input between the two files. The problem is that a newline is being inserted after the first file, the rest of it works as desired.
touch newFile
cat file1 > newFile
echo -n $userInput >> newFile
cat file2 >> newFile
How do I prevent or remove the newline when file1 is added to newFile? If I cat file1 there seems to be a newline added by cat but everything I see about cat says it doesn't do that. If I vim file1 there's not a blank line at the end of the file that would indicate the newline is a part of the file, so either cat is actually adding a newline, or the redirect > is doing it, or echo adds a newline at the beginning of its output, none of which would be desirable in this situation. One solution I saw was to use
cat file1 | tr -d '\n'
but that discards all the newlines in the file, also not desirable. So, to repeat my question:
How do I cat file1 into the new file and add user input without adding the newline between them?
(cat is not a requirement, but I am not familiar with printf, so if that's the solution then please elaborate on its use).
With these inputs:
userInput="Test Test Test"
echo "Line 1
Line 2
Line 3" >file1
echo "Line 4
Line 5
Line 6" >file2
I would do:
printf "%s%s%s" "$(cat file1)" "$userInput" "$(cat file2)" >newfile
The creation of >newfile is equivalent to touch and adding content in your first step. A bit easier to see intent with this.
I get:
$ cat newfile
Line 1
Line 2
Line 3Test Test TestLine 4
Line 5
Line 6
Like all other Unix tools, Vim considers \n a line terminator and not a line separator.
This means that a linefeed after the last piece of text will be considered part of the last line, and will not show an additional blank line.
If there is no trailing linefeed, Vim will instead show [noeol] in the status bar when the file is loaded:
foo
~
~
~
~
~
"file" [noeol] 1L, 3C 1,1 All
^---- Here
So no, the linefeed is definitely part of your file and not being added by bash in any way.
If you want to strip all trailing linefeeds, you can do this as a side effect of command expansion:
printf '%s' "$(<file1)" >> newfile
touch newFile
echo -n "$(cat file1)" > newFile
echo -n $userInput >> newFile
cat file2 >> newFile
That did the trick.
Related
I have a big txt file where I want to add a fasta symbol before every line as a new line. I tried with sed, I can add it before the line but not as a new line.
I have file like this
AAAAAAAAAACA
AAAAAAAAAACTTAT
AAAAAAAAACATGTGACTA
AAAAAAAAACTTATTCTTTTT
AAAAAAAACATGTGACT
And I want something like this
>
AAAAAAAAAACA
>
AAAAAAAAAACTTAT
>
AAAAAAAAACATGTGACTA
>
AAAAAAAAACTTATTCTTTTT
>
AAAAAAAACATGTGACT
Thanks,
you can use the sed command like this:
sed 's/^/>\n/g' file.txt> file2.txt
SED
# If you want to edit the file in-place
sed -i -e 's/^/prefix/' file
# If you want to create a new file
sed -e 's/^/prefix/' file > file.new
How?
simple format is : sed 's/x/y/g' - with this, x will be replaced with y
Use -e to add the script to the command (better to use it always) (More described here (click here))
Use -i for input in the same file
AWK
awk '{print ">\n"$0}' file >> newFile
How?
Simple format : awk '{print $0}' file will print whole lines as it is.
Then just add the prefix you need "xyz\n"\
WHILE LOOP
while read line ;
do
echo -e ">\n$line" ;
done < file
With this you can play around your each lines
If you want to do for particular lines alone use the file input as done < <(cat file | grep "TT")
If you want to add some condition, if line contains... then.. echo "prefix"$line is also possible.
Note: it removes all leading and trailing whitespace characters (spaces and tabs, or any whitespace characters present in IFS)
Note:
If prefix contains /, you can use any other character not in prefix, or
escape the /, so the sed command becomes
's#^#/opt/workdir#'
# or
's/^/\/opt\/workdir/'
$ awk '$0=">\n"$0' <(echo -e "foo\nbar")
>
foo
>
bar
# change inplace
$ cat file
foo
bar
baz
$ awk -i inplace '$0=">\n"$0' file
$ cat file
>
foo
>
bar
>
baz
I have a text file, say, text.
What I want to do is to put the text file into pipe (as in cat text |), but with added line at top and bottom - but without creating a new file (because the text file is kind of big) or modifying it.
Is that possible? I was thinking echo $(echo "line"; cat text; echo "line") but I don't like that.
From man bash:
(list) list is executed in a subshell environment (see COMMAND EXECUTION ENVIRONMENT below). Variable assignments and builtin commands that affect the shell's environment do not remain in effect after the command completes.
The return status is the exit status of list.
So this should work:
(echo first line; cat file; echo last line) | some_command
As suggested by chepner, when you don't need a subshell you can use {} instead of (). In this case, the ; is mandatory including for the last command in the group.
In the comments Pyrolistical remembers this also works if you have an incoming pipe instead of a file:
some_program | (echo first line; cat -; echo last line) | some-other-command
As in many unix command line applications, if the filename argument for cat is - it means "read from standard input".
Copied from this question Add new lines to top and bottom of every text file:
cat text |sed -e '1 i beginning_line'| sed -e '$s#$#\nending_line#' |
it seems to work.
add lines to the pipe output,
add a line to the top: (source - #Karel Bilek's comment)
cat text.txt | sed -e '1 i my first line'
add a line to the bottom: ( source - https://askubuntu.com/a/968585 )
cat text.txt | sed -e '$a my last line'
e.g: try this with single quotes
cat text.txt | sed -e '1 i my first line' | sed -e '$a my last line'
output:
my first line
this is
my
text
file
my last line
I have a file as below:
line1
line2
line3
And I want to get:
prefixline1
prefixline2
prefixline3
I could write a Ruby script, but it is better if I do not need to.
prefix will contain /. It is a path, /opt/workdir/ for example.
# If you want to edit the file in-place
sed -i -e 's/^/prefix/' file
# If you want to create a new file
sed -e 's/^/prefix/' file > file.new
If prefix contains /, you can use any other character not in prefix, or
escape the /, so the sed command becomes
's#^#/opt/workdir#'
# or
's/^/\/opt\/workdir/'
awk '$0="prefix"$0' file > new_file
In awk the default action is '{print $0}' (i.e. print the whole line), so the above is equivalent to:
awk '{print "prefix"$0}' file > new_file
With Perl (in place replacement):
perl -pi 's/^/prefix/' file
You can use Vim in Ex mode:
ex -sc '%s/^/prefix/|x' file
% select all lines
s replace
x save and close
If your prefix is a bit complicated, just put it in a variable:
prefix=path/to/file/
Then, you pass that variable and let awk deal with it:
awk -v prefix="$prefix" '{print prefix $0}' input_file.txt
Here is a hightly readable oneliner solution using the ts command from moreutils
$ cat file | ts prefix | tr -d ' '
And how it's derived step by step:
# Step 0. create the file
$ cat file
line1
line2
line3
# Step 1. add prefix to the beginning of each line
$ cat file | ts prefix
prefix line1
prefix line2
prefix line3
# Step 2. remove spaces in the middle
$ cat file | ts prefix | tr -d ' '
prefixline1
prefixline2
prefixline3
If you have Perl:
perl -pe 's/^/PREFIX/' input.file
Using & (the whole part of the input that was matched by the pattern”):
cat in.txt | sed -e "s/.*/prefix&/" > out.txt
OR using back references:
cat in.txt | sed -e "s/\(.*\)/prefix\1/" > out.txt
Using the shell:
#!/bin/bash
prefix="something"
file="file"
while read -r line
do
echo "${prefix}$line"
done <$file > newfile
mv newfile $file
While I don't think pierr had this concern, I needed a solution that would not delay output from the live "tail" of a file, since I wanted to monitor several alert logs simultaneously, prefixing each line with the name of its respective log.
Unfortunately, sed, cut, etc. introduced too much buffering and kept me from seeing the most current lines. Steven Penny's suggestion to use the -s option of nl was intriguing, and testing proved that it did not introduce the unwanted buffering that concerned me.
There were a couple of problems with using nl, though, related to the desire to strip out the unwanted line numbers (even if you don't care about the aesthetics of it, there may be cases where using the extra columns would be undesirable). First, using "cut" to strip out the numbers re-introduces the buffering problem, so it wrecks the solution. Second, using "-w1" doesn't help, since this does NOT restrict the line number to a single column - it just gets wider as more digits are needed.
It isn't pretty if you want to capture this elsewhere, but since that's exactly what I didn't need to do (everything was being written to log files already, I just wanted to watch several at once in real time), the best way to lose the line numbers and have only my prefix was to start the -s string with a carriage return (CR or ^M or Ctrl-M). So for example:
#!/bin/ksh
# Monitor the widget, framas, and dweezil
# log files until the operator hits <enter>
# to end monitoring.
PGRP=$$
for LOGFILE in widget framas dweezil
do
(
tail -f $LOGFILE 2>&1 |
nl -s"^M${LOGFILE}> "
) &
sleep 1
done
read KILLEM
kill -- -${PGRP}
Using ed:
ed infile <<'EOE'
,s/^/prefix/
wq
EOE
This substitutes, for each line (,), the beginning of the line (^) with prefix. wq saves and exits.
If the replacement string contains a slash, we can use a different delimiter for s instead:
ed infile <<'EOE'
,s#^#/opt/workdir/#
wq
EOE
I've quoted the here-doc delimiter EOE ("end of ed") to prevent parameter expansion. In this example, it would work unquoted as well, but it's good practice to prevent surprises if you ever have a $ in your ed script.
Here's a wrapped up example using the sed approach from this answer:
$ cat /path/to/some/file | prefix_lines "WOW: "
WOW: some text
WOW: another line
WOW: more text
prefix_lines
function show_help()
{
IT=$(CAT <<EOF
Usage: PREFIX {FILE}
e.g.
cat /path/to/file | prefix_lines "WOW: "
WOW: some text
WOW: another line
WOW: more text
)
echo "$IT"
exit
}
# Require a prefix
if [ -z "$1" ]
then
show_help
fi
# Check if input is from stdin or a file
FILE=$2
if [ -z "$2" ]
then
# If no stdin exists
if [ -t 0 ]; then
show_help
fi
FILE=/dev/stdin
fi
# Now prefix the output
PREFIX=$1
sed -e "s/^/$PREFIX/" $FILE
You can also achieve this using the backreference technique
sed -i.bak 's/\(.*\)/prefix\1/' foo.txt
You can also use with awk like this
awk '{print "prefix"$0}' foo.txt > tmp && mv tmp foo.txt
Using Pythonize (pz):
pz '"preix"+s' <filename
Simple solution using a for loop on the command line with bash:
for i in $(cat yourfile.txt); do echo "prefix$i"; done
Save the output to a file:
for i in $(cat yourfile.txt); do echo "prefix$i"; done > yourfilewithprefixes.txt
You can do it using AWK
echo example| awk '{print "prefix"$0}'
or
awk '{print "prefix"$0}' file.txt > output.txt
For suffix: awk '{print $0"suffix"}'
For prefix and suffix: awk '{print "prefix"$0"suffix"}'
For people on BSD/OSX systems there's utility called lam, short for laminate. lam -s prefix file will do what you want. I use it in pipelines, eg:
find -type f -exec lam -s "{}: " "{}" \; | fzf
...which will find all files, exec lam on each of them, giving each file a prefix of its own filename. (And pump the output to fzf for searching.)
If you need to prepend a text at the beginning of each line that has a certain string, try following. In the following example, I am adding # at the beginning of each line that has the word "rock" in it.
sed -i -e 's/^.*rock.*/#&/' file_name
SETLOCAL ENABLEDELAYEDEXPANSION
YourPrefix=blabla
YourPath=C:\path
for /f "tokens=*" %%a in (!YourPath!\longfile.csv) do (echo !YourPrefix!%%a) >> !YourPath!\Archive\output.csv
How do I copy the first few lines of a giant file and add a line of text at the end of it, using some Linux commands?
The head command can get the first n lines. Variations are:
head -7 file
head -n 7 file
head -7l file
which will get the first 7 lines of the file called "file". The command to use depends on your version of head. Linux will work with the first one.
To append lines to the end of the same file, use:
echo 'first line to add' >> file
echo 'second line to add' >> file
echo 'third line to add' >> file
or:
echo 'first line to add
second line to add
third line to add' >> file
to do it in one hit.
So, tying these two ideas together, if you wanted to get the first 10 lines of the input.txt file to output.txt and append a line with five "=" characters, you could use something like:
( head -10 input.txt ; echo '=====' ) > output.txt
In this case, we do both operations in a sub-shell so as to consolidate the output streams into one, which is then used to create or overwrite the output file.
I am assuming what you are trying to achieve is to insert a line after the first few lines of of a textfile.
head -n10 file.txt >> newfile.txt
echo "your line >> newfile.txt
tail -n +10 file.txt >> newfile.txt
If you don't want to rest of the lines from the file, just skip the tail part.
First few lines: man head.
Append lines: use the >> operator (?) in Bash:
echo 'This goes at the end of the file' >> file
sed -n '1,10p' filename > newfile
echo 'This goes at the end of the file' >> newfile
Trying to debug an issue with a server and my only log file is a 20GB log file (with no timestamps even! Why do people use System.out.println() as logging? In production?!)
Using grep, I've found an area of the file that I'd like to take a look at, line 347340107.
Other than doing something like
head -<$LINENUM + 10> filename | tail -20
... which would require head to read through the first 347 million lines of the log file, is there a quick and easy command that would dump lines 347340100 - 347340200 (for example) to the console?
update I totally forgot that grep can print the context around a match ... this works well. Thanks!
I found two other solutions if you know the line number but nothing else (no grep possible):
Assuming you need lines 20 to 40,
sed -n '20,40p;41q' file_name
or
awk 'FNR>=20 && FNR<=40' file_name
When using sed it is more efficient to quit processing after having printed the last line than continue processing until the end of the file. This is especially important in the case of large files and printing lines at the beginning. In order to do so, the sed command above introduces the instruction 41q in order to stop processing after line 41 because in the example we are interested in lines 20-40 only. You will need to change the 41 to whatever the last line you are interested in is, plus one.
# print line number 52
sed -n '52p' # method 1
sed '52!d' # method 2
sed '52q;d' # method 3, efficient on large files
method 3 efficient on large files
fastest way to display specific lines
with GNU-grep you could just say
grep --context=10 ...
No there isn't, files are not line-addressable.
There is no constant-time way to find the start of line n in a text file. You must stream through the file and count newlines.
Use the simplest/fastest tool you have to do the job. To me, using head makes much more sense than grep, since the latter is way more complicated. I'm not saying "grep is slow", it really isn't, but I would be surprised if it's faster than head for this case. That'd be a bug in head, basically.
What about:
tail -n +347340107 filename | head -n 100
I didn't test it, but I think that would work.
I prefer just going into less and
typing 50% to goto halfway the file,
43210G to go to line 43210
:43210 to do the same
and stuff like that.
Even better: hit v to start editing (in vim, of course!), at that location. Now, note that vim has the same key bindings!
You can use the ex command, a standard Unix editor (part of Vim now), e.g.
display a single line (e.g. 2nd one):
ex +2p -scq file.txt
corresponding sed syntax: sed -n '2p' file.txt
range of lines (e.g. 2-5 lines):
ex +2,5p -scq file.txt
sed syntax: sed -n '2,5p' file.txt
from the given line till the end (e.g. 5th to the end of the file):
ex +5,p -scq file.txt
sed syntax: sed -n '2,$p' file.txt
multiple line ranges (e.g. 2-4 and 6-8 lines):
ex +2,4p +6,8p -scq file.txt
sed syntax: sed -n '2,4p;6,8p' file.txt
Above commands can be tested with the following test file:
seq 1 20 > file.txt
Explanation:
+ or -c followed by the command - execute the (vi/vim) command after file has been read,
-s - silent mode, also uses current terminal as a default output,
q followed by -c is the command to quit editor (add ! to do force quit, e.g. -scq!).
I'd first split the file into few smaller ones like this
$ split --lines=50000 /path/to/large/file /path/to/output/file/prefix
and then grep on the resulting files.
If your line number is 100 to read
head -100 filename | tail -1
Get ack
Ubuntu/Debian install:
$ sudo apt-get install ack-grep
Then run:
$ ack --lines=$START-$END filename
Example:
$ ack --lines=10-20 filename
From $ man ack:
--lines=NUM
Only print line NUM of each file. Multiple lines can be given with multiple --lines options or as a comma separated list (--lines=3,5,7). --lines=4-7 also works.
The lines are always output in ascending order, no matter the order given on the command line.
sed will need to read the data too to count the lines.
The only way a shortcut would be possible would there to be context/order in the file to operate on. For example if there were log lines prepended with a fixed width time/date etc.
you could use the look unix utility to binary search through the files for particular dates/times
Use
x=`cat -n <file> | grep <match> | awk '{print $1}'`
Here you will get the line number where the match occurred.
Now you can use the following command to print 100 lines
awk -v var="$x" 'NR>=var && NR<=var+100{print}' <file>
or you can use "sed" as well
sed -n "${x},${x+100}p" <file>
With sed -e '1,N d; M q' you'll print lines N+1 through M. This is probably a bit better then grep -C as it doesn't try to match lines to a pattern.
Building on Sklivvz' answer, here's a nice function one can put in a .bash_aliases file. It is efficient on huge files when printing stuff from the front of the file.
function middle()
{
startidx=$1
len=$2
endidx=$(($startidx+$len))
filename=$3
awk "FNR>=${startidx} && FNR<=${endidx} { print NR\" \"\$0 }; FNR>${endidx} { print \"END HERE\"; exit }" $filename
}
To display a line from a <textfile> by its <line#>, just do this:
perl -wne 'print if $. == <line#>' <textfile>
If you want a more powerful way to show a range of lines with regular expressions -- I won't say why grep is a bad idea for doing this, it should be fairly obvious -- this simple expression will show you your range in a single pass which is what you want when dealing with ~20GB text files:
perl -wne 'print if m/<regex1>/ .. m/<regex2>/' <filename>
(tip: if your regex has / in it, use something like m!<regex>! instead)
This would print out <filename> starting with the line that matches <regex1> up until (and including) the line that matches <regex2>.
It doesn't take a wizard to see how a few tweaks can make it even more powerful.
Last thing: perl, since it is a mature language, has many hidden enhancements to favor speed and performance. With this in mind, it makes it the obvious choice for such an operation since it was originally developed for handling large log files, text, databases, etc.
print line 5
sed -n '5p' file.txt
sed '5q' file.txt
print everything else than line 5
`sed '5d' file.txt
and my creation using google
#!/bin/bash
#removeline.sh
#remove deleting it comes move line xD
usage() { # Function: Print a help message.
echo "Usage: $0 -l LINENUMBER -i INPUTFILE [ -o OUTPUTFILE ]"
echo "line is removed from INPUTFILE"
echo "line is appended to OUTPUTFILE"
}
exit_abnormal() { # Function: Exit with error.
usage
exit 1
}
while getopts l:i:o:b flag
do
case "${flag}" in
l) line=${OPTARG};;
i) input=${OPTARG};;
o) output=${OPTARG};;
esac
done
if [ -f tmp ]; then
echo "Temp file:tmp exist. delete it yourself :)"
exit
fi
if [ -f "$input" ]; then
re_isanum='^[0-9]+$'
if ! [[ $line =~ $re_isanum ]] ; then
echo "Error: LINENUMBER must be a positive, whole number."
exit 1
elif [ $line -eq "0" ]; then
echo "Error: LINENUMBER must be greater than zero."
exit_abnormal
fi
if [ ! -z $output ]; then
sed -n "${line}p" $input >> $output
fi
if [ ! -z $input ]; then
# remove this sed command and this comes move line to other file
sed "${line}d" $input > tmp && cp tmp $input
fi
fi
if [ -f tmp ]; then
rm tmp
fi
You could try this command:
egrep -n "*" <filename> | egrep "<line number>"
Easy with perl! If you want to get line 1, 3 and 5 from a file, say /etc/passwd:
perl -e 'while(<>){if(++$l~~[1,3,5]){print}}' < /etc/passwd
I am surprised only one other answer (by Ramana Reddy) suggested to add line numbers to the output. The following searches for the required line number and colours the output.
file=FILE
lineno=LINENO
wb="107"; bf="30;1"; rb="101"; yb="103"
cat -n ${file} | { GREP_COLORS="se=${wb};${bf}:cx=${wb};${bf}:ms=${rb};${bf}:sl=${yb};${bf}" grep --color -C 10 "^[[:space:]]\\+${lineno}[[:space:]]"; }