How to insert a text at the beginning of a file? - linux

So far I've been able to find out how to add a line at the beginning of a file but that's not exactly what I want. I'll show it with an example:
File content
some text at the beginning
Result
<added text> some text at the beginning
It's similar but I don't want to create any new line with it...
I would like to do this with sed if possible.

sed can operate on an address:
$ sed -i '1s/^/<added text> /' file
What is this magical 1s you see on every answer here? Line addressing!.
Want to add <added text> on the first 10 lines?
$ sed -i '1,10s/^/<added text> /' file
Or you can use Command Grouping:
$ { echo -n '<added text> '; cat file; } >file.new
$ mv file{.new,}

If you want to add a line at the beginning of a file, you need to add \n at the end of the string in the best solution above.
The best solution will add the string, but with the string, it will not add a line at the end of a file.
sed -i '1s/^/your text\n/' file

If the file is only one line, you can use:
sed 's/^/insert this /' oldfile > newfile
If it's more than one line. one of:
sed '1s/^/insert this /' oldfile > newfile
sed '1,1s/^/insert this /' oldfile > newfile
I've included the latter so that you know how to do ranges of lines. Both of these "replace" the start line marker on their affected lines with the text you want to insert. You can also (assuming your sed is modern enough) use:
sed -i 'whatever command you choose' filename
to do in-place editing.

Use subshell:
echo "$(echo -n 'hello'; cat filename)" > filename
Unfortunately, command substitution will remove newlines at the end of file. So as to keep them one can use:
echo -n "hello" | cat - filename > /tmp/filename.tmp
mv /tmp/filename.tmp filename
Neither grouping nor command substitution is needed.

To insert just a newline:
sed '1i\\'

You can use cat -
printf '%s' "some text at the beginning" | cat - filename

To add a line to the top of the file:
sed -i '1iText to add\'

my two cents:
sed -i '1i /path/of/file.sh' filename
This will work even is the string containing forward slash "/"

Hi with carriage return:
sed -i '1s/^/your text\n/' file

Note that on OS X, sed -i <pattern> file, fails. However, if you provide a backup extension, sed -i old <pattern> file, then file is modified in place while file.old is created. You can then delete file.old in your script.

There is a very easy way:
echo "your header" > headerFile.txt
cat yourFile >> headerFile.txt

PROBLEM: tag a file, at the top of the file, with the base name of the parent directory.
I.e., for
/mnt/Vancouver/Programming/file1
tag the top of file1 with Programming.
SOLUTION 1 -- non-empty files:
bn=${PWD##*/} ## bn: basename
sed -i '1s/^/'"$bn"'\n/' <file>
1s places the text at line 1 of the file.
SOLUTION 2 -- empty or non-empty files:
The sed command, above, fails on empty files. Here is a solution, based on https://superuser.com/questions/246837/how-do-i-add-text-to-the-beginning-of-a-file-in-bash/246841#246841
printf "${PWD##*/}\n" | cat - <file> > temp && mv -f temp <file>
Note that the - in the cat command is required (reads standard input: see man cat for more information). Here, I believe, it's needed to take the output of the printf statement (to STDIN), and cat that and the file to temp ... See also the explanation at the bottom of http://www.linfo.org/cat.html.
I also added -f to the mv command, to avoid being asked for confirmations when overwriting files.
To recurse over a directory:
for file in *; do printf "${PWD##*/}\n" | cat - $file > temp && mv -f temp $file; done
Note also that this will break over paths with spaces; there are solutions, elsewhere (e.g. file globbing, or find . -type f ... -type solutions) for those.
ADDENDUM: Re: my last comment, this script will allow you to recurse over directories with spaces in the paths:
#!/bin/bash
## https://stackoverflow.com/questions/4638874/how-to-loop-through-a-directory-recursively-to-delete-files-with-certain-extensi
## To allow spaces in filenames,
## at the top of the script include: IFS=$'\n'; set -f
## at the end of the script include: unset IFS; set +f
IFS=$'\n'; set -f
# ----------------------------------------------------------------------------
# SET PATHS:
IN="/mnt/Vancouver/Programming/data/claws-test/corpus test/"
# https://superuser.com/questions/716001/how-can-i-get-files-with-numeric-names-using-ls-command
# FILES=$(find $IN -type f -regex ".*/[0-9]*") ## recursive; numeric filenames only
FILES=$(find $IN -type f -regex ".*/[0-9 ]*") ## recursive; numeric filenames only (may include spaces)
# echo '$FILES:' ## single-quoted, (literally) prints: $FILES:
# echo "$FILES" ## double-quoted, prints path/, filename (one per line)
# ----------------------------------------------------------------------------
# MAIN LOOP:
for f in $FILES
do
# Tag top of file with basename of current dir:
printf "[top] Tag: ${PWD##*/}\n\n" | cat - $f > temp && mv -f temp $f
# Tag bottom of file with basename of current dir:
printf "\n[bottom] Tag: ${PWD##*/}\n" >> $f
done
unset IFS; set +f

Just for fun, here is a solution using ed which does not have the problem of not working on an empty file. You can put it into a shell script just like any other answer to this question.
ed Test <<EOF
a
.
0i
<added text>
.
1,+1 j
$ g/^$/d
wq
EOF
The above script adds the text to insert to the first line, and then joins the first and second line. To avoid ed exiting on error with an invalid join, it first creates a blank line at the end of the file and remove it later if it still exists.
Limitations: This script does not work if <added text> is exactly equal to a single period.

echo -n "text to insert " ;tac filename.txt| tac > newfilename.txt
The first tac pipes the file backwards (last line first) so the "text to insert" appears last. The 2nd tac wraps it once again so the inserted line is at the beginning and the original file is in its original order.

The simplest solution I found is:
echo -n "<text to add>" | cat - myFile.txt | tee myFile.txt
Notes:
Remove | tee myFile.txt if you don't want to change the file contents.
Remove the -n parameter if you want to append a full line.
Add &> /dev/null to the end if you don't want to see the output (the generated file).
This can be used to append a shebang to the file. Example:
# make it executable (use u+x to allow only current user)
chmod +x cropImage.ts
# append the shebang
echo '#''!'/usr/bin/env ts-node | cat - cropImage.ts | tee cropImage.ts &> /dev/null
# execute it
./cropImage.ts myImage.png

Another solution with aliases. Add to your init rc/ env file:
addtail () { find . -type f ! -path "./.git/*" -exec sh -c "echo $# >> {}" \; }
addhead () { find . -type f ! -path "./.git/*" -exec sh -c "sed -i '1s/^/$#\n/' {}" \; }
Usage:
addtail "string to add at the beginning of file"
addtail "string to add at the end of file"

With the echo approach, if you are on macOS/BSD like me, lose the -n switch that other people suggest. And I like to define a variable for the text.
So it would be like this:
Header="my complex header that may have difficult chars \"like these quotes\" and line breaks \n\n "
{ echo "$Header"; cat "old.txt"; } > "new.txt"
mv new.txt old.txt

TL;dr -
Consider using ex. Since you want the front of a given line, then the syntax is basically the same as what you might find for sed but the option of "in place editing" is built-in.
I cannot imagine an environment where you have sed but not ex/vi, unless it is a MS Windows box with some special "sed.exe", maybe.
sed & grep sort of evolved from ex / vi, so it might be better to say sed syntax is the same as ex.
You can change the line number to something besides #1 or search for a line and change that one.
source=myFile.txt
Front="This goes IN FRONT "
man true > $source
ex -s ${source} <<EOF
1s/^/$Front/
wq
EOF
$ head -n 3 $source
This goes IN FRONT TRUE(1) User Commands TRUE(1)
NAME
Long version, I recommend ex (or ed if you are one of the cool kids).
I like ex because it is portable, extremely powerful, allows me to write in-place, and/or make backups all without needing GNU (or even BSD) extensions.
Additionally, if you know the ex way, then you know how to do it in vi - and probably vim if that is your jam.
Notice that EOF is not quoted when we use "i"nsert and using echo:
str="+++ TOP +++" && ex -s <<EOF
r!man true
1i
`echo "$str"`
.
"0r!echo "${str}"
wq! true.txt
EOF
0r!echo "${str}" might also be used as shorthand for :0read! or :0r! that you have likely used in vi mode (it is literally the same thing) but the : is optional here and some implementations do not support "r"ead address of zero.
"r"eading directly to the special line #0 (or from line 1) would automatically push everything "down", and then you just :wq to save your changes.
$ head -n 3 true.txt | nl -ba
1 +++ TOP +++
2 TRUE(1) User Commands TRUE(1)
3
Also, most classic sed implementations do not have extensions (like \U&) that ex should have by default.

cat concatenates multiple files. <() sends output of a command as a file. Combining these two, we can insert lines at the beginning and end of a file by,
cat <(echo "line before the file") file.txt <(echo "line after the file")

Related

Removing 10 Characters of Filename in Linux

I just downloaded about 600 files from my server and need to remove the last 11 characters from the filename (not including the extension). I use Ubuntu and I am searching for a command to achieve this.
Some examples are as follows:
aarondyne_kh2_13thstruggle_or_1250556383.mus should be renamed to aarondyne_kh2_13thstruggle_or.mus
aarondyne_kh2_darknessofunknow_1250556659.mp3 should be renamed to aarondyne_kh2_darknessofunknow.mp3
It seems that some duplicates might exist after I do this, but if the command fails to complete and tells me what the duplicates would be, I can always remove those manually.
Try using the rename command. It allows you to rename files based on a regular expression:
The following line should work out for you:
rename 's/_\d+(\.[a-z0-9A-Z]+)$/$1/' *
The following changes will occur:
aarondyne_kh2_13thstruggle_or_1250556383.mus renamed as aarondyne_kh2_13thstruggle_or.mus
aarondyne_kh2_darknessofunknow_1250556659.mp3 renamed as aarondyne_kh2_darknessofunknow.mp3
You can check the actions rename will do via specifying the -n flag, like this:
rename -n 's/_\d+(\.[a-z0-9A-Z]+)$/$1/' *
For more information on how to use rename simply open the manpage via: man rename
Not the prettiest, but very simple:
echo "$filename" | sed -e 's!\(.*\)...........\(\.[^.]*\)!\1\2!'
You'll still need to write the rest of the script, but it's pretty simple.
find . -type f -exec sh -c 'mv {} `echo -n {} | sed -E -e "s/[^/]{10}(\\.[^\\.]+)?$/\\1/"`' ";"
one way to go:
you get a list of your files, one per line (by ls maybe) then:
ls....|awk '{o=$0;sub(/_[^_.]*\./,".",$0);print "mv "o" "$0}'
this will print the mv a b command
e.g.
kent$ echo "aarondyne_kh2_13thstruggle_or_1250556383.mus"|awk '{o=$0;sub(/_[^_.]*\./,".",$0);print "mv "o" "$0}'
mv aarondyne_kh2_13thstruggle_or_1250556383.mus aarondyne_kh2_13thstruggle_or.mus
to execute, just pipe it to |sh
I assume there is no space in your filename.
This script assumes each file has just one extension. It would, for instance, rename "foo.something.mus" to "foo.mus". To keep all extensions, remove one hash mark (#) from the first line of the loop body. It also assumes that the base of each filename has at least 12 character, so that removing 11 doesn't leave you with an empty name.
for f in *; do
ext=${f##*.}
new_f=${base%???????????.$ext}
if [ -f "$new_f" ]; then
echo "Will not rename $f, $new_f already exists" >&2
else
mv "$f" "$new_f"
fi
done

Unix: How to delete files listed in a file

I have a long text file with list of file masks I want to delete
Example:
/tmp/aaa.jpg
/var/www1/*
/var/www/qwerty.php
I need delete them. Tried rm `cat 1.txt` and it says the list is too long.
Found this command, but when I check folders from the list, some of them still have files
xargs rm <1.txt Manual rm call removes files from such folders, so no issue with permissions.
This is not very efficient, but will work if you need glob patterns (as in /var/www/*)
for f in $(cat 1.txt) ; do
rm "$f"
done
If you don't have any patterns and are sure your paths in the file do not contain whitespaces or other weird things, you can use xargs like so:
xargs rm < 1.txt
Assuming that the list of files is in the file 1.txt, then do:
xargs rm -r <1.txt
The -r option causes recursion into any directories named in 1.txt.
If any files are read-only, use the -f option to force the deletion:
xargs rm -rf <1.txt
Be cautious with input to any tool that does programmatic deletions. Make certain that the files named in the input file are really to be deleted. Be especially careful about seemingly simple typos. For example, if you enter a space between a file and its suffix, it will appear to be two separate file names:
file .txt
is actually two separate files: file and .txt.
This may not seem so dangerous, but if the typo is something like this:
myoldfiles *
Then instead of deleting all files that begin with myoldfiles, you'll end up deleting myoldfiles and all non-dot-files and directories in the current directory. Probably not what you wanted.
Use this:
while IFS= read -r file ; do rm -- "$file" ; done < delete.list
If you need glob expansion you can omit quoting $file:
IFS=""
while read -r file ; do rm -- $file ; done < delete.list
But be warned that file names can contain "problematic" content and I would use the unquoted version. Imagine this pattern in the file
*
*/*
*/*/*
This would delete quite a lot from the current directory! I would encourage you to prepare the delete list in a way that glob patterns aren't required anymore, and then use quoting like in my first example.
You could use '\n' for define the new line character as delimiter.
xargs -d '\n' rm < 1.txt
Be careful with the -rf because it can delete what you don't want to if the 1.txt contains paths with spaces. That's why the new line delimiter a bit safer.
On BSD systems, you could use -0 option to use new line characters as delimiter like this:
xargs -0 rm < 1.txt
xargs -I{} sh -c 'rm "{}"' < 1.txt should do what you want. Be careful with this command as one incorrect entry in that file could cause a lot of trouble.
This answer was edited after #tdavies pointed out that the original did not do shell expansion.
You can use this one-liner:
cat 1.txt | xargs echo rm | sh
Which does shell expansion but executes rm the minimum number of times.
Just to provide an another way, you can also simply use the following command
$ cat to_remove
/tmp/file1
/tmp/file2
/tmp/file3
$ rm $( cat to_remove )
In this particular case, due to the dangers cited in other answers, I would
Edit in e.g. Vim and :%s/\s/\\\0/g, escaping all space characters with a backslash.
Then :%s/^/rm -rf /, prepending the command. With -r you don't have to worry to have directories listed after the files contained therein, and with -f it won't complain due to missing files or duplicate entries.
Run all the commands: $ source 1.txt
cat 1.txt | xargs rm -f | bash Run the command will do the following for files only.
cat 1.txt | xargs rm -rf | bash Run the command will do the following recursive behaviour.
Here's another looping example. This one also contains an 'if-statement' as an example of checking to see if the entry is a 'file' (or a 'directory' for example):
for f in $(cat 1.txt); do if [ -f $f ]; then rm $f; fi; done
Here you can use set of folders from deletelist.txt while avoiding some patterns as well
foreach f (cat deletelist.txt)
rm -rf ls | egrep -v "needthisfile|*.cpp|*.h"
end
This will allow file names to have spaces (reproducible example).
# Select files of interest, here, only text files for ex.
find -type f -exec file {} \; > findresult.txt
grep ": ASCII text$" findresult.txt > textfiles.txt
# leave only the path to the file removing suffix and prefix
sed -i -e 's/:.*$//' textfiles.txt
sed -i -e 's/\.\///' textfiles.txt
#write a script that deletes the files in textfiles.txt
IFS_backup=$IFS
IFS=$(echo "\n\b")
for f in $(cat textfiles.txt);
do
rm "$f";
done
IFS=$IFS_backup
# save script as "some.sh" and run: sh some.sh
In case somebody prefers sed and removing without wildcard expansion:
sed -e "s/^\(.*\)$/rm -f -- \'\1\'/" deletelist.txt | /bin/sh
Reminder: use absolute pathnames in the file or make sure you are in the right directory.
And for completeness the same with awk:
awk '{printf "rm -f -- '\''%s'\''\n",$1}' deletelist.txt | /bin/sh
Wildcard expansion will work if the single quotes are remove, but this is dangerous in case the filename contains spaces. This would need to add quotes around the wildcards.

Appending a line to a file only if it does not already exist

I need to add the following line to the end of a config file:
include "/configs/projectname.conf"
to a file called lighttpd.conf
I am looking into using sed to do this, but I can't work out how.
How would I only insert it if the line doesn't already exist?
Just keep it simple :)
grep + echo should suffice:
grep -qxF 'include "/configs/projectname.conf"' foo.bar || echo 'include "/configs/projectname.conf"' >> foo.bar
-q be quiet
-x match the whole line
-F pattern is a plain string
https://linux.die.net/man/1/grep
Edit:
incorporated #cerin and #thijs-wouters suggestions.
This would be a clean, readable and reusable solution using grep and echo to add a line to a file only if it doesn't already exist:
LINE='include "/configs/projectname.conf"'
FILE='lighttpd.conf'
grep -qF -- "$LINE" "$FILE" || echo "$LINE" >> "$FILE"
If you need to match the whole line use grep -xqF
Add -s to ignore errors when the file does not exist, creating a new file with just that line.
Try this:
grep -q '^option' file && sed -i 's/^option.*/option=value/' file || echo 'option=value' >> file
Using sed, the simplest syntax:
sed \
-e '/^\(option=\).*/{s//\1value/;:a;n;ba;q}' \
-e '$aoption=value' filename
This would replace the parameter if it exists, else would add it to the bottom of the file.
Use the -i option if you want to edit the file in-place.
If you want to accept and keep white spaces, and in addition to remove the comment, if the line already exists, but is commented out, write:
sed -i \
-e '/^#\?\(\s*option\s*=\s*\).*/{s//\1value/;:a;n;ba;q}' \
-e '$aoption=value' filename
Please note that neither option nor value must contain a slash /, or you will have to escape it to \/.
To use bash-variables $option and $value, you could write:
sed -i \
-e '/^#\?\(\s*'${option//\//\\/}'\s*=\s*\).*/{s//\1'${value//\//\\/}'/;:a;n;ba;q}' \
-e '$a'${option//\//\\/}'='${value//\//\\/} filename
The bash expression ${option//\//\\/} quotes slashes, it replaces all / with \/.
Note: Just trapped into a problem. In bash you may quote "${option//\//\\/}", but in the sh of busybox, this does not work, so you should avoid the quotes, at least in non-bourne-shells.
All combined in a bash function:
# call option with parameters: $1=name $2=value $3=file
function option() {
name=${1//\//\\/}
value=${2//\//\\/}
sed -i \
-e '/^#\?\(\s*'"${name}"'\s*=\s*\).*/{s//\1'"${value}"'/;:a;n;ba;q}' \
-e '$a'"${name}"'='"${value}" $3
}
Explanation:
/^\(option=\).*/: Match lines that start with option= and (.*) ignore everything after the =. The \(…\) encloses the part we will reuse as \1later.
/^#?(\s*'"${option//////}"'\s*=\s*).*/: Ignore commented out code with # at the begin of line. \? means «optional». The comment will be removed, because it is outside of the copied part in \(…\). \s* means «any number of white spaces» (space, tabulator). White spaces are copied, since they are within \(…\), so you do not lose formatting.
/^\(option=\).*/{…}: If matches a line /…/, then execute the next command. Command to execute is not a single command, but a block {…}.
s//…/: Search and replace. Since the search term is empty //, it applies to the last match, which was /^\(option=\).*/.
s//\1value/: Replace the last match with everything in (…), referenced by \1and the textvalue`
:a;n;ba;q: Set label a, then read next line n, then branch b (or goto) back to label a, that means: read all lines up to the end of file, so after the first match, just fetch all following lines without further processing. Then q quit and therefore ignore everything else.
$aoption=value: At the end of file $, append a the text option=value
More information on sed and a command overview is on my blog:
https://marc.wäckerlin.ch/computer/stream-editor-sed-overview-and-reference
If writing to a protected file, #drAlberT and #rubo77 's answers might not work for you since one can't sudo >>. A similarly simple solution, then, would be to use tee --append (or, on MacOS, tee -a):
LINE='include "/configs/projectname.conf"'
FILE=lighttpd.conf
grep -qF "$LINE" "$FILE" || echo "$LINE" | sudo tee --append "$FILE"
Here's a sed version:
sed -e '\|include "/configs/projectname.conf"|h; ${x;s/incl//;{g;t};a\' -e 'include "/configs/projectname.conf"' -e '}' file
If your string is in a variable:
string='include "/configs/projectname.conf"'
sed -e "\|$string|h; \${x;s|$string||;{g;t};a\\" -e "$string" -e "}" file
If, one day, someone else have to deal with this code as "legacy code", then that person will be grateful if you write a less exoteric code, such as
grep -q -F 'include "/configs/projectname.conf"' lighttpd.conf
if [ $? -ne 0 ]; then
echo 'include "/configs/projectname.conf"' >> lighttpd.conf
fi
another sed solution is to always append it on the last line and delete a pre existing one.
sed -e '$a\' -e '<your-entry>' -e "/<your-entry-properly-escaped>/d"
"properly-escaped" means to put a regex that matches your entry, i.e. to escape all regex controls from your actual entry, i.e. to put a backslash in front of ^$/*?+().
this might fail on the last line of your file or if there's no dangling newline, I'm not sure, but that could be dealt with by some nifty branching...
Here is a one-liner sed which does the job inline. Note that it preserves the location of the variable and its indentation in the file when it exists. This is often important for the context, like when there are comments around or when the variable is in an indented block. Any solution based on "delete-then-append" paradigm fails badly at this.
sed -i '/^[ \t]*option=/{h;s/=.*/=value/};${x;/^$/{s//option=value/;H};x}' test.conf
With a generic pair of variable/value you can write it this way:
var=c
val='12 34' # it handles spaces nicely btw
sed -i '/^[ \t]*'"$var"'=/{h;s/=.*/='"$val"'/};${x;/^$/{s//c='"$val"'/;H};x}' test.conf
Finally, if you want also to keep inline comments, you can do it with a catch group. E.g. if test.conf contains the following:
a=123
# Here is "c":
c=999 # with its own comment and indent
b=234
d=567
Then running this
var='c'
val='"yay"'
sed -i '/^[ \t]*'"$var"'=/{h;s/=[^#]*\(.*\)/='"$val"'\1/;s/'"$val"'#/'"$val"' #/};${x;/^$/{s//'"$var"'='"$val"'/;H};x}' test.conf
Produces that:
a=123
# Here is "c":
c="yay" # with its own comment and indent
b=234
d=567
As an awk-only one-liner:
awk -v s=option=value '/^option=/{$0=s;f=1} {a[++n]=$0} END{if(!f)a[++n]=s;for(i=1;i<=n;i++)print a[i]>ARGV[1]}' file
ARGV[1] is your input file. It is opened and written to in the for loop of theEND block. Opening file for output in the END block replaces the need for utilities like sponge or writing to a temporary file and then mving the temporary file to file.
The two assignments to array a[] accumulate all output lines into a. if(!f)a[++n]=s appends the new option=value if the main awk loop couldn't find option in file.
I have added some spaces (not many) for readability, but you really need just one space in the whole awk program, the space after print.
If file includes # comments they will be preserved.
Here's an awk implementation
/^option *=/ {
print "option=value"; # print this instead of the original line
done=1; # set a flag, that the line was found
next # all done for this line
}
{print} # all other lines -> print them
END { # end of file
if(done != 1) # haven't found /option=/ -> add it at the end of output
print "option=value"
}
Run it using
awk -f update.awk < /etc/fdm_monitor.conf > /etc/fdm_monitor.conf.tmp && \
mv /etc/fdm_monitor.conf.tmp /etc/fdm_monitor.conf
or
awk -f update.awk < /etc/fdm_monitor.conf | sponge /etc/fdm_monitor.conf
EDIT:
As a one-liner:
awk '/^option *=/ {print "option=value";d=1;next}{print}END{if(d!=1)print "option=value"}' /etc/fdm_monitor.conf | sponge /etc/fdm_monitor.conf
use awk
awk 'FNR==NR && /configs.*projectname\.conf/{f=1;next}f==0;END{ if(!f) { print "your line"}} ' file file
sed -i 's/^option.*/option=value/g' /etc/fdm_monitor.conf
grep -q "option=value" /etc/fdm_monitor.conf || echo "option=value" >> /etc/fdm_monitor.conf
here is an awk one-liner:
awk -v s="option=value" '/^option/{f=1;$0=s}7;END{if(!f)print s}' file
this doesn't do in-place change on the file, you can however :
awk '...' file > tmpfile && mv tmpfile file
Using sed, you could say:
sed -e '/option=/{s/.*/option=value/;:a;n;:ba;q}' -e 'aoption=value' filename
This would replace the parameter if it exists, else would add it to the bottom of the file.
Use the -i option if you want to edit the file in-place:
sed -i -e '/option=/{s/.*/option=value/;:a;n;:ba;q}' -e 'aoption=value' filename
sed -i '1 h
1 !H
$ {
x
s/^option.*/option=value/g
t
s/$/\
option=value/
}' /etc/fdm_monitor.conf
Load all the file in buffer, at the end, change all occurence and if no change occur, add to the end
The answers using grep are wrong. You need to add an -x option to match the entire line otherwise lines like #text to add will still match when looking to add exactly text to add.
So the correct solution is something like:
grep -qxF 'include "/configs/projectname.conf"' foo.bar || echo 'include "/configs/projectname.conf"' >> foo.bar
Using sed: It will insert at the end of line. You can also pass in variables as usual of course.
grep -qxF "port=9033" $light.conf
if [ $? -ne 0 ]; then
sed -i "$ a port=9033" $light.conf
else
echo "port=9033 already added"
fi
Using oneliner sed
grep -qxF "port=9033" $lightconf || sed -i "$ a port=9033" $lightconf
Using echo may not work under root, but will work like this. But it will not let you automate things if you are looking to do it since it might ask for password.
I had a problem when I was trying to edit from the root for a particular user. Just adding the $username before was a fix for me.
grep -qxF "port=9033" light.conf
if [ $? -ne 0 ]; then
sudo -u $user_name echo "port=9033" >> light.conf
else
echo "already there"
fi
I elaborated on kev's grep/sed solution by setting variables in order to reduce duplication.
Set the variables in the first line (hint: $_option shall match everything on the line up until the value [including any seperator like = or :]).
_file="/etc/ssmtp/ssmtp.conf" _option="mailhub=" _value="my.domain.tld" \
sh -c '\
grep -q "^$_option" "$_file" \
&& sed -i "s/^$_option.*/$_option$_value/" "$_file" \
|| echo "$_option$_value" >> "$_file"\
'
Mind that the sh -c '...' just has the effect of widening the scope of the variables without the need for an export. (See Setting an environment variable before a command in bash not working for second command in a pipe)
You can use this function to find and search config changes:
#!/bin/bash
#Find and Replace config values
find_and_replace_config () {
file=$1
var=$2
new_value=$3
awk -v var="$var" -v new_val="$new_value" 'BEGIN{FS=OFS="="}match($1, "^\\s*" var "\\s*") {$2=" " new_val}1' "$file" > output.tmp && sudo mv output.tmp $file
}
find_and_replace_config /etc/php5/apache2/php.ini max_execution_time 60
If you want to run this command using a python script within a Linux terminal...
import os,sys
LINE = 'include '+ <insert_line_STRING>
FILE = <insert_file_path_STRING>
os.system('grep -qxF $"'+LINE+'" '+FILE+' || echo $"'+LINE+'" >> '+FILE)
The $ and double quotations had me in a jungle, but this worked.
Thanks everyone
Try:
LINE='include "/configs/projectname.conf"'
sed -n "\|$LINE|q;\$a $LINE" lighttpd.conf >> lighttpd.conf
Use the pipe as separator and quit if $LINE has been found. Otherwise, append $LINE at the end.
Since we only read the file in sed command, I suppose we have no clobber issue in general (it depends on your shell settings).
Using only sed I'd suggest the following solution:
sed -i \
-e 's#^include "/configs/projectname.conf"#include "/configs/projectname.conf"#' \
-e t \
-e '$ainclude "/configs/projectname.conf"' lighttpd.conf
s replace the line include "/configs/projectname.conf with itself (using # as delimiter here)
t if the replacement was successful skip the rest of the commands
$a otherwise jump to the last line and append include "/configs/projectname.conf after it
Almost all of the answers work but not in all scenarios or OS as per my experience. Only thing that worked on older systems and new and different flavours of OS is the following.
I needed to append KUBECONFIG path to bashrc file if it doesnt exist. So, what I did is
I assume that it exists and delete it.
with sed I append the string I want.
sed -i '/KUBECONFIG=/d' ~/.bashrc
echo 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml' >> ~/.bashrc
I needed to edit a file with restricted write permissions so needed sudo. working from ghostdog74's answer and using a temp file:
awk 'FNR==NR && /configs.*projectname\.conf/{f=1;next}f==0;END{ if(!f) { print "your line"}} ' file > /tmp/file
sudo mv /tmp/file file

Add a prefix string to beginning of each line

I have a file as below:
line1
line2
line3
And I want to get:
prefixline1
prefixline2
prefixline3
I could write a Ruby script, but it is better if I do not need to.
prefix will contain /. It is a path, /opt/workdir/ for example.
# If you want to edit the file in-place
sed -i -e 's/^/prefix/' file
# If you want to create a new file
sed -e 's/^/prefix/' file > file.new
If prefix contains /, you can use any other character not in prefix, or
escape the /, so the sed command becomes
's#^#/opt/workdir#'
# or
's/^/\/opt\/workdir/'
awk '$0="prefix"$0' file > new_file
In awk the default action is '{print $0}' (i.e. print the whole line), so the above is equivalent to:
awk '{print "prefix"$0}' file > new_file
With Perl (in place replacement):
perl -pi 's/^/prefix/' file
You can use Vim in Ex mode:
ex -sc '%s/^/prefix/|x' file
% select all lines
s replace
x save and close
If your prefix is a bit complicated, just put it in a variable:
prefix=path/to/file/
Then, you pass that variable and let awk deal with it:
awk -v prefix="$prefix" '{print prefix $0}' input_file.txt
Here is a hightly readable oneliner solution using the ts command from moreutils
$ cat file | ts prefix | tr -d ' '
And how it's derived step by step:
# Step 0. create the file
$ cat file
line1
line2
line3
# Step 1. add prefix to the beginning of each line
$ cat file | ts prefix
prefix line1
prefix line2
prefix line3
# Step 2. remove spaces in the middle
$ cat file | ts prefix | tr -d ' '
prefixline1
prefixline2
prefixline3
If you have Perl:
perl -pe 's/^/PREFIX/' input.file
Using & (the whole part of the input that was matched by the pattern”):
cat in.txt | sed -e "s/.*/prefix&/" > out.txt
OR using back references:
cat in.txt | sed -e "s/\(.*\)/prefix\1/" > out.txt
Using the shell:
#!/bin/bash
prefix="something"
file="file"
while read -r line
do
echo "${prefix}$line"
done <$file > newfile
mv newfile $file
While I don't think pierr had this concern, I needed a solution that would not delay output from the live "tail" of a file, since I wanted to monitor several alert logs simultaneously, prefixing each line with the name of its respective log.
Unfortunately, sed, cut, etc. introduced too much buffering and kept me from seeing the most current lines. Steven Penny's suggestion to use the -s option of nl was intriguing, and testing proved that it did not introduce the unwanted buffering that concerned me.
There were a couple of problems with using nl, though, related to the desire to strip out the unwanted line numbers (even if you don't care about the aesthetics of it, there may be cases where using the extra columns would be undesirable). First, using "cut" to strip out the numbers re-introduces the buffering problem, so it wrecks the solution. Second, using "-w1" doesn't help, since this does NOT restrict the line number to a single column - it just gets wider as more digits are needed.
It isn't pretty if you want to capture this elsewhere, but since that's exactly what I didn't need to do (everything was being written to log files already, I just wanted to watch several at once in real time), the best way to lose the line numbers and have only my prefix was to start the -s string with a carriage return (CR or ^M or Ctrl-M). So for example:
#!/bin/ksh
# Monitor the widget, framas, and dweezil
# log files until the operator hits <enter>
# to end monitoring.
PGRP=$$
for LOGFILE in widget framas dweezil
do
(
tail -f $LOGFILE 2>&1 |
nl -s"^M${LOGFILE}> "
) &
sleep 1
done
read KILLEM
kill -- -${PGRP}
Using ed:
ed infile <<'EOE'
,s/^/prefix/
wq
EOE
This substitutes, for each line (,), the beginning of the line (^) with prefix. wq saves and exits.
If the replacement string contains a slash, we can use a different delimiter for s instead:
ed infile <<'EOE'
,s#^#/opt/workdir/#
wq
EOE
I've quoted the here-doc delimiter EOE ("end of ed") to prevent parameter expansion. In this example, it would work unquoted as well, but it's good practice to prevent surprises if you ever have a $ in your ed script.
Here's a wrapped up example using the sed approach from this answer:
$ cat /path/to/some/file | prefix_lines "WOW: "
WOW: some text
WOW: another line
WOW: more text
prefix_lines
function show_help()
{
IT=$(CAT <<EOF
Usage: PREFIX {FILE}
e.g.
cat /path/to/file | prefix_lines "WOW: "
WOW: some text
WOW: another line
WOW: more text
)
echo "$IT"
exit
}
# Require a prefix
if [ -z "$1" ]
then
show_help
fi
# Check if input is from stdin or a file
FILE=$2
if [ -z "$2" ]
then
# If no stdin exists
if [ -t 0 ]; then
show_help
fi
FILE=/dev/stdin
fi
# Now prefix the output
PREFIX=$1
sed -e "s/^/$PREFIX/" $FILE
You can also achieve this using the backreference technique
sed -i.bak 's/\(.*\)/prefix\1/' foo.txt
You can also use with awk like this
awk '{print "prefix"$0}' foo.txt > tmp && mv tmp foo.txt
Using Pythonize (pz):
pz '"preix"+s' <filename
Simple solution using a for loop on the command line with bash:
for i in $(cat yourfile.txt); do echo "prefix$i"; done
Save the output to a file:
for i in $(cat yourfile.txt); do echo "prefix$i"; done > yourfilewithprefixes.txt
You can do it using AWK
echo example| awk '{print "prefix"$0}'
or
awk '{print "prefix"$0}' file.txt > output.txt
For suffix: awk '{print $0"suffix"}'
For prefix and suffix: awk '{print "prefix"$0"suffix"}'
For people on BSD/OSX systems there's utility called lam, short for laminate. lam -s prefix file will do what you want. I use it in pipelines, eg:
find -type f -exec lam -s "{}: " "{}" \; | fzf
...which will find all files, exec lam on each of them, giving each file a prefix of its own filename. (And pump the output to fzf for searching.)
If you need to prepend a text at the beginning of each line that has a certain string, try following. In the following example, I am adding # at the beginning of each line that has the word "rock" in it.
sed -i -e 's/^.*rock.*/#&/' file_name
SETLOCAL ENABLEDELAYEDEXPANSION
YourPrefix=blabla
YourPath=C:\path
for /f "tokens=*" %%a in (!YourPath!\longfile.csv) do (echo !YourPrefix!%%a) >> !YourPath!\Archive\output.csv

Quick unix command to display specific lines in the middle of a file?

Trying to debug an issue with a server and my only log file is a 20GB log file (with no timestamps even! Why do people use System.out.println() as logging? In production?!)
Using grep, I've found an area of the file that I'd like to take a look at, line 347340107.
Other than doing something like
head -<$LINENUM + 10> filename | tail -20
... which would require head to read through the first 347 million lines of the log file, is there a quick and easy command that would dump lines 347340100 - 347340200 (for example) to the console?
update I totally forgot that grep can print the context around a match ... this works well. Thanks!
I found two other solutions if you know the line number but nothing else (no grep possible):
Assuming you need lines 20 to 40,
sed -n '20,40p;41q' file_name
or
awk 'FNR>=20 && FNR<=40' file_name
When using sed it is more efficient to quit processing after having printed the last line than continue processing until the end of the file. This is especially important in the case of large files and printing lines at the beginning. In order to do so, the sed command above introduces the instruction 41q in order to stop processing after line 41 because in the example we are interested in lines 20-40 only. You will need to change the 41 to whatever the last line you are interested in is, plus one.
# print line number 52
sed -n '52p' # method 1
sed '52!d' # method 2
sed '52q;d' # method 3, efficient on large files
method 3 efficient on large files
fastest way to display specific lines
with GNU-grep you could just say
grep --context=10 ...
No there isn't, files are not line-addressable.
There is no constant-time way to find the start of line n in a text file. You must stream through the file and count newlines.
Use the simplest/fastest tool you have to do the job. To me, using head makes much more sense than grep, since the latter is way more complicated. I'm not saying "grep is slow", it really isn't, but I would be surprised if it's faster than head for this case. That'd be a bug in head, basically.
What about:
tail -n +347340107 filename | head -n 100
I didn't test it, but I think that would work.
I prefer just going into less and
typing 50% to goto halfway the file,
43210G to go to line 43210
:43210 to do the same
and stuff like that.
Even better: hit v to start editing (in vim, of course!), at that location. Now, note that vim has the same key bindings!
You can use the ex command, a standard Unix editor (part of Vim now), e.g.
display a single line (e.g. 2nd one):
ex +2p -scq file.txt
corresponding sed syntax: sed -n '2p' file.txt
range of lines (e.g. 2-5 lines):
ex +2,5p -scq file.txt
sed syntax: sed -n '2,5p' file.txt
from the given line till the end (e.g. 5th to the end of the file):
ex +5,p -scq file.txt
sed syntax: sed -n '2,$p' file.txt
multiple line ranges (e.g. 2-4 and 6-8 lines):
ex +2,4p +6,8p -scq file.txt
sed syntax: sed -n '2,4p;6,8p' file.txt
Above commands can be tested with the following test file:
seq 1 20 > file.txt
Explanation:
+ or -c followed by the command - execute the (vi/vim) command after file has been read,
-s - silent mode, also uses current terminal as a default output,
q followed by -c is the command to quit editor (add ! to do force quit, e.g. -scq!).
I'd first split the file into few smaller ones like this
$ split --lines=50000 /path/to/large/file /path/to/output/file/prefix
and then grep on the resulting files.
If your line number is 100 to read
head -100 filename | tail -1
Get ack
Ubuntu/Debian install:
$ sudo apt-get install ack-grep
Then run:
$ ack --lines=$START-$END filename
Example:
$ ack --lines=10-20 filename
From $ man ack:
--lines=NUM
Only print line NUM of each file. Multiple lines can be given with multiple --lines options or as a comma separated list (--lines=3,5,7). --lines=4-7 also works.
The lines are always output in ascending order, no matter the order given on the command line.
sed will need to read the data too to count the lines.
The only way a shortcut would be possible would there to be context/order in the file to operate on. For example if there were log lines prepended with a fixed width time/date etc.
you could use the look unix utility to binary search through the files for particular dates/times
Use
x=`cat -n <file> | grep <match> | awk '{print $1}'`
Here you will get the line number where the match occurred.
Now you can use the following command to print 100 lines
awk -v var="$x" 'NR>=var && NR<=var+100{print}' <file>
or you can use "sed" as well
sed -n "${x},${x+100}p" <file>
With sed -e '1,N d; M q' you'll print lines N+1 through M. This is probably a bit better then grep -C as it doesn't try to match lines to a pattern.
Building on Sklivvz' answer, here's a nice function one can put in a .bash_aliases file. It is efficient on huge files when printing stuff from the front of the file.
function middle()
{
startidx=$1
len=$2
endidx=$(($startidx+$len))
filename=$3
awk "FNR>=${startidx} && FNR<=${endidx} { print NR\" \"\$0 }; FNR>${endidx} { print \"END HERE\"; exit }" $filename
}
To display a line from a <textfile> by its <line#>, just do this:
perl -wne 'print if $. == <line#>' <textfile>
If you want a more powerful way to show a range of lines with regular expressions -- I won't say why grep is a bad idea for doing this, it should be fairly obvious -- this simple expression will show you your range in a single pass which is what you want when dealing with ~20GB text files:
perl -wne 'print if m/<regex1>/ .. m/<regex2>/' <filename>
(tip: if your regex has / in it, use something like m!<regex>! instead)
This would print out <filename> starting with the line that matches <regex1> up until (and including) the line that matches <regex2>.
It doesn't take a wizard to see how a few tweaks can make it even more powerful.
Last thing: perl, since it is a mature language, has many hidden enhancements to favor speed and performance. With this in mind, it makes it the obvious choice for such an operation since it was originally developed for handling large log files, text, databases, etc.
print line 5
sed -n '5p' file.txt
sed '5q' file.txt
print everything else than line 5
`sed '5d' file.txt
and my creation using google
#!/bin/bash
#removeline.sh
#remove deleting it comes move line xD
usage() { # Function: Print a help message.
echo "Usage: $0 -l LINENUMBER -i INPUTFILE [ -o OUTPUTFILE ]"
echo "line is removed from INPUTFILE"
echo "line is appended to OUTPUTFILE"
}
exit_abnormal() { # Function: Exit with error.
usage
exit 1
}
while getopts l:i:o:b flag
do
case "${flag}" in
l) line=${OPTARG};;
i) input=${OPTARG};;
o) output=${OPTARG};;
esac
done
if [ -f tmp ]; then
echo "Temp file:tmp exist. delete it yourself :)"
exit
fi
if [ -f "$input" ]; then
re_isanum='^[0-9]+$'
if ! [[ $line =~ $re_isanum ]] ; then
echo "Error: LINENUMBER must be a positive, whole number."
exit 1
elif [ $line -eq "0" ]; then
echo "Error: LINENUMBER must be greater than zero."
exit_abnormal
fi
if [ ! -z $output ]; then
sed -n "${line}p" $input >> $output
fi
if [ ! -z $input ]; then
# remove this sed command and this comes move line to other file
sed "${line}d" $input > tmp && cp tmp $input
fi
fi
if [ -f tmp ]; then
rm tmp
fi
You could try this command:
egrep -n "*" <filename> | egrep "<line number>"
Easy with perl! If you want to get line 1, 3 and 5 from a file, say /etc/passwd:
perl -e 'while(<>){if(++$l~~[1,3,5]){print}}' < /etc/passwd
I am surprised only one other answer (by Ramana Reddy) suggested to add line numbers to the output. The following searches for the required line number and colours the output.
file=FILE
lineno=LINENO
wb="107"; bf="30;1"; rb="101"; yb="103"
cat -n ${file} | { GREP_COLORS="se=${wb};${bf}:cx=${wb};${bf}:ms=${rb};${bf}:sl=${yb};${bf}" grep --color -C 10 "^[[:space:]]\\+${lineno}[[:space:]]"; }

Resources