Is there an invocation of sed todo in-place editing without backups that works both on Linux and Mac? While the BSD sed shipped with OS X seems to need sed -i '' …, the GNU sed Linux distributions usually come with interprets the quotes as empty input file name (instead of the backup extension), and needs sed -i … instead.
Is there any command line syntax which works with both flavors, so I can use the same script on both systems?
If you really want to just use sed -i the 'easy' way, the following DOES work on both GNU and BSD/Mac sed:
sed -i.bak 's/foo/bar/' filename
Note the lack of space and the dot.
Proof:
# GNU sed
% sed --version | head -1
GNU sed version 4.2.1
% echo 'foo' > file
% sed -i.bak 's/foo/bar/' ./file
% ls
file file.bak
% cat ./file
bar
# BSD sed
% sed --version 2>&1 | head -1
sed: illegal option -- -
% echo 'foo' > file
% sed -i.bak 's/foo/bar/' ./file
% ls
file file.bak
% cat ./file
bar
Obviously you could then just delete the .bak files.
This works with GNU sed, but not on OS X:
sed -i -e 's/foo/bar/' target.file
sed -i'' -e 's/foo/bar/' target.file
This works on OS X, but not with GNU sed:
sed -i '' -e 's/foo/bar/' target.file
On OS X you
can't use sed -i -e since the extension of the backup file would be set to -e
can't use sed -i'' -e for the same reasons—it needs a space between -i and ''.
When on OSX, I always install GNU sed version via Homebrew, to avoid problems in scripts, because most scripts were written for GNU sed versions.
brew install gnu-sed --with-default-names
Then your BSD sed will be replaced by GNU sed.
Alternatively, you can install without default-names, but then:
Change your PATH as instructed after installing gnu-sed
Do check in your scripts to chose between gsed or sed depending on your system
As Noufal Ibrahim asks, why can't you use Perl? Any Mac will have Perl, and there are very few Linux or BSD distributions that don't include some version of Perl in the base system. One of the only environments that might actually lack Perl would be BusyBox (which works like GNU/Linux for -i, except that no backup extension can be specified).
As ismail recommends,
Since perl is available everywhere I just do perl -pi -e s,foo,bar,g target.file
and this seems like a better solution in almost any case than scripts, aliases, or other workarounds to deal with the fundamental incompatibility of sed -i between GNU/Linux and BSD/Mac.
Answer: No.
The originally accepted answer actually doesn't do what is requested (as noted in the comments). (I found this answer when looking for the reason a file-e was appearing "randomly" in my directories.)
There is apparently no way of getting sed -i to work consistently on both MacOS and Linuces.
My recommendation, for what it is worth, is not to update-in-place with sed (which has complex failure modes), but to generate new files and rename them afterwards. In other words: avoid -i.
There is no way to have it working.
One way is to use a temporary file like:
TMP_FILE=`mktemp /tmp/config.XXXXXXXXXX`
sed -e "s/abc/def/" some/file > $TMP_FILE
mv $TMP_FILE some/file
This works on both
Here's another version that works on Linux and macOS without using eval and without having to delete backup files. It uses Bash arrays for storing the sed parameters, which is cleaner than using eval:
# Default case for Linux sed, just use "-i"
sedi=(-i)
case "$(uname)" in
# For macOS, use two parameters
Darwin*) sedi=(-i "")
esac
# Expand the parameters in the actual call to "sed"
sed "${sedi[#]}" -e 's/foo/bar/' target.file
This does not create a backup file, neither a file with appended quotes.
The -i option is not part of POSIX Sed. A more portable method would be
to use Vim in Ex mode:
ex -sc '%s/alfa/bravo/|x' file
% select all lines
s replace
x save and close
Steve Powell's answer is quite correct, consulting the MAN page for sed on OSX and Linux (Ubuntu 12.04) highlights the in-compatibility within 'in-place' sed usage across the two operating systems.
JFYI, there should be no space between the -i and any quotes (which denote an empty file extension) using the Linux version of sed, thus
sed Linux Man Page
#Linux
sed -i""
and
sed OSX Man page
#OSX (notice the space after the '-i' argument)
sed -i ""
I got round this in a script by using an alias'd command and the OS-name output of 'uname' within a bash 'if'. Trying to store OS-dependant command strings in variables was hit and miss when interpreting the quotes. The use of 'shopt -s expand_aliases' is necessary in order to expand/use the aliases defined within your script. shopt's usage is dealt with here.
Portable script for both GNU systems and OSX:
if [[ $(uname) == "Darwin" ]]; then
SP=" " # Needed for portability with sed
fi
sed -i${SP}'' -e "s/foo/bar/g" -e "s/ping/pong/g" foobar.txt
I ran into this problem. The only quick solution was to replace the sed in mac to the gnu version:
brew install gnu-sed
If you need to do sed in-place in a bash script, and you do NOT want the in-place to result with .bkp files, and you have a way to detect the os (say, using ostype.sh), -- then the following hack with the bash shell built-in eval should work:
OSTYPE="$(bash ostype.sh)"
cat > myfile.txt <<"EOF"
1111
2222
EOF
if [ "$OSTYPE" == "osx" ]; then
ISED='-i ""'
else # $OSTYPE == linux64
ISED='-i""'
fi
eval sed $ISED 's/2222/bbbb/g' myfile.txt
ls
# GNU and OSX: still only myfile.txt there
cat myfile.txt
# GNU and OSX: both print:
# 1111
# bbbb
# NOTE:
# if you just use `sed $ISED 's/2222/bbbb/g' myfile.txt` without `eval`,
# then you will get a backup file with quotations in the file name,
# - that is, `myfile.txt""`
The problem is that sed is a stream editor, therefore in-place editing is a non-POSIX extension and everybody may implement it differently. That means for in-place editing you should use ed for best portability. E.g.
ed -s foobar.txt <<<$',s/foo/bar/g\nw'
Also see https://wiki.bash-hackers.org/howto/edit-ed.
You can use sponge. Sponge is an old unix program, found in moreutils package (both in ubuntu and probably debian, and in homebrew in mac).
It will buffer all the content from the pipe, wait until the pipe is close (probably meaning that the input file is already close) and then overwrite:
From the man page:
Synopsis
sed '...' file | grep '...' | sponge file
The following works for me on Linux and OS X:
sed -i' ' <expr> <file>
e.g. for a file f containing aaabbaaba
sed -i' ' 's/b/c/g' f
yields aaaccaaca on both Linux and Mac. Note there is a quoted string containing a space, with no space between the -i and the string. Single or double quotes both work.
On Linux I am using bash version 4.3.11 under Ubuntu 14.04.4 and on the Mac version 3.2.57 under OS X 10.11.4 El Capitan (Darwin 15.4.0).
I'm trying to create an alias for getting memory on my machine, currently I have alias mem="lshw | grep size | awk -F: '{print $2}'", and when I run it as a non-super user, I get the following warning message:
WARNING: you should run this program as super-user.
WARNING: output may be incomplete or inaccurate, you should run this program as super-user.
size: 23GiB
I'm not worried about the results being potentially incomplete, in fact when I diff the output when running as root vs a standard user, it's exactly the same. Does anybody know how to get rid of these warnings? I tried piping stderr to /dev/null, but that didn't work. Does anyone else know how to get rid of these warnings?
Can I interest you in
alias mem='free -g | grep Mem | awk '\''{print $2 " GiB"}'\'
free -m will give MiB; you can change the " GiB" part to whatever you want (or remove it).
I don't have lshw installed on my machine, so I can't help you debug your version, unfortunately.
alias mem="lshw 2> /dev/null| grep size | awk -F: '{print $2}'"
Alternatively you can use free or read from /proc/meminfo
cat /proc/meminfo |grep MemTotal
I'm not sure how you piped to dev/null, but this works for me:
lshw 2> /dev/null | grep size | awk -F: '{print $2}'
Ignoring that there are other tools more suited to getting the memory, if there is something you need and lshw is your only option, you would be better suited to use -json or -xml output and use a tool to parse it like jq or xmllint. The version of lshw on my distro outputs invalid json that can't be parsed, but does have valid xml output.
This would accomplish your goal, although the path may very well be different for you:
lshw -xml 2> /dev/null | xmllint --xpath '/list/node/node/node[#id="memory"]/size/text()' -
Or add a one grep:
... | grep "size:"
I am receiving the following error:
awk: cmd. line:1: (FILENAME=- FNR=798) warning: Invalid multibyte data detected. There may be a mismatch between your data and your locale.
The command I'm running is the following:
cat file.txt | awk 'length($0)<10000' > output-file.txt
The weird part is that if I pipe to other commands like awk '{ sub("\r$", ""); print }', it works just fine without an error.
Anyone see why I would get this error? Or, should I just ignore it?
Make the locale as C to use only ASCII character set with single byte encoding, pass LC_ALL=C to awk's environment:
LC_ALL=C awk 'length($0)<10000' file.txt >output-file.txt
Also you don't need to use cat as awk takes filename(s) as argument(s).
I've found three solutions on my machines:
Change environment variable
This has been answered on the approved one.
Add variable export LC_ALL=C to the environment.
Add parameter (only possible on gawk)
Add -b (binary) parameter. Like in:
cat file.txt | awk -b 'length($0)<10000' > output-file.txt
Use mawk instead of gawk
You can check if you are using gawk or mawk implementation on Linux (the first one is installed with a package of the same name on Ubuntu). For Ubuntu you can run
sudo update-alternatives --config awk
Source answer
rsync --version
gives lots of info but I just want to grab the first line
rsync version 3.1.1
How to do this? I tried to pipe into grep but I can't
There are lots of ways to slice this pie. If you want the whole first line, you can use any of these:
rsync --version | head -n 1
rsync --version | awk NR==1
rsync --version | sed -n 1p
rsync --version | grep '^rsync *version'
If you want just the version number without the rest of the line, that's not much harder, but it depends which part of the line you want. On my Mac, the version line reads rsync version 2.6.9 protocol version 29, and a naïve grab would likely yield the 29 - presumably not what you want. Either of the following will output just the 2.6.9 by itself:
rsync --version | awk 'NR==1 {print $3}'
rsync --version | sed -n '1s/^rsync *version \([0-9.]*\).*$/\1/p'
If you just need to get the first line of output use head command
rsync --version | head -1
I just recently updated my Ubuntu VM to 14.04 and before this I could run the grep command
grep -nr "piece of text"
or any other grep command for that matter and it would finish in 2-10 seconds in the directory I was working on. Now for some reason (no idea if this was caused by the update) running any grep command in the same dir I was working in just seems to hang there (idk if it does complete because I don't want to wait over a min or so for a search to happen) instead of showing my results. Any idea of what's happening or what I can do and try to fix it?
You aren't specifying the files to grep, so it's trying to read STDIN. I think you wanted
grep -nr "piece of text" *
Note the asterisk is globbed to all files in the current path.
which grep are you using ? -r option default to '.' directory in gnu case
which grep :
/bin/grep --version
/bin/grep (GNU grep) 2.18)
...
this is gnu grep :
echo "truc" >/tmp/truc
cd /tmp; grep -nr "truc";
using gnu grep -> it search recursively files in . of current directory ( here in /tmp ) and displays them containing "truc"
if /bin/grep --version is busybox :
BusyBox v1.22.1 (Debian 1:1.22.0-8) multi-call binary.
Usage:
grep [-HhnlLoqvsriwFEz] [-m N] [-A/B/C N] PATTERN/-e PATTERN.../-f FILE [FILE]...
using busybox : cd /tmp; busybox grep -nr "truc";
wait indefinitely that you type content of file on STDIN ( stop with Ctrl+C ).
it reads STDIN as a file ( not even caring that -n indicates a directory recursion ).
As others answered, and as POSIX grep or GNU grep documentation tells, you need to specify some file to the standard grep command. (But GNU grep defaults to current directory . if -r is given without a directory name; however if -r is not given at all, grep defaults to read - i.e. the stdin).
BTW, I would also recommend the ack utility, packaged as the ack-grep package on Ubuntu or Debian. It does not need any files (default is the source code files under current directory) and it avoids useless non-source files (version control or object files...):
ack "piece of text"
Also, under emacs, you can use M-x grep