Below given sed command is working fine on online BASH & KSH shell, but getting an error "Illegal operation --r" while trying to run it on linux server.
I'm trying to make a regex to parse MFBBMYKLAXXX from first line.
echo "{1:F01MFBBMYKLAXXX2474811384}{2:O3001434181108BKKBTHBKBXXX12203020241811081534N}{3:{108:241C182AFFD4403C}}{4:
:15A:
:20:10168957
:22A:NEWT
:94A:BILA
:22C:BKKBBK8308MFBBKL
:82A:BKKBTHBK
:87A:MFBBMYKL
:15B:
:30T:20181108
:30V:20181109
:36:32,8308
:32B:THB2500000,
:53A:/610165
BKKBTHBK
:57A:BKKBTHBK
:33B:USD76148,01
:53A:CHASUS33
:57A:/04058664
BKTRUS33
:58A:MFBBMYKL
:15C:
:24D:ELEC/REUTERS D-3000
-}{5:{CHK:4117CD0206B7}}{S:{COP:S}}
" | sed -rn 's/.*\{1:F01([A-Z]{12}).*/\1/p'
The use of sed -r (or in some dialects sed -E) is nonstandard and optional.
It selects a regex dialect called extended regular expressions, which allows you to express some things more succinctly.
POSIX basic regular expressions support pretty much the same facilities, but with an oddball syntax where you have to backslash some characters to obtain their special meaning (which in other words does exactly the opposite of what backslash escaping originally did).
So if you have an extended regular expression like a+(b{2})c then if your sed does not support either -r or -E, try a\+\(b\{2\}\}c without any special option, and hope that your sed is at least roughly on par with what POSIX specifies. (If you're serious about retrocomputing, this is unlikely, though.)
The original regular expression implementation by Ken Thompson only supported the regex metacharacters [...] and . and *, and for a long time, that's all sed supported, too.
Of course, you could always install a more modern sed. I know SunOS used to have some goodies in their xpg4 directory but I have no idea if this was is the case in Solaris; if so, maybe you just need to add /usr/xpg4/bin to your PATH. (According to this it was true at one point in time at least.)
Related
I noticed something a bit odd while fooling around with sed. If you try to remove multiple line intervals (by number) from a file, but any interval specified later in the list is fully contained within an interval earlier in the list, then an additional single line is removed after the specified (larger) interval.
seq 10 > foo.txt
sed '2,7d;3,6d' foo.txt
1
9
10
This behaviour was behind an annoying bug for me, since in my script I generated the interval endpoints on the fly, and in some cases the intervals produced were redundant. I can clean this up, but I can't think of a good reason why sed would behave this way on purpose.
Since this question was highlighted as needing an answer in the Stack Overflow Weekly Newsletter email for 2015-02-24, I'm converting the comments above (which provide the answer) into a formal answer. Unattributed comments here were made by me in essentially equivalent form.
Thank you for a concise, complete question. The result is interesting. I can reproduce it with your script. Intriguingly, sed '3,6d;2,7d' foo.txt (with the delete operations in the reverse order) produces the expected answer with 8 included in the output. That makes it look like it might be a reportable bug in (GNU) sed, especially as BSD sed (on Mac OS X 10.10.2 Yosemite) works correctly with the operations in either order. I tested using 'sed (GNU sed) 4.2.2' from an Ubuntu 14.04 derivative.
More data points for you/them. Both of these include 8 in the output:
sed -e '/2/,/7/d' -e '/3/,/6/d' foo.txt
sed -e '2,7d' -e '/3/,/6/d' foo.txt
By contrast, this does not:
sed -e '/2/,/7/d' -e '3,6d' foo.txt
The latter surprised me (even accepting the basic bug).
Beats me. I thought given some of sed's arcane constructs that you might be missing the batman symbol or something from the middle of your command but sed -e '2,7d' -e '3,6d' foo.txt behaves the same way and swapping the order produces the expected results (GNU sed 4.2.2 on Cygwin). /bin/sed on Solaris always produces the expected result and interestingly so does GNU sed 3.02. Ed Morton
More data: it only seems to happen with sed 4.2.2 if the 2nd range is a subset of the first: sed '2,5d;2,5d' shows the bug, sed '2,5d;1,5d' and sed '2,5d;2,6d' do not. glenn jackman
The GNU sed home page says "Please send bug reports to bug-sed at gnu.org" (except it has an # in place of ' at '). You've got a good reproduction; be explicit about the output you expect vs the output you get (they'll get the point, but it's best to make sure they can't misunderstand). Point out that the reverse ordering of the commands works as expected, and give the various other commands as examples of working or not working. (You could even give this Q&A URL as a cross-reference, but make sure that the bug report is self-contained so that it can be understood even if no-one follows the URL.)
You can also point to BSD sed (and the Solaris version, and the older GNU 3.02 sed) as behaving as expected. With the old version GNU sed working, it means this is arguably a regression. […After a little experimentation…] The breakage occurred in the 4.1 release; the 4.0.9 release is OK. (I also checked 4.1.5 and 4.2.1; both are broken.) That will help the maintainers if they want to find the trouble by looking at what changed.
The OP noted:
Thanks everyone for comments and additional tests. I'll submit a bug report to GNU sed and post their response. santayana
I am new to Linux.
I was debugging some code. I encountered the following command:
PROGRAM_ID=$(echo $PROGRAM_ID|sed 's/-/,/g')
Can anybody explain what the g represents here?
I understand hyphen is being replaced with comma.
The /g flag means, perform the substitution globally on a line. Without that flag, only the first hyphen on every line would get substituted.
A better way with Bash would be
PROGRAM_ID=${PROGRAM_ID//-/,}
but if you have to be portable to Bourne shell in general, this replacement facility is not available.
(In which case you should take care to keep "$PROGRAM_ID" in double quotes in the echo.)
Its easy to see how g (global) works with these two example:
echo "test-one-two-three" | sed 's/-/,/g'
test,one,two,three
echo "test-one-two-three" | sed 's/-/,/'
test,one-two-three
Without the g it only replace the first hit.
I am trying to understand how
sed 's/\^\[/\o33/g;s/\[1G\[/\[27G\[/' /var/log/boot
worked and what the pieces mean. The man page I read just confused me more and I tried the info sai Id but had no idea how to work it! I'm pretty new to Linux. Debian is my first distro but seemed like a rather logical place to start as it is a root of many others and has been around a while so probably is doing stuff well and fairly standardized. I am running Wheezy 64 bit as fyi if needed.
The sed command is a stream editor, reading its file (or STDIN) for input, applying commands to the input, and presenting the results (if any) to the output (STDOUT).
The general syntax for sed is
sed [OPTIONS] COMMAND FILE
In the shell command you gave:
sed 's/\^\[/\o33/g;s/\[1G\[/\[27G\[/' /var/log/boot
the sed command is s/\^\[/\o33/g;s/\[1G\[/\[27G\[/' and /var/log/boot is the file.
The given sed command is actually two separate commands:
s/\^\[/\o33/g
s/\[1G\[/\[27G\[/
The intent of #1, the s (substitute) command, is to replace all occurrences of '^[' with an octal value of 033 (the ESC character). However, there is a mistake in this sed command. The proper bash syntax for an escaped octal code is \nnn, so the proper way for this sed command to have been written is:
s/\^\[/\033/g
Notice the trailing g after the replacement string? It means to perform a global replacement; without it, only the first occurrence would be changed.
The purpose of #2 is to replace all occurrences of the string \[1G\[ with \[27G\[. However, this command also has a mistake: a trailing g is needed to cause a global replacement. So, this second command needs to be written like this:
s/\[1G\[/\[27G\[/g
Finally, putting all this together, the two sed commands are applied across the contents of the /var/log/boot file, where the output has had all occurrences of ^[ converted into \033, and the strings \[1G\[ have been converted to \[27G\[.
In my vim, I can use :%!sed "s/^/ /", got the wrong output when I use :%!sed 's/^/ /' .
sed: -e expression #1, char 0: no previous regular expression
Is there differences between single quote and double quote in vim command mode?
In my sed, single quote is the same as double quote.
$ echo "wha012" | sed 's/w/haha/'
hahaha012
$ echo "wha012" | sed "s/w/haha/"
hahaha012
my system is xp+vim 7.3 for windows.
In my system:
[1] "c://cygwin/bin/ash.exe"
[2] "c://cygwin/bin/bash.exe"
[3] "c://cygwin/bin/dash.exe"
[4] "c://cygwin/bin/sh.exe"
if i set set shell=\"c:\cygwin\bin\sh.exe"\ -f in _vimrc,i get the new wrong messages:
sed command can not found.
Funny, when I try :%!sed "/^/ /" I get the same error message as when I use single quotes:
sed: 1: "/^/ /": invalid command code /
(This line replaces the content of my file.) I expect to get an error message there because, as #Birei pointed out, you left out the sed s command. This works as expected, with either single or double quotes:
:%!sed "s/^/ /"
#Birei is also right that you can use vim to do things like this, but I assume you have simplified the example from what you were really trying to do.
To answer the original question, Vim uses single quotes for literal strings. The only special character in a literal string is ' itself. Strings delimited with double quotes use \ to denote special character, such as `"\<Esc>".
:echo 'a''b' == "a'b"
:help expr-string
:help literal-string
my system is xp+vim 7.3 for windows
By default Vim uses cmd.exe to run :! commands on Windows, which behaves differently with regard to quoting from the POSIX shell that your s/w/haha/ examples suggest you've been testing with. Try something like
:set shell=\"C:\path\to\sh.exe\"\ -f
to tell it to use your POSIX shell instead. Or if you're using cygwin then try the cygwin version of vim instead of the Windows native one.
The difference is in the sed command, that lets interpolate variables when you execute it directly from the shell, like:
sed "s/$pattern/$replacement/"
but your problem is that you have to use a substitution command that begins with letter s, like:
:%!sed "s/^/ /"
Also you can have same behaviour inside vim without an external command, like:
:%s/^/ /
I am using the greplace plugin for vim and am not sure how to escape brackets in a search.
I want to search for cookies[:parent] and have tried:
:Gsearch cookies[:parent] # returns nothing
:Gsearch cookies\[:parent\] # returns nothing
How should I be doing this?
Thanks
Try
Gsearch cookies\\\[:parent\\\]
or
Gsearch 'cookies\[:parent\]'
. If I understood correctly, shell invoked by :grep! invoked by :Gsearch gets string grep -n cookies\[:parent\] /dev/null (assuming grepprg option has default value) and thus your escapes are interpreted by shell that thinks they are for escaping [ in order to prevent glob expansion. But after globbing done by shell grep takes argument as a pattern, so you need to escape it for grep also and it is why I have three backslashes here: two are to make grep get a backslash and third to prevent glob expansion.
:Gsearch cookies\\\[:parent] works for me.
Remember that :Gsearch requires a file mask in addition to the pattern, so in reality, you'd want to type something like :Gsearch \\\[:parent] *.php or whatever, to specify which files you want to have searched.
:Gsearch cookies\[:parent]
[ is the start of a character class, so needs to be escaped. The ] isn't particularly special so doesn't need to be escaped.