Fastest way to extract pattern - linux

Question

What is the fastest way to extract a substring of interest from input such as the following?
MsgTrace(65/26)noop:user=xxx=INBOX:cmd=534
ImapFetchComplete(56/39)user=xxxxxxxxxx
Desired output (i.e., the :-terminated string following the string MsgTrace(65/26) in this example):
noop
I tried the following, but without success:
egrep -i "[a-zA-Z]+\(.*\)[a-z]+:"

Answer 1

grep by default returns the entire line when a match is found on a given input line.
While option -o restricts the output to only that part of the line that the regex matched, that is still not enough in this case, because you want a substring of that match.
However, since you're on Linux, you can use GNU grep's -P option (for support of PCREs, Perl-compatible regular expression), which allows extracting a submatch by way of features such as \K (drop everything matched so far) and (?=...) (a look-ahead assertion that does not contribute to the match):
$ grep -Po "[a-zA-Z]\(.*\)\K[a-z]+(?=:)" <<'EOF'
MsgTrace(65/26)noop:user=xxx=INBOX:cmd=534
ImapFetchComplete(56/39)user=xxxxxxxxxx
EOF
noop # output
Optional background information:
Ed Morton points out (in a since-deleted comment) that GNU grep's man page still calls the -P option "highly experimental" that may "warn of unimplemented features", but the option has been around for years, and in practice I have yet to see a warning or a performance problem - YMMV.
In the case at hand, the above command even outperforms sed and awk solutions - see NeronLeVelu's helpful performance comparison.
The interesting article Ed points to discusses a potential performance problem that can surface with regex engines such as used by grep -P (via the PCRE library), Perl itself, and many other widely used (and mature) regex engines, such as in Python, Ruby, and PHP:
In short: the recursive backtracking algorithm employed by these engines can result in severe performance degradation with "pathological" regexes that string together long sequences of subexpressions with variable-length quantifiers, such as (a longer version of) a?a?a?a?aaaa to match aaaa.
The article argues that backtracking is only truly required when a regex contains backreferences, and that a different, much faster algorithm should be employed in their absence.

Answer 2

You could try this:
$ sed -n 's/[[:alpha:]]*([^)]*)\([[:lower:]]*\):.*/\1/p' file
noop
It's portable to all POSIX seds and doesn't employ PCREs, just BREs, so the regexp matching part at least should be fast.

Answer 3

Little quick and dirty test on a 2469120 lines text of such a sample entry give grep -PO as winner
time sed -n -e 's/^MsgTrace[^)]\{4,\})//;t M' -e 'b' -e ':M' -e 's/:.*//p' YourFile >/dev/null
real 0m7.61s
user 0m7:10s
sys 0m0.13s
time awk -F ':' '/^MsgTrace/{ sub( /.*)/, "", $1); print $1}' YourFile >/dev/null
real 0m17.43s
user 0m16.19s
sys 0m0.17s
time grep -Po "[a-zA-Z]\(.*\)\K[a-z]+(?=:)" YourFile >/dev/null
real 0m6.72s
user 0m6.23s
sys 0m0.11s
time sed -n 's/[[:alpha:]]*([^)]*)\([[:lower:]]*\):.*/\1/p' YourFile >/dev/null
real 0m17.43s
user 0m16.29s
sys 0m0.12s
time grep -Po '(?<=MsgTrace\(65/26\)).*?(?=:)' YourFile >/dev/null
real 0m16.38s
user 0m15.22s
sys 0m0.15s
for #EdMorton question (i redo the same original sed to have compare value in same context of machine load). The exact string is lot faster, i imagine that sed try several combination before selecting which is the longest one for all criteria where a .*l give lot more possibility than pool is full
time sed -n -e 's/^MsgTrace([^)]\{3,\})//;T' -e 's/:.*//p' YourFile >/dev/null
real 0m7.28s
user 0m6.60s
sys 0m0.13s
time sed -n -e 's/^[[:alpha:]]*([^)]\{3,\})//;T' -e 's/:.*//p' YourFile >/dev/null
real 0m10.44s
user 0m9.67s
sys 0m0.14s
time sed -n -e 's/^[[:alpha:]]*([^)]*)//;T' -e 's/:.*//p' YourFile >/dev/null
real 0m10.54s
user 0m9.75s
sys 0m0.11s