combining log4j log lines - linux

Question

I am dealing with some hive logs created with log4j on linux. The regex used is
(%d{ISO8601} %-5p [%t]: %c{2} (%F:%M(%L)) - %m%n)
. I realise some lines are broken down into several lines. for example
2017-02-10 10:03:29,933 INFO [main]: parse.ParseDriver (ParseDriver.java:parse(185)) - Parsing command:
create table my_table
(std_id STRING, std_number STRING)
2017-02-10 10:03:31,296 INFO [main]: parse.ParseDriver (ParseDriver.java:parse(209)) - Parse Completed
Is there a command in linux which I can use to combine the broken lines to give an output like this ;
2017-02-10 10:03:29,933 INFO [main]: parse.ParseDriver (ParseDriver.java:parse(185)) - Parsing command: create table my_table (std_id STRING, std_number STRING)
2017-02-10 10:03:31,296 INFO [main]: parse.ParseDriver (ParseDriver.java:parse(209)) - Parse Completed

Answer 1

That should work.
<myfile.log sed -nr \
-e '/^.{4}-.{2}-.{2} .{2}:.{2}:.{2},.{3} [A-Z]+ /{x;1!{s/\n/ /g;p};${g;p};d}' \
-e 'H' \
-e '${g;s/\n/ /g;p}'
Explanation
Let /^.{4}-.{2}-.{2} .{2}:.{2}:.{2},.{3} [A-Z]+ be the pattern that indicates a new log record, e.g. 2017-03-17 03:20:19,372 WARN
1st -e
When the current row indicates a new log record
x: Exchange the contents of the pattern space (which hold the current row) and the hold space.
1!{s/\n/ /g;p}: If this is not the first row of the file, (within the pattern space) replace the new lines with spaces and print it
${g;p} If this was the last row of the file, get it from the hold space and print it
d: Delete the pattern space and start a new cycle (ignoring following commands)
2nd -e
'H': append the pattern space to the hold space (only when not a new log record)
3rd -e
If this was the last row of the file ($) and not a new log record
g: Overwrite the pattern space with the content of the hold space
s/\n/ /g;p: replace the new lines with spaces and print it
Demo
$ cat>myfile.log
2017-02-10 10:03:27,374 INFO [main]: ql.Driver (Driver.java:compile(463)) - Semantic Analysis Completed
2017-02-10 10:03:29,933 INFO [main]: parse.ParseDriver (ParseDriver.java:parse(185)) - Parsing command:
create table my_table
(std_id STRING, std_number STRING)
2017-02-10 10:03:31,296 INFO [main]: parse.ParseDriver (ParseDriver.java:parse(209)) - Parse Completed
2017-03-17 03:04:09,297 INFO [main]: ql.Driver (SessionState.java:printInfo(927)) - OK
2017-03-17 03:20:19,372 WARN [Driver]: client.SparkClientImpl (SparkClientImpl.java:run(451)) - Child process exited with code 1.
2017-03-17 03:03:55,282 ERROR [main]: ql.Driver (SessionState.java:printError(936)) - FAILED: ParseException line 1:14 cannot recognize input near 'valeus' '(' '1' in statement
org.apache.hadoop.hive.ql.parse.ParseException: line 1:14 cannot recognize input near 'valeus' '(' '1' in statement
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204)
$ <myfile.log sed -nr -e '/^.{4}-.{2}-.{2} .{2}:.{2}:.{2},.{3} [A-Z]+ /{x;1!{s/\n/ /g;p};${g;p};d}' -e 'H' -e '${g;s/\n/ /g;p}'
2017-02-10 10:03:27,374 INFO [main]: ql.Driver (Driver.java:compile(463)) - Semantic Analysis Completed
2017-02-10 10:03:29,933 INFO [main]: parse.ParseDriver (ParseDriver.java:parse(185)) - Parsing command: create table my_table (std_id STRING, std_number STRING)
2017-02-10 10:03:31,296 INFO [main]: parse.ParseDriver (ParseDriver.java:parse(209)) - Parse Completed
2017-03-17 03:04:09,297 INFO [main]: ql.Driver (SessionState.java:printInfo(927)) - OK
2017-03-17 03:20:19,372 WARN [Driver]: client.SparkClientImpl (SparkClientImpl.java:run(451)) - Child process exited with code 1.
2017-03-17 03:03:55,282 ERROR [main]: ql.Driver (SessionState.java:printError(936)) - FAILED: ParseException line 1:14 cannot recognize input near 'valeus' '(' '1' in statement org.apache.hadoop.hive.ql.parse.ParseException: line 1:14 cannot recognize input near 'valeus' '(' '1' in statement at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204)