logstash for custom logs - logstash

Question

Im trying write a grok filter for below logs but getting grokfailure. I'm new to logstash, please help me.
Logs:
msg.repository.routed.ABC_MAS:101::20170526-05:03:08: got from remote host at t-rate <0.068> and f-size <68> into tmp dir
msg.repository.routed.ABC_MAS:101::20170526-05:03:07: got from remote host at t-rate <0.068> and f-size <68> into tmp dir
msg.repository.routed.BCD_MAS:101::20170526-00:04:34: sftp connected to 1.2.2.1(msg), timeOut:1000
msg.repository.routed.ABC_MAS:101::20170526-00:04:37: sftp connected to 1.2.2.1(msg), timeOut:1000
Grok pattern which I used:
filter {
grok {
match => { "message" => '(?: %{GREEDYDATA:pullpathinfo}):%{NUMBER:thread}::%{NUMBER:date}-%{NUMBER:HOUR}:%{NUMBER:MINUTE}:%{NUMBER:SECOND}: (?: sftp connected to %{IPORHOST:remoteip} %{GREEDYDATA:msg})' }
match => { "message" => '(?: %{GREEDYDATA:pullpathinfo}):%{NUMBER:thread}::%{NUMBER:date}-%{NUMBER:HOUR}:%{NUMBER:MINUTE}:%{NUMBER:SECOND}: (?: got \<%{GREEDYDATA:filename}> %{GREEDYDATA:rate_size})' }
}
}

Answer 1

To develop grok patterns I suggest you to use Grok Debugger. It allows you to build up the grok patterns incrementally.
For the follow log (on of the log lines in your question):
msg.repository.routed.ABC_MAS:101::20170526-00:04:37: sftp connected to 1.2.2.1(msg), timeOut:1000
the following grok pattern will work:
%{USERNAME:pullpathinfo}:%{NUMBER:thread}::%{NUMBER:date}-%{TIME:time}: sftp connected to %{IPORHOST:remoteip}%{GREEDYDATA:msg}
The follow changes are relevant:
The grok pattern has to be exact about every character in the pattern. This is also true for every space between the grok patterns (%{WORD} %{WORD} is not the same as %{WORD}%{WORD}. In your pattern there was a space too much between %{IPORHOST:remoteip} and %{GREEDYDATA:msg}.
%{USERNAME} instead of %{GREEDYDATA} (GREEDYDATA should only be used for remaining parts in a log line. Even if the pattern USERNAME has a name, which does not fit, the pattern behind it looks like a good fit, because it includes [a-zA-Z0-9._-]+ (but not the colon :)
%{TIME} instead of `%{NUMBER:HOUR}:%{NUMBER:MINUTE}:%{NUMBER:SECOND}
I hope this helps.