Logstash Multiline filter pattern - logstash

Question

Can you please help me with multiline filters, below is the exception that I need to capture:
<<ERROR>> [Jan 17 19:37:43] [[ACTIVE] ExecuteThread: '53' for queue: 'weblogic.kernel.Default (self-tuning)'] [RPL] [ABPBatchUser]
amdocs.rpm.utils.exceptions.RPMResourceException: (RPL1-000115) The activity failed in Amdocs Balance Manager with the <10, API RECEIVE RESPONSE :Read timed out> error code.
at amdocs.rpm.utils.balance.TCPBalanceManagerServer.connErrorHandle(TCPBalanceManagerServer.java:312)
at amdocs.rpm.utils.balance.TCPBalanceManagerServer.invokeOlcActivity(TCPBalanceManagerServer.java:292)
at amdocs.rpm.utils.balance.RPLOlcImpl.invokeMessage(RPLOlcImpl.java:126)
at amdocs.rpm.flowcontrol.RPL9AddOfferFC.l9AddOffer(RPL9AddOfferFC.java:114)
at amdocs.rpm.local.implementation.RPL9AddOfferProxy.l9AddOffer(RPL9AddOfferProxy.java:291)
at amdocs.rpm.sessions.implementation.RPL1RechargeServicesBean.l9AddOffer(RPL1RechargeServicesBean.java:4784)
at amdocs.rpm.sessions.implementation.RPL1RechargeServicesBean_uqrjz6_EOImpl.__WL_invoke(Unknown Source)
at weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invokeInternal(SessionRemoteMethodInvoker.java:54)
at weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invoke(SessionRemoteMethodInvoker.java:21)
at amdocs.rpm.sessions.implementation.RPL1RechargeServicesBean_uqrjz6_EOImpl.l9AddOffer(Unknown Source)
at amdocs.rpm.sessions.implementation.RPL1RechargeServicesBean_uqrjz6_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:252)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:553)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:313)
at amdocs.rpm.sessions.implementation.RPL1RechargeServicesBean_uqrjz6_EOImpl_12212_WLStub.l9AddOffer(Unknown Source)
at amdocs.csm3g.flowcontrol.CM9SubscriberServicesAddOfferFC.rplAddOffer(CM9SubscriberServicesAddOfferFC.java:1100)
at amdocs.csm3g.flowcontrol.CM9SubscriberServicesAddOfferFC.l9AddOffer(CM9SubscriberServicesAddOfferFC.java:605)
at amdocs.csm3g.local.implementation.CM9SubscriberServicesAddOfferProxy.l9AddOffer(CM9SubscriberServicesAddOfferProxy.java:519)
at amdocs.csm3g.sessions.implementation.SubscriberServicesBean.l9AddOffer(SubscriberServicesBean.java:18499)
at amdocs.csm3g.sessions.implementation.SubscriberServicesBean_gl6i2n_EOImpl.__WL_invoke(Unknown Source)
at weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invokeInternal(SessionRemoteMethodInvoker.java:54)
at weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invoke(SessionRemoteMethodInvoker.java:21)
at amdocs.csm3g.sessions.implementation.SubscriberServicesBean_gl6i2n_EOImpl.l9AddOffer(Unknown Source)
at amdocs.csm3g.sessions.implementation.SubscriberServicesBean_gl6i2n_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:645)
at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:246)
at weblogic.rmi.internal.BasicServerRef$2.run(BasicServerRef.java:534)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:368)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:163)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:531)
at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:138)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:348)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:333)
at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:54)
at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:640)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:406)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:346)
<<ERROR>> [Jan 17 19:37:43] [[ACTIVE] ExecuteThread: '53' for queue: 'weblogic.kernel.Default (self-tuning)'] [RPL] <RPL1RechargeServicesBean.l9AddOffer> encountered an exception from which recovery-by-retry is NOT possible. This flow will NOT be retried. Last retry count = <0 out of 0>
I want to capture from <<ERROR>> till next immediate <<ERROR>> as one message.
I am using filter pattern below:
{
pattern => "(^d+s<<)|(^.+Exception: .+)|(^s+at .+)|(^s+... d+ more)|(^s*Caused by:.+)"
negate => true
what => "previous"
}
but instead of capturing only the first exception it captures the entire thing.

Answer 1

Try with:
multiline {
pattern => "(^.+Exception: .+)|(^\s+at.+)|(^\s+... \\d+ more)|(^\s*Caused by:.+)"
negate => false
what => "previous"
}
If it does not work, can could you please post a longer example, with expected results, with tags other than ERROR?

Answer 2

I would prefer to parse multiline messages using filebeat, because doing it on Logstash side can make system working slower:
multiline.pattern: '(<<ERROR>>\s*\[[A-Z]{1}[a-z]{2}\s[0-9]{2}\s[0-9]{2}:[0-9]{2}:[0-9]{2}\])|(\[[A-Z]{1}[a-z]{2}\s[0-9]{2}\s[0-9]{2}:[0-9]{2}:[0-9]{2}\])'
multiline.negate: true
multiline.match: after
See https://www.elastic.co/guide/en/beats/filebeat/master/multiline-examples.html.