logs from snoopy in AlienVault/Ossim - alienvault

I try to analyze logs from snoopy.
For example:
Dec 2 07:58:31 local.server snoopy[14165]: [uid:1660 sid:14056 tty:/dev/pts/1 cwd:/home/myuser filename:/usr/bin/ssh]: ssh root#remote.server
I wrote a decoder:
<decoder name="snoopy-logger">
<program_name>^snoopy</program_name>
</decoder>
and:
<group name="snoopy-test">
<rule id="100040" level="0">
<decoded_as>snoopy-logger</decoded_as>
<description>Ignore Snoopy logger events</description>
</rule>
<rule id="100041" level="15">
<if_sid>100040</if_sid>
<match>ssh root#</match>
<description>snoopy root</description>
</rule>
</group>
And when I tested via logtest, I got:
**Phase 1: Completed pre-decoding.
full event: 'Dec 2 07:58:31 local.server snoopy[14165]: [uid:1660 sid:14056 tty:/dev/pts/1 cwd:/home/myuser filename:/usr/bin/ssh]: ssh root#remote.server'
hostname: 'local.server'
program_name: 'snoopy'
log: '[uid:1660 sid:14056 tty:/dev/pts/1 cwd:/home/myuser filename:/usr/bin/ssh]: ssh root#remote.server'
**Phase 2: Completed decoding.
decoder: 'snoopy-logger'
**Phase 3: Completed filtering (rules).
Rule id: '100041'
Level: '15'
Description: 'snoopy root'
**Alert to be generated.
So it works, but in SIEM i got event with src_ip and dst_ip = 0.0.0.0.
What I missed? I need src_ip = local.server and dst_ip = remote.server.
Thanks in advance for any suggestions :)

Looks like my answer is a little bit late, but unfortunately the OSSEC rules are only half of the parsing issue in AlienVault.
Once OSSEC parses the event and it has a high enough level to generate an OSSEC alert, it gets written to /var/ossec/logs/alerts/alerts.log where it is then picked up by ossim-agent which is reading the alerts file. ossim-agent is the sensor process that is responsible for reading the raw text logs and then parsing them using regular expressions defined in a plugin (in this case, the ossec-single-line.cfg plugin in /etc/ossim/agent/plugins/).
You will probably need to add an additional rule to your plugin by creating an ossec-single-line.cfg.local file in /etc/ossim/agent/plugins/ to add rules to the original plugin for OSSEC.
More info on creating rules and plugin files can be found in AlienVault's docs here:
https://www.alienvault.com/doc-repo/usm/security-intelligence/AlienVault-USM-Plugins-Management-Guide.pdf
Check out the Customizing Plugins section starting on page 35.
Happy Tuning!

Related

Nlog / event-properties: how to hint NLog to ignore/skip empty/null properties from the final log entry

Basically the title itself kinda explains what i'm trying to achieve but in greater detail:
Let's say the one has similar to the following XML setup for the layout:
layout="<log level='${level:lowerCase=True}' time='${longdate:universalTime=true}' myCustomProperty1='${event-properties:item=myCustomProperty1}' myCustomProperty2='${event-properties:item=myCustomProperty2}'>${newline}
...."
Now when myCustomProperty1 is set to let's say 'blah1' but myCustomProperty2 is not added to eventInfo.Properties collection the resulting entry looks like following:
<log level='blah' time='blah' myCustomProperty1='blah1' myCustomProperty2=''>
...
The question is - what can be done (preferably in the config file) to exclude the myCustomProperty2 attribute from finally rendered result so the output looks as following:
<log level='blah' time='blah' myCustomProperty1='blah1'>
...
Here is the gotcha - the same logger is used by multiple threads so i can't simply alter target's layout configuration at the runtime since it may negatively affect the rest of the threads
Thank you in advance for your suggestions.
-K
You could try using When :
<variables>
<variable name="var_myCustomProperty1" value="${when:when=length('${event-properties:item=myCustomProperty1}')>0:Inner= myCustomProperty1="${event-properties:item=myCustomProperty1}"}"/>
<variable name="var_myCustomProperty2" value="${when:when=length('${event-properties:item=myCustomProperty2}')>0:Inner= myCustomProperty2="${event-properties:item=myCustomProperty2}"}"/>
</variables>
<targets>
<target name="test" type="Console" layout="<log level='${level:lowerCase=True}' time='${longdate:universalTime=true}'${var_myCustomProperty1}${var_myCustomProperty2} />" />
</targets>
NLog 4.6 will include the XmlLayout, that might make things easier:
https://github.com/NLog/NLog/pull/2670
Alternative you can use the JsonLayout, if xml-output is not a requirement (renderEmptyObject="false")

Spring File Inbound Channel Adapter slows down

I have a spring-integration flow that starts with a file inbound-channel-adapter activated by a transactional poller (tx is handled by atomikos).
The text in the file is processed and the message goes down through the flow until it gets sent to one of the JMS queues (JMS outbound-channel-adapter).
In the middle, there are some database writes within a nested transaction.
The system is meant to run 24/7.
It happens that the single message flow, progressively slows down and when I investigated, I found that the stage that is responsable for the increasing delay is the read from filesystem.
Below, the first portion fo the integration flow:
<logging-channel-adapter id="logger" level="INFO"/>
<transaction-synchronization-factory id="sync-factory">
<after-commit expression="payload.delete()" channel="after-commit"/>
</transaction-synchronization-factory>
<service-activator input-channel="after-commit" output-channel="nullChannel" ref="tx-after-commit-service"/>
<!-- typeb inbound from filesystem -->
<file:inbound-channel-adapter id="typeb-file-inbound-adapter"
auto-startup="${fs.typeb.enabled:true}"
channel="typeb-inbound"
directory="${fs.typeb.directory.in}"
filename-pattern="${fs.typeb.filter.filenamePattern:*}"
prevent-duplicates="${fs.typeb.preventDuplicates:false}" >
<poller id="poller"
fixed-delay="${fs.typeb.polling.millis:1000}"
max-messages-per-poll="${fs.typeb.polling.numMessages:-1}">
<transactional synchronization-factory="sync-factory"/>
</poller>
</file:inbound-channel-adapter>
<channel id="typeb-inbound">
<interceptors>
<wire-tap channel="logger"/>
</interceptors>
</channel>
I read something about issues related to the prevent-duplicates option that stores a list of seen files, but that is not the case because I turned it off.
I don't think that it may be related to the filter (filename-pattern) because the expression I use in my config (*.RCV) is easy to apply and the input folder does not contain a lot of files (less than 100) at the same time.
Still, there is something that gradually makes the read from filesystem slower and slower over time, from a few millis to over 3 seconds within a few days of up-time.
Any hints?
You should remove, or move files after they have been processed; otherwise the whole directory has to be rescanned.
In newer versions, you can use a WatchServiceDirectoryScanner which is more efficient.
But it's still best practice to clean up old files.
Finally I got the solution.
The issue was related to the specific version of Spring I was using (4.3.4) that is affected by a bug I had not discovered yet.
The problem is something about DefaultConversionService and the use of converterCache (look at this for more details https://jira.spring.io/browse/SPR-14929).
Upgrading to a more recent version has resolved.

How can I clear an external ID on a record in NetSuite?

I need to clear/reset the external ID on a record in NetSuite, but nothing I do is working.
Some of the InventoryItem records are incorrectly mapped to records in another system. I have an application that can sync up the two systems, but I need to clear NetSuite's external IDs first.
Responses don't really need to be SOAP-specific. If you know how to do it with some specific NetSuite/SuiteTalk client, that might point me in the right direction.
What I've Tried
First up, I tried using the nullFieldList... but maybe it doesn't work because externalId is an attribute, not an element?
<messages:update>
<messages:record internalId="7777" xsi:type="accounting:InventoryItem">
<core:nullFieldList xsi:type="core:NullField">
<core:name>externalId</core:name>
</core:nullFieldList>
</messages:record>
</messages:update>
The external ID is just a string, so I tried just setting it to blank. Didn't work either.
<messages:update>
<messages:record internalId="7777" xsi:type="accounting:InventoryItem">
<accounting:externalId></accounting:externalId>
</messages:record>
</messages:update>
I even tried setting the external ID to 0, but I get back a "not unique identifier" error
<messages:update>
<messages:record internalId="7777" xsi:type="accounting:InventoryItem">
<accounting:externalId>0</accounting:externalId>
</messages:record>
</messages:update>
Other Info
I'm using NetSuite's SOAP API v.2013_1
When I say "it doesn't work", I mean: after I do the update, I get a success response similar to the following:
<readResponse>
<platformCore:status isSuccess="true" xmlns:platformCore="urn:core_2013_1.platform.webservices.netsuite.com"/>
<record internalId="7777" externalId="42" xsi:type="listAcct:InventoryItem" xmlns:listAcct="urn:accounting_2013_1.lists.webservices.netsuite.com">
<!-- snip -->
</record>
</readResponse>
If you are using scripts in netsuite you can run a scheduled script to clear records in NS by loading each record and setting the externalid to '' using the following simple code:
var rec= nlapiLoadRecord(type,id);
rec.setFieldValue('externalid','');
nlapiSubmitRecord(rec);
This seemed to work for me in my enviornment which was on 2015.2.
Unfortunately my understanding is that once you set an externalid you cannot clear it, you can set it to another value, but not back to null. I have experienced this both using SuiteScript as well as a Boomi process that uses the 2014.1 endpoint. This may have changed in the recent releases, as I have not tried it recently with SuiteScript nor with a newer endpoint.
You can eliminate the externalId on a record once it's been set. Here's an example using the NetSuite gem:
ns_customer = NetSuite::Records::Customer.get external_id: 'ext_id'
ns_customer.external_id = ''
ns_customer.update
Here's the corresponding XML for update:
<env:Body>
<platformMsgs:update>
<platformMsgs:record xsi:type="listRel:Customer" platformMsgs:internalId="199113" platformMsgs:externalId=""/>
</platformMsgs:update>
</env:Body>
I have had to attempt this before as well. I know the pains you describe. I ended up putting a "-" in front of my external ID to unlink it between my systems. You can do this in SOAP or even as a simple one time csv upload. As it was one time, I did csv.

Nlog: send mail when Loglevel >= Loglevel.Error with last x messages

I'd like to do this (from log4net docu) with nlog:
This example shows how to deliver only significant events. A LevelEvaluator is specified with a threshold of WARN. This means that an email will be sent for each WARN or higher level message that is logged. Each email will also contain up to 512 (BufferSize) previous messages of any level to provide context. Messages not sent will be discarded.
Is it possible?
I only found this on codeproject.
But it uses a wrapper target that flushes in behalf of the number of messages, not on the log level.
Thanks
Tobi
There is a QueuedTargetWrapper ( a target that buffers log events and sends them in batches to the wrapped target)
that seems address the requirement. I haven't tried it yet.
There is a related discussion "The Benefits of Trace Level Logging in Production Without the Drawback of Enormous Files"
Simple solution that will write last 200 log-events on error:
<target name="String" xsi:type="AutoFlushWrapper" condition="level >= LogLevel.Error" flushOnConditionOnly="true">
<target xsi:type="BufferingWrapper"
bufferSize="200"
overflowAction="Discard">
<target xsi:type="wrappedTargetType" ...target properties... />
</target>
</target>
See also: https://github.com/nlog/NLog/wiki/BufferingWrapper-target

WSO2 - Using get-property() function in Property/Xquery Mediators

Our current service has 7 operations. when writing an outbound xquery "local entry" in wso2, we're trying to retrieve the name of the current operation being executed (how can this be so difficult?).
After reading what i could find in wso2's documentation. it appears as if we need to set up both a Property and an Xquery mediator. supposedly the property mediator would pull the value doing something like get-property('OperationName') and then this would be referenced and passed thru the Xquery mediator.
The other idea was that we needed to define it as a variable in the "Local Registry entry definitions" and than it would be around at all parts of the sequence.
I've tried for 2 days but haven't quite got it.
Please tell me what I'm missing...
Did you try the following xquery sample[1]? I modified the query mediator to get the operation name as follows.
<variable xmlns:ax21="http://services.samples/xsd" xmlns:m0="http://services.samples" name="code" expression="get-property('OperationName')" type="STRING" />
this worked fine. I could see the getQuote in the response message.
[1] http://wso2.org/project/esb/java/4.0.2/docs/samples/advanced_mediation_samples.html#Sample390

Resources