I recently add si to my project and have been very happy with it. I'm having an issue with my logging. I am using logback and all works ok; however, I have sensitive information flowing through the system that cannot be logged "as is". Certain data must be intercepted and masked before logging. If I configure my log level below WARN then I see this sensitive info. What is the best way to filter all messages bound for logging? I am using a inbound gateway with a service activator for an si server. I am using an async outbound gateway for an si client.
It isn't clear how the logging is relevant to the Spring Integration.
Seems for me it's enough to write some custom Appender accodrding to your logging framework.
From other side you can filter messages before log them using SI <filter> and <logging-channel-adapter> after <filter>.
But in this case you can't use logger.debug() (or similar) from your code: you alway should send the message to the channel for logging.
Please, explain further from where and how you want to filter (or mask) logging messages.
Using logback and slf4j I used this to mask the si messages...
public class LoggingFilter extends TurboFilter {
private static final Marker secured = MarkerFactory.getMarker("secured");
#Override
public FilterReply decide(Marker marker, Logger logger, Level level, String format, Object[] params, Throwable t) {
if (format != null && logger.getName().startsWith("org.springframework.integration")) {
boolean isSecured = marker != null && (marker.equals(secured) || marker.contains(secured));
if (isSecured) {
return NEUTRAL;
} else {
if (marker == null) {
marker = secured;
} else {
marker.add(secured);
}
String message = MessageFormatter.arrayFormat(format, params).getMessage();
//TODO: mask message here.
logger.log(marker, logger.getName(), toLocationAwareLoggerInteger(level), message, null, t);
return DENY;
}
} else {
return NEUTRAL;
}
}
}
Related
I have a Service that is Producing and Consuming messages from different Spring Cloud Stream Channels (bound to EventHub/Kafka topics). There are several such Services which are setup similarly.
The configuration looks like below
public interface MessageStreams {
String WORKSPACE = "workspace";
String UPLOADNOTIFICATION = "uploadnotification";
String BLOBNOTIFICATION = "blobnotification";
String INGESTIONSTATUS = "ingestionstatusproducer";
#Input(WORKSPACE)
SubscribableChannel workspaceChannel();
#Output(UPLOADNOTIFICATION)
MessageChannel uploadNotificationChannel();
#Input(BLOBNOTIFICATION)
SubscribableChannel blobNotificationChannel();
#Output(INGESTIONSTATUS)
MessageChannel ingestionStatusChannel();
}
#EnableBinding(MessageStreams.class)
public class EventHubStreamsConfiguration {
}
The Producer/Publisher code looks like below
#Service
#Slf4j
public class IngestionStatusEventPublisher {
private final MessageStreams messageStreams;
public IngestionStatusEventPublisher(MessageStreams messageStreams) {
this.messageStreams = messageStreams;
}
public void sendIngestionStatusEvent() {
log.info("Sending ingestion status event");
System.out.println("Sending ingestion status event");
MessageChannel messageChannel = messageStreams.ingestionStatusChannel();
boolean messageSent = messageChannel.send(MessageBuilder
.withPayload(IngestionStatusMessage.builder()
.correlationId("some-correlation-id")
.status("done")
.source("some-source")
.eventTime(OffsetDateTime.now())
.build())
.setHeader("tenant-id", "some-tenant")
.build());
log.info("Ingestion status event sent successfully {}", messageSent);
}
}
Similarly I have multiple other Publishers which publish to different Event Hubs/Topics. Notice that there is a tenant-id header being set for each published message. This is something specific to my multi-tenant application to track the tenant context. Also notice that I am getting the channel to be published to while sending the message.
My Consumer code looks like below
#Component
#Slf4j
public class IngestionStatusEventHandler {
private AtomicInteger eventCount = new AtomicInteger();
#StreamListener(TestMessageStreams.INGESTIONSTATUS)
public void handleEvent(#Payload IngestionStatusMessage message, #Header(name = "tenant-id") String tenantId) throws Exception {
log.info("New ingestion status event received: {} in Consumer: {}", message, Thread.currentThread().getName());
// set the tenant context as thread local from the header.
}
Again I have several such consumers and also there is a tenant context that is set in each consumer based on the incoming tenant-id header that is sent by the Publisher.
My questions is
How do I get rid of the boiler plate code of setting the tenant-id header in Publisher and setting the tenant context in the Consumer by abstracting it into a library which could be included in all the different Services that I have.
Also, is there a way of dynamically identifying the Channel based on the Type of the Message being published. for ex IngestionStatusMessage.class in the given scenario
To set and tenant-id header in the common code and to avoid its copy/pasting in every microservice you can use a ChannelInterceptor and make it as global one with a #GlobalChannelInterceptor and its patterns option.
See more info in Spring Integration: https://docs.spring.io/spring-integration/docs/5.3.0.BUILD-SNAPSHOT/reference/html/core.html#channel-interceptors
https://docs.spring.io/spring-integration/docs/5.3.0.BUILD-SNAPSHOT/reference/html/overview.html#configuration-enable-integration
You can't make a channel selection by the payload type because the payload type is really determined from the #StreamListener method signature.
You can try to have a general #Router with a Message<?> expectation and then return a particular channel name to route according that request message context.
See https://docs.spring.io/spring-integration/docs/5.3.0.BUILD-SNAPSHOT/reference/html/message-routing.html#messaging-routing-chapter
I am using java dsl to configure sfp outbound flow.
Gateway:
#MessagingGateway
public interface SftpGateway {
#Gateway(requestChannel = "sftp-channel")
void sendFiles(List<Message> messages);
}
Config:
#Bean
public IntegrationFlow sftpFlow(DefaultSftpSessionFactory sftpSessionFactory) {
return IntegrationFlows
.from("sftp-channel")
.split()
.handle(Sftp.outboundAdapter(sftpSessionFactory, FileExistsMode.REPLACE)
.useTemporaryFileName(false)
.remoteDirectory(REMOTE_DIR_TO_CREATE).autoCreateDirectory(true)).get();
}
#Bean
public DefaultSftpSessionFactory sftpSessionFactory() {
...
}
How can i configure flow to make my gateway reply with Messages that were failed?
In other words i want my gateway to be able to return list of messages which were failed, not void.
I marked gateway with
#MessagingGateway(errorChannel = "errorChannel")
and wrote error channel
#Bean
public IntegrationFlow errorFlow() {
return IntegrationFlows.from("errorChannel").handle(new GenericHandler<MessagingException>() {
public Message handle(MessagingException payload, Map headers) {
System.out.println(payload.getFailedMessage().getHeaders());
return payload.getFailedMessage();
}
})
.get();
}
#Bean
public MessageChannel errorChannel() {
return MessageChannels.direct().get();
}
and in case of some errors(i.e. no connection to SFTP) i get only one error (payload of first message in list).
Where should i put Advice to aggregate all messages?
This is not the question to Spring Integration Java DSL.
This is mostly a design and architecture task.
Currently you don't have any choice because you use Sftp.outboundAdapter() which is one-way, therefore without any reply. And your SftpGateway is ready for that behavior with the void return type.
If you have a downstream errorr, you can only throw them or catch and send to some error-channel.
According to your request of:
i want my gateway to be able to return list of messages which were failed, not void.
I'd say it depends. Actually it is just return from your gateway. So, if you return an empty list into gateway that may mean that there is no errors.
Since Java doesn't provide multi-return capabilities we don't have choice unless do something in our stream which builds that single message to return. As we decided list of failed messages.
Since you have there .split(), you should look into .aggregate() to build a single reply.
Aggregator correlates with the Splitter enough easy, via default applySequence = true.
To send to aggregator I'd suggest to take a look into ExpressionEvaluatingRequestHandlerAdvice on the Sftp.outboundAdapter() endpoint (second param of the .handle()). With that you should send both good and bad messages to the same .aggregate() flow. Than you can iterate a result list to clean up it from the good result. The result after that can be send to the SftpGateway using replyChannel header.
I understand that it sounds a bit complicated, but what you want doesn't exist out-of-the-box. Need to think and play yourself to figure out what can be reached.
We have an Azure web role deployed that has been using Application Insights (ver. 1.0.0.4220), however, we're going over our data quota. Is it possible to configure Application Insights ignore a specific URL?
We have a status web service that gets a huge amount of traffic but never throws any errors. If I could exclude this one service URL I could cut my data usage in half.
Out of the box it is not supported. Sampling feature is coming but that would not be configurable by specific url. You can implement your own channel that would have your custom filtering. Basically your channel will get event to be sent, you check if you want to send it or not and then if yes pass to standard AI channel. Here you can read more about custom channels.
There are two things that changed since this blog post has been written:
channel should implement only ITelemetryChannel interface (ISupportConfiguration was removed)
and instead of PersistenceChannel you should use Microsoft.ApplicationInsights.Extensibility.Web.TelemetryChannel
UPDATE: Latest version has filtering support: https://azure.microsoft.com/en-us/blog/request-filtering-in-application-insights-with-telemetry-processor/
My team had a similiar situation where we needed to filter out urls that were successful image requests (we had a lot of these which made us hit the 30k datapoints/min limit).
We ended up using a modified version of the class in Sergey Kanzhelevs blog post to filter these out.
We created a RequestFilterChannel class which is an instance of ServerTelemetryChannel and extended the Send method. In this method we test each telemetry item to be sent to see if it is an image request and if so, we prevent it from being sent.
public class RequestFilterChannel : ITelemetryChannel, ITelemetryModule
{
private ServerTelemetryChannel channel;
public RequestFilterChannel()
{
this.channel = new ServerTelemetryChannel();
}
public void Initialize(TelemetryConfiguration configuration)
{
this.channel.Initialize(configuration);
}
public void Send(ITelemetry item)
{
if (item is RequestTelemetry)
{
var requestTelemetry = (RequestTelemetry) item;
if (requestTelemetry.Success && isImageRequest((RequestTelemetry) item))
{
// do nothing
}
else
{
this.channel.Send(item);
}
}
else
{
this.channel.Send(item);
}
}
public bool? DeveloperMode
{
get { return this.channel.DeveloperMode; }
set { this.channel.DeveloperMode = value; }
}
public string EndpointAddress
{
get { return this.channel.EndpointAddress; }
set { this.channel.EndpointAddress = value; }
}
public void Flush()
{
this.channel.Flush();
}
public void Dispose()
{
this.channel.Dispose();
}
private bool IsImageRequest(RequestTelemetry request)
{
if (request.Url.AbsolutePath.StartsWith("/image.axd"))
{
return true;
}
return false;
}
}
Once the class has been created you need to add it to your ApplicationInsights.config file.
Replace this line:
<TelemetryChannel Type="Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.ServerTelemetryChannel, Microsoft.AI.ServerTelemetryChannel"/>
with a link to your class:
<TelemetryChannel Type="XXX.RequestFilterChannel, XXX" />
Alternatively, you can disable the automated request collection and keep only exception auto-collection, just remove the RequestTrackingModule line from applicationinsights.config.
If you still need to collect some of the requests, not just filter all out, you can then call TrackRequest() (in the object of TelemetryClient class) from your code in the appropriate place after you know that you certainly need to log this request to AI.
Update: Filtering feature has been released some time ago and allows for exclusion of certain telemetry items way easier.
I'm trying to build a component for application level (java/java ee) logging using log4j.where i can create the jar of the component and put it in class path of any application and use it. Below approach i have followed
I override log method like debug , trace, info etc.
single and multiple argument substutution e.g. MessageFormatter.format("Hi {}. My name is {}.", "Alice", "Bob"); will return the string "Hi Alice. My name is Bob.".
say for example for trace message
public boolean isTraceEnabled() { return logger.isTraceEnabled();}
public void trace(String msg, Throwable throwable, Object... args) {
log(isTraceEnabled(),throwable,msg,args);//
}
private void log(boolean isEnabled, Throwable throwable, String msg,Object... args)
{
if(throwable!=null){
String message=MessageFormatter.getFormattedMessage(throwable);//Formated the exception message
msg=msg+message;
throwable=null;
}
if (args == null || args.length == 0) {
logger.log(FQCN,LEVELmsg, throwable);
} else {
if (isEnabled) {
String formattedMsg = MessageFormatter.arrayFormat(msg, args);//single and multiple argument substutution
logger.log(FQCN, UtilConstant. Level.TRACEformattedMsg, throwable);
}
}
}
My aim is to build the component which can cater all the Java EE applications. Is that two approach sufficient or I need to do more on that. Please help.
It seems to me you're reinventing the wheel. Check out the SLF4J, it can do the things that you're aiming to implement, and it shields you from underlying messaging system, which you can change at any time (it works with log4j out of the box, too).
Is there some configuration or available module in Spring Security to limit login attempts (ideally, I'd like to have an increasing wait time between subsequent failed attempts)? If not, which part of the API should be used for this?
From Spring 4.2 upwards annotation based event listeners are available:
#Component
public class AuthenticationEventListener {
#EventListener
public void authenticationFailed(AuthenticationFailureBadCredentialsEvent event) {
String username = (String) event.getAuthentication().getPrincipal();
// update the failed login count for the user
// ...
}
}
Implement an AuthenticationFailureHandler that updates a count/time in the DB. I wouldn't count on using the session because the attacker is not going to be sending cookies anyway.
I recently implemented a similar functionality to monitor login failures using JMX. Please see the code in my answer to question Publish JMX notifications in using Spring without NotificationPublisherAware. An aspect on the authenticate method of authentication provider updates MBean and works with a notification listener (code not shown in that question) to block user and IP, send alert emails and even suspend the login if failures exceed a threshold.
Edit
Similar to my answer to question Spring security 3 : Save informations about authentification in database, I think that capturing an authentication failure event (as opposed to customizing a handler) and storing information in database will also work and it will keep the code decoupled as well.
As suggested by Rob Winch in http://forum.springsource.org/showthread.php?108640-Login-attempts-Spring-security, I just subclassed DaoAuthenticationProvider (which could also have been done using an aspect as Ritesh suggests) to limit the number of failed logins, but you could also assert pre-conditions as well:
public class LimitingDaoAuthenticationProvider extends DaoAuthenticationProvider {
#Autowired
private UserService userService;
#Override
public Authentication authenticate(Authentication authentication)
throws AuthenticationException {
// Could assert pre-conditions here, e.g. rate-limiting
// and throw a custom AuthenticationException if necessary
try {
return super.authenticate(authentication);
} catch (BadCredentialsException e) {
// Will throw a custom exception if too many failed logins have occurred
userService.recordLoginFailure(authentication);
throw e;
}
}
}
In Spring config XML, simply reference this bean:
<beans id="authenticationProvider"
class="mypackage.LimitingDaoAuthenticationProvider"
p:userDetailsService-ref="userDetailsService"
p:passwordEncoder-ref="passwordEncoder"/>
<security:authentication-manager>
<security:authentication-provider ref="authenticationProvider"/>
</security:authentication-manager>
Note that I think that solutions which rely on accessing an AuthenticationException's authentication or extraInformation properties (such as implementing an AuthenticationFailureHandler) should probably not be used because those properties are now deprecated (in Spring Security 3.1 at least).
You could also use a service which implements ApplicationListener<AuthenticationFailureBadCredentialsEvent> to update the record in DB.
See spring application events.
Here is my implementation, hope help.
Create a table to store any invalid login attempts.
If invalid attempts > max allowed, set UserDetail.accountNonLocked to false
Spring Security will handle the "lock process" for you. (refer to AbstractUserDetailsAuthenticationProvider)
Last, extends DaoAuthenticationProvider, and integrate the logic inside.
#Component("authenticationProvider")
public class YourAuthenticationProvider extends DaoAuthenticationProvider {
#Autowired
UserAttemptsDao userAttemptsDao;
#Override
public Authentication authenticate(Authentication authentication)
throws AuthenticationException {
try {
Authentication auth = super.authenticate(authentication);
//if corrent password, reset the user_attempts
userAttemptsDao.resetFailAttempts(authentication.getName());
return auth;
} catch (BadCredentialsException e) {
//invalid login, update user_attempts, set attempts+1
userAttemptsDao.updateFailAttempts(authentication.getName());
throw e;
}
}
}
For full source code and implementation, please refer to this - Spring Security limit login attempts example,
create a table to store the values of failed attempts ex : user_attempts
Write custom event listener
#Component("authenticationEventListner")
public class AuthenticationEventListener
implements AuthenticationEventPublisher
{
#Autowired
UserAttemptsServices userAttemptsService;
#Autowired
UserService userService;
private static final int MAX_ATTEMPTS = 3;
static final Logger logger = LoggerFactory.getLogger(AuthenticationEventListener.class);
#Override
public void publishAuthenticationSuccess(Authentication authentication) {
logger.info("User has been logged in Successfully :" +authentication.getName());
userAttemptsService.resetFailAttempts(authentication.getName());
}
#Override
public void publishAuthenticationFailure(AuthenticationException exception, Authentication authentication) {
logger.info("User Login failed :" +authentication.getName());
String username = authentication.getName().toString();
UserAttempts userAttempt = userAttemptsService.getUserAttempts(username);
User userExists = userService.findBySSO(username);
int attempts = 0;
String error = "";
String lastAttempted = "";
if (userAttempt == null) {
if(userExists !=null ){
userAttemptsService.insertFailAttempts(username); }
} else {
attempts = userAttempt.getAttempts();
lastAttempted = userAttempt.getLastModified();
userAttemptsService.updateFailAttempts(username, attempts);
if (attempts + 1 >= MAX_ATTEMPTS) {
error = "User account is locked! <br>Username : "
+ username+ "<br>Last Attempted on : " + lastAttempted;
throw new LockedException(error);
}
}
throw new BadCredentialsException("Invalid User Name and Password");
}
}
3.Security Configuration
1) #Autowired
#Qualifier("authenticationEventListner")
AuthenticationEventListener authenticationEventListner;
2) #Bean
public AuthenticationEventPublisher authenticationListener() {
return new AuthenticationEventListener();
}
3) #Autowired
public void
configureGlobalSecurity(AuthenticationManagerBuilder auth) throws Exception {
auth.userDetailsService(userDetailsService).passwordEncoder(passwordEncoder());
//configuring custom user details service
auth.authenticationProvider(authenticationProvider);
// configuring login success and failure event listener
auth.authenticationEventPublisher(authenticationEventListner);
}