I am attempting to process a file exactly like this question:
read a remote file line by line
The one answer to this question suggests to use RemoteFileTemplate but I am
attempting to use the -stream option as suggested in the last comment to the
answer. Also, this kinda seems like the point of the -stream option; to get a stream.
My implementation successfully obtains the InputStream and kick off a separate thread to
process in a BufferedReader.
This works happily on windows laptop but deployment on a linux machine I
sometimes get a "Write end dead" exception caught when trying to read the
BufferedReader in my processing thread.
Research into this suggests the writer is not closing the stream properly:
Write end dead exception using PipedInputStream
So, either this is a bug in spring-integration or there is something missing
in my configuration. I am hoping it is the latter and could use feedback on
the way I am obtaining the InputStream. If I am getting the InputStream
correctly, then how can I get the writer to close the input stream after writing?
Thanks!
Here is outbound gateway configuration:
<int-ftp:outbound-gateway session-factory="ftpClientFactory"
request-channel="inboundGetStream" command="get" command-options="-stream"
expression="payload" remote-directory="/" reply-channel="stream">
</int-ftp:outbound-gateway>
<int:channel id="stream">
<int:queue/>
</int:channel>
Here is where I obtain the InputStream:
public InputStream openFileStream(final String filename, final String directory) throws Exception {
if (inboundGetStream.send(MessageBuilder.withPayload(directory + "/" + filename).build(), ftpTimeout)) {
return getInputStream();
}
return null;
}
private InputStream getInputStream() {
Message<?> msgs = stream.receive(ftpTimeout);
if (msgs == null) {
return null;
}
InputStream is = (InputStream) msgs.getPayload();
return is;
}
Would be better if you share more StackTrace to investigate.
Plus, I don't see that you close the session as it is recommended by the solution:
When consuming remote files as streams, the user is responsible for closing the Session after the stream is consumed. For convenience, the Session is provided in the file_remoteSession header.
<int:service-activator input-channel="markers"
expression="payload.mark.toString().equals('END') ? headers['file_remoteSession'].close() : null"/>
As a sample from the Reference Manual.
On the other hand it would be better to consume the InputStream in the same thread as it has been obtained. Not shifting such a low level, session tied resource to the Queue.
Also check, please, this bug: https://bugs.eclipse.org/bugs/show_bug.cgi?id=359184 . Maybe you really should upgrade to something more fresh: http://search.maven.org/#search|ga|1|g%3A%22com.jcraft%22
Related
I'm trying to use a "ServerSocket" to receive HTTP messages from a client which is POSTing them as chunked. I just want to capture the text content being posted (plain text). Any input on how to do this better would be welcomed.
Here is my existing and very not-elegant solution. When dealing with ServerSocket, one has to handle the headers and chunk barriers manually, and this is what I came up with. I looked at filterLine() method on the reader, maybe that's part of a solution, i'm not sure. Don't know how to elegantly/reliably identify the chunked barriers.
socket.withStreams { input, output ->
BufferedReader reader = new BufferedReader(new InputStreamReader(input))
while (currentLineCount < processor.newLineCount) {
line = reader.readLine()
if (line && line.size() > 3) {
processor.processFormats(line)
}
currentLineCount++
}
}
Caveats:
I have been trying to process line by line to minimize memory impact, rather than buffering the whole collection. I'd like to keep it that way.
These 4 Jetty libraries are available on the classpath, so I could leverage them, but can't add other libraries.
compile 'org.eclipse.jetty:jetty-server:8.1.2.v20120308'
compile 'org.eclipse.jetty:jetty-continuation:8.1.2.v20120308'
compile 'org.eclipse.jetty:jetty-io:8.1.2.v20120308'
compile 'org.eclipse.jetty:jetty-util:8.1.2.v20120308'
I would re-make the whole HTTP server and Handler in Jetty, but I've never used Jetty and can't find any good examples that fit my situation. Any suggestions with code samples for that would be great!
In
AbstractByteArraySerializer why we need this method?
protected void checkClosure(int bite) throws IOException {
if (bite < 0) {
logger.debug("Socket closed during message assembly");
throw new IOException("Socket closed during message assembly");
}
}
When asking questions like this, it's generally best to explain why you are asking it - like are you experiencing some problem that you are trying to understand.
TCP is a streaming protocol; we need some way to delimit a message from the stream, for example CRLF; that's what the (de)serializers do - serializers add structure to the stream, deserializers decode the messages from the stream.
We need to detect if the client closes the socket before a complete message is received.
If you wish the client to signal the end of the message by closing the socket, then you can use the ByteArrayRawSerializer.
I am building a system with Spring Integration that processes all lines in a file as records. Because some of the String records are malformed I have multiple paths through the application via a Splitter and Aggregator combination (I'm building the Aggregator as we speak).
Further, some of the records are so malformed that they are effectively errors. However I have a requirement that all records must be processed therefore I must identify and log gross malformation errors separately and finish processing the file. In other words, I can not fail to process the file but instead must only log errors.
Aggregator
I intend to do achieve the goal of processing grossly malformed records by modifying the headers on the incoming message and passing the message on-ward to the Aggregator which can search for the existence of such a parameter. I'll effectively be hand coding in some error handling situations to my processors and aggregator.
My Release Strategy for the Aggregator will be when all messages are processed.
Code Extract
This code comes from a blog entry by Matt Vickery. He constructs an entirely new message (using MessageBuilder and transferring headers) whereas I will just add something to the Message headers. He includes this code in a gateway which subsequently transfers the Message onto the Aggregator.
public Message<AvsResponse> service(Message<AvsRequest> message) {
Assert.notNull(message, MISSING_MANDATORY_ARG);
Assert.notNull(message.getPayload(), MISSING_MANDATORY_ARG);
MessageHeaders requestMessageHeaders = message.getHeaders();
Message<AvsResponse> responseMessage = null;
try {
logger.debug("Entering AVS Gateway");
responseMessage = avsGateway.send(message);
if (responseMessage == null)
responseMessage = buildNewResponse(requestMessageHeaders,
AvsResponseType.NULL_RESULT);
logger.debug("Exited AVS Gateway");
return responseMessage;
}
catch (Exception e) {
return buildNewResponse(responseMessage, requestMessageHeaders,
AvsResponseType.EXCEPTION_RESULT, e);
}
}
Confusion (...at least, that which I know about)
My questions are as follows:
When I have such a release strategy (all messages processed), is that the best way to ensure all messages get through to the Aggregator?
When using an Aggregator it seems like in practical cases, it would be very common to need access to the Message in some previous step, as opposed to just passing and processing simple POJOs. Would that be true or is there something I should be doing to simplify my design so I can avoid Message
I came across a blog entry by Matt Vickery showing how he achieves what seems to be similar with an Aggregator. I'm using his work as a guide.
P.S. Per Artem Bilan's advice, I'm avoiding creating my own messages and letting SI turn them into Messages
There is no difference for Aggregator if payload is valid or not. Its general purpose is to build a List (by default) of payloads to one Message. And it does it via some sequenceDetails from MessageHeaders. It is first.
If you use Splitter, it is responsible to enrich each produced Message with default sequenceDetails. So, if you have this configuration:
<splitter/>
<aggregator/>
And if your inbound payload is List, you end up with List after aggregator as well.
I assume, that your Splitter just produces String payloads from File lines.
Then you pass each Message to some service/transformer.
The result of that you may pass to the Aggregator.
But as you say some of payloads are not valid and your processor fails with an Exception.
So, how about just try...catch within that POJO method and return some payload with error indicator, e.g. simple String "Oops!".
As I described before: the result of POJO method will be pushed to payload of the Message by Framework. And what is magic, that sequenceDetails will be there in the MessageHeaders too.
I don't see reason to write some custom ReleaseStrategy for this task, or even any other Aggregator's strategies...
Let me know, what you don't understand.
UPDATE
To add some error-indicator to message headers and don't throw Exception, it really will be simpler to build a new Message from code, not via some error-channel flow:
try {
return [GOOD_RESULT];
}
catch(Exception e) {
return MessageBuilder.withPayload(payload).setHeader("ERROR", e.getMessage()).build();
}
But in this case you should use <service-activator> instead of <transformer>, because the last one doesn't copy headers from inbound Message. And you really need them - setHeader for aggregator.
We have an application where we use an H2 embedded database to store the data. We have a synchronized write method which does DB inserts. Since the H2 DB is a small Java embedded DB, we use "synchronized" on the write method to handle the transaction management in embedded DB rather than in DB.
But during heavy load, we could see that the write thread is getting hung. We are not sure for which resource, this thread is getting hung.
Please look at this snippet of code:
public synchronized int write(IEvent event) {
String methodName = "write";
Connection conn = null;
PreparedStatement updtStmt = null;
Statement stmt = null;
ResultSet rSet = null;
int status = 0;
try {
dbConnect.checkDBSizeExceed();
conn = dbConnect.getConnection();
updtStmt = conn.prepareStatement(insertQuery);
updtStmt.setString(1, (String) event.getAttributeValue());
......
updtStmt.setString(30, (String) event.getAttributeValue());
updtStmt.setBoolean(31, false);
status = updtStmt.executeUpdate();
}catch(SQLException ex){
logger.log(methodName,logger.print(ex),Logger.ERROR);
} catch(Exception ex){
logger.log(methodName,logger.print(ex),Logger.ERROR);
} finally {
try {
if (updtStmt != null)
updtStmt.close();
if (conn != null)
conn.close();
}catch(SQLException ex) {
logger.log(methodName,logger.print(ex),Logger.ERROR);
return status;
}
return status;
}
}
We have multiple write methods which can access this DB. From the code we could see that the code is straightforward. But we are not sure where the resource is locked.
Another problem is in the thread dump in the (Websphere) system.out, we could see the thread stacktrace as below.
[6/15/12 3:13:38:225 EDT] 00000032 ThreadMonitor W WSVR0605W: Thread "WebContainer : 3" (00000066) has been active for 632062 milliseconds and may be hung. There is/are 2
thread(s) in total in the server that may be hung.
at com.xxxx.eaws.di.agent.handlers.AuditEmbeddedDBHandler.store(Unknown Source)
at com.xxxx.eaws.di.agent.eventlogger.2LoggerImpl.logEvent(Unknown Source)
at com.xxxx.eecs.eventlogger.EventLoggerAdapter.logAuditEvent(EventLoggerAdapter.java:682)
at com.xxxx.eecs.eventlogger.EventLoggerAdapter.logEvent(EventLoggerAdapter.java:320)
at com.xxxx.eecs.eventlogger.EventLogger.logEventInternal(EventLogger.java:330)
at com.xxxx.eecs.eventlogger.EventLogger.logEvent(EventLogger.java:283)
at com.ibm.wps.auth.impl.ImplicitLoginFilterChainImpl.login(ImplicitLoginFilterChainImpl.java:55)
at com.ibm.wps.auth.impl.AuthenticationFilterChainHandlerImpl.invokeImplicitLoginFilterChain(AuthenticationFilterChainHandlerImpl.java:393)
at com.ibm.wps.auth.impl.InitialAuthenticationHandlerImpl.checkAuthentication(InitialAuthenticationHandlerImpl.java:204)
at com.ibm.wps.state.phases.PhaseManagerImpl.callInitialAuthenticationHandler(PhaseManagerImpl.java:240)
In the above stacktrace, I need to know the reason why I am getting "Unknown Source" in the stack trace. Those jars are available in the class path and we also have the H2.jar in the classpath. We are not sure why, if the thread gets in hung in H2, we are not able to get the thread stacktrace.
If not, I also need to know why the thread stack trace is showing "Unknown Source".
Appreciate your help.
Thanks in advance.
Are you using ejbs? How do you get the connection? Is it injected by the aop server? Do you retrieve it from jndi? You should Not synchronize the method.
Even if it is an embedded db you should rely on the app server facilities.
You need to configure the connection as a datasource, even if your db is in memory. If you want a serialized write on the db you need to configure the connection pool to the serialized ansi isolation level (there are 4 ansi isolation levels). In this way you should obtain the same effect in a managed environment (the app server) without the synchronized, that should be avoided inside an app server.
Unknown Source typically implies that the line numbers are available.
When you compile, the compiler can add in debug information like line numbers. If they are not present in the JAR or .class files, the JVM can't provide you that information.
HTH
It looks like the conn = dbConnect.getConnection(); is waiting for more than 60000ms.
The error thrown by WAS is because resource adapter has one mechanism poll period. It is considered the rate (in milliseconds) at which to poll the enterprise information system (EIS) event store for new inbound events. The poll cycle is established at a fixed rate, meaning that if execution of the poll cycle is delayed for any reason, the next cycle will occur immediately to “catch up”. During the poll period, the polling thread will be sleeping.
Once the time is calculated to 60000 milliseconds, the WebSphere Application Server thread monitor regards this polling thread as hung and throws the exception.
I need to develop a netty based Client, that accepts messages from a Notification Server, and places these messages as Http Requests to another Server in real time.
I have already coded a working application which does this, but I need to add multi-threading to this.
At this point, I am getting confused on how to handle Netty Channels inside a multi-threaded program, as I am all loaded with the conventional approach of sockets and threads.
When I tried to separate the Netty requesting part into a method, It complains about the Channels not being closed.
Can anyone guide me how to handle this?
I would like to use ExecutionHandler and OrderedMemoryAwareThreadPoolExecutor, but I am really new into this.
Help with some examples would be a real favour at this time.
Thanks in advance.
Just add an ExecutionHandler to the ChannelPipeline. This will make sure that every ChannelUpstreamHandler which is added behind the ExecutionHandler will get executed in an extra thread and so does not block the worker-thread.
Have you looked at the example code on the Netty site? The TelnetServer looks to do what you are talking about. The factory creates new handlers whenever it gets a connection. Threads from the Executors will be used whenever there is a new connection. You could use any thread pool and executor there I suspect:
// Configure the server.
ServerBootstrap bootstrap = new ServerBootstrap(
new NioServerSocketChannelFactory(
Executors.newCachedThreadPool(), << change
Executors.newCachedThreadPool())); << change
// Configure the pipeline factory.
bootstrap.setPipelineFactory(new TelnetServerPipelineFactory());
// Bind and start to accept incoming connections.
bootstrap.bind(new InetSocketAddress(8080));
The TelnetServerHandler then handles the individual results.
#Override
public void messageReceived(ChannelHandlerContext ctx, MessageEvent e) {
// Cast to a String first.
// We know it is a String because we put some codec in TelnetPipelineFactory.
String request = (String) e.getMessage();
// Generate and write a response.
String response;
boolean close = false;
if (request.length() == 0) {
response = "Please type something.\r\n";
When the telnet is ready to close the connection it does this:
ChannelFuture future = e.getChannel().write(response);
if (close) {
future.addListener(ChannelFutureListener.CLOSE);
}