How to calculate checksum/digest of a transferred file in spring-integration - spring-integration

How/where can I compute md5 digest for a file I need to transfer to a samba location in spring-integration in order to validate it against the digest I receive at the beginning of the flow. I get the file from a rest service and I have to make sure file is safely landing to samba location. The middle flow looks like this: (the digest to be compared against is stored somewhere in the messages)
GenericHandler smbUploader;
HttpRequestExecutingMessageHandler httpDownloader;
from(inbound()) //here I receive a notification with url where to download file + a checksum to be validated against
...
.handle(httpDownloader) //here I get file effectively
.handle(smbUploader) //here I upload the file to samba
...
and httpDownloader is defined like this:
public HttpRequestExecutingMessageHandler httpDownloader(){
HttpRequestExecutingMessageHandler h = new HttpRequestExecutingMessageHandler ("payload.url");
h.setExpectedResponseType(String.class);
h.setHttpMethod(GET);
return h;
}
and smbUploader is defined like this:
public GenericHandler smbUploader (MessageHandler smbMessageHandler){
return new GenericHandler<Message>(){
#Override
public Message handle(Message m, MessageHeaders h){
smbMessageHandler.handleMessage(m);
return m;
}
}
and smbMessageHandler is defined like this:
public MessageHandler smbMessageHandler (SmbRemoteFileTemplate template, FileNameGenerator g){
SmbMessageHandler h = new smbMessageHandler (template, REPLACE);
h.setAutoCreateDirectory(true);
h.setRemoteDirectoryExpression(getExpression("headers['msg'].smbFolder"));
h.setFileNameGenerator(g);
return h;
}
the inbound (starting the flow) is defined like this:
public HttpRequestHandlerEndpointSpec inbound(){
return Http.inboundChannelAdapter ("/notification")
.requestMapping(m->m.methods(POST))
.requestPayloadType(String.class)
.validator(notificationValidator);
}

First of all you should store a digest in the message headers in the beginning of the flow.
Then you need to write a service method to calculate a checksum of the file you got downloaded. And insert a new handle() in between:
.handle(httpDownloader) //here I get file effectively
.handle(smbUploader) //here I upload the file to samba
to call your service method. The input for that method must be a whole Message, so you got access to the downloaded file in the payload and digest in the headers. The result of this method could be just your file to proceed into an SMB handler for uploading.
How to calculate a checksum you can find in this SO thread: Getting a File's MD5 Checksum in Java

Related

Spring SFTP polling multiple directory using Outbound Message Gateway and IntegrationFlow

I want to get all file under a particular remote directory in a periodic manner. I am able to get the file under that directory only once after application startup. Not sure why the Poller is not working. This is registered in a spring boot project and the version is 2.2.1
#InboundChannelAdapter(value = "sftpReportChannel",
poller = #Poller(fixedDelay = "5000"))
public String filesForGET(){
return "/etl/biq/autoscore/output/report-data/";
}
#Bean
public IntegrationFlow sftpGetFlow(SessionFactory<ChannelSftp.LsEntry> csf) {
return IntegrationFlows.from("sftpReportChannel")
.handle(Sftp.outboundGateway(csf,
AbstractRemoteFileOutboundGateway.Command.LS, "payload")
.options(AbstractRemoteFileOutboundGateway.Option.RECURSIVE, AbstractRemoteFileOutboundGateway.Option.NAME_ONLY)
//Persistent file list filter using the server's file timestamp to detect if we've already 'seen' this file.
.filter(new SftpPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "autoscore-meta-data")))
.split()
.log(message -> "file path -> "+message.getPayload())
.handle(Sftp.outboundGateway(csf, AbstractRemoteFileOutboundGateway.Command.GET, "'/etl/biq/autoscore/output/report-data/' + payload")
.options(AbstractRemoteFileOutboundGateway.Option.STREAM))
.handle(new ReportHandler()) //get the payload and create email content and send eamil to recipients
.get();
}
The .filter(new SftpPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "autoscore-meta-data"))) makes it working the way that it doesn't pick up the same file again and again on the subsequent poll activities.
Make sure you add new files in that remote dir at runtime or modify already processed file. The SftpPersistentAcceptOnceFileListFilter logic relies on the mtime property of the LsEntry to determine that the file has been changed therefore it is good for processing again.

Azure Blob Change Feed missing blob append events

I'm trying out Azure Blob Change Feed feature and it behaves strange to me with Append Blobs: append events are missing in the feed.
My scenario is:
Create storage account, enable change feed feature:
Change feed enabled
Create Append Blob if not exists (1) and appending some input into it (2).
private void WriteBlob(string input)
{
MemoryStream stream = new MemoryStream(Encoding.UTF8.GetBytes(input));
try
{
if (client == null)
{
var credential = new ClientSecretCredential("...", "...");
client = new AppendBlobClient(new Uri("..."), credential);
}
client.CreateIfNotExists(); // (1)
client.AppendBlock(stream); // (2)
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
Fetch Change Feed entries in separate console app.
public static List<BlobChangeFeedEvent> GetChanges()
{
var credential = new ClientSecretCredential("...", "...");
BlobChangeFeedClient blobChangeFeedClient = new BlobChangeFeedClient(new Uri("..."), credential);
List<BlobChangeFeedEvent> events = new List<BlobChangeFeedEvent>();
foreach (BlobChangeFeedEvent changeFeedEvent in blobChangeFeedClient.GetChanges())
{
events.Add(changeFeedEvent);
}
return events;
}
The problem is that after a few runs of WriteBlob method I only get single change feed event that corresponds to the blob creation, and subsequent appends are missing in the feed, however inputs are being appended successfully to the blob resource.
The question is why it is working this way? I didn't find anything special about Append Blob blob type regarding Change feed in docs.
Currently, the append event for an append blob is not supported.
As per this doc, only the following event types are supported:
BlobCreated
BlobDeleted
BlobPropertiesUpdated
BlobSnapshotCreated
And in the source code of Azure.Storage.Blobs.ChangeFeed package, there is no append event type.
A feature request of this is submitted, hope it can be added in the future release.

How to mock Transport.send() method in this method using MockitoJUnitRunner?

I want to write the testcase for the given method. But there exist a static Transport.sendEmail method.
How can I mock this method using MockitoJunitRunner.
public void sendEmail(final String message, final String contentType) {
final Session session = Session.getDefaultInstance(PROPERTIES, null);
final Multipart mpart = new MimeMultipart();
final MimeBodyPart body = new MimeBodyPart();
try {
body.setContent(message, contentType);
mpart.addBodyPart(body);
Transport.send(createMessage(session, mpart));
LOGGER.info("Email Notification Sent Successfully");
} catch (MessagingException | UnsupportedEncodingException e) {
LOGGER.error("Was not able to send mail", e);
}
So:
Transport.send(createMessage(session, mpart));
that static call means: you can't "control" it using Mockito. Plain and simple. If that call just "passes" in your unit test environment, well, then you can test it, but not verify that the call really took place. Worse, if that call throws some exception in the unit test setup, then heck, what could you do?
Options:
turn to PowerMock(ito) or JMockit, as they allow you to gain control
recommended: improve your design to be easy-to-test
That last idea comes in various flavors:
For example, you could create a minimal interface EmailSenderService that offers a void send(Message whatever) method. Next: you create one implementation of that interface that simply invokes that static method. Now your code that actually has to send that message ... simply gets passed in an instance of EmailSenderService. And within your unit tests, you can now #Mock that interface, and you gain control over it.
Alternatively, you simply deprecate that static method (maybe the whole class), and you design a new "real" EmailSenderService that doesn't rely on static methods.

gwt AsyncCallback - java

I'm doing an asynchronous call to a servlet. In the servlet I am taking some pictures from DB and sending them back to client as Strings. In the Client..in onSuccess(String pictureAsString) method I need to do something with the images after they are loaded and BEFORE going any further. I know the number of images and currently I'm doing some testing like this:
public void onSuccess(String result) {
numberOfReceivedImages = numberOfReceivedImages-1;
//add image as string to a list of images
if(numberOfReceivedImages == 0){
////call method that draws the images from the list }
}
The order in wich the images are drawn should be the order from the list in which the images are stored after receiving them from server, but this order is not kept...and I suppose this is because I'm not receiving all the images by the time I'm drawing them.
Thank you,
any help is appreciated
Are you sure the images are sent in the right order from the server? How do you store them client-side?
I would suggest that you change the signature of your RPC method to somthing like
AsyncCallback<List<String>> callback = new AsyncCallback()
{
public void onSuccess( List<String> result ) {
...
}
...
}
and of course relevant changes on server side. That way, you're guaranteed the same ordering client side as on server side.

spring integration modify error message payload

For error handling in my spring integration flow I want to catch exceptions in a service activator which receives its input from a aggregator so its working on a collection of messages. When the exception is thrown though the full collection is sent as the message payload. Instead I want to put the actual item that threw the exception as the content of the error message.
public Collection<File> move(Collection<File> files){
...
//process all files
for(File file : files){
if(file.getName().equals("file-2.done")){
throw new RuntimeException("SOMETHING WENT WRONG");
}
... process the file
}
My exception handler expects to retrieve the file that causes the error
File file = (File) message.getPayload().getFailedMessage().getPayload();
but in this case a collection is send as the payload not a single file. Any help would be appreciated.
The framework doesn't know what happens inside your move method.
You could do something like...
public classs MyFileFailureException extends RuntimeException {
private final File file;
public MyFileFailureException(String msg, File file) {
super(msg);
this.file = file;
}
public File getFailedFile() {return this.file}
}
Then in move()...
throw new MyFileFailureException("message", file);
Then, access it with...
message.getPayload().getCause().getFailedFile().

Resources