I've managed to send all local files to the target ftp server folder with following config:
#Bean
#ServiceActivator(inputChannel = FtpDef.FTP_OUTBOUND_CHANNEL)
public MessageHandler handler() {
FtpMessageHandler handler = new FtpMessageHandler(ftpSessionFactory());
handler.setRemoteDirectoryExpression(
// only one path can be set here
new LiteralExpression("/path/on/ftp/"));
return handler;
}
now I need each file saved in a directory structure as the local.
e.g.
/base/a/a.txt => /path/on/ftp/a/a.txt
/base/a/aa.txt => /path/on/ftp/a/aa.txt
/base/b/b.txt => /path/on/ftp/b/b.txt
/base/b/bb.txt => /path/on/ftp/b/bb.txt
how can I accomplish that, I
new LiteralExpression("/path/on/ftp/")
Don't use a LiteralExpression, which is, er... literal.
Instead, use:
new SpelExpressionParser().parseExpression(rdExpression)
Where rdExpression is something like...
"'/path/on/ftp/' + payload.absolutePath"
Related
How/where can I compute md5 digest for a file I need to transfer to a samba location in spring-integration in order to validate it against the digest I receive at the beginning of the flow. I get the file from a rest service and I have to make sure file is safely landing to samba location. The middle flow looks like this: (the digest to be compared against is stored somewhere in the messages)
GenericHandler smbUploader;
HttpRequestExecutingMessageHandler httpDownloader;
from(inbound()) //here I receive a notification with url where to download file + a checksum to be validated against
...
.handle(httpDownloader) //here I get file effectively
.handle(smbUploader) //here I upload the file to samba
...
and httpDownloader is defined like this:
public HttpRequestExecutingMessageHandler httpDownloader(){
HttpRequestExecutingMessageHandler h = new HttpRequestExecutingMessageHandler ("payload.url");
h.setExpectedResponseType(String.class);
h.setHttpMethod(GET);
return h;
}
and smbUploader is defined like this:
public GenericHandler smbUploader (MessageHandler smbMessageHandler){
return new GenericHandler<Message>(){
#Override
public Message handle(Message m, MessageHeaders h){
smbMessageHandler.handleMessage(m);
return m;
}
}
and smbMessageHandler is defined like this:
public MessageHandler smbMessageHandler (SmbRemoteFileTemplate template, FileNameGenerator g){
SmbMessageHandler h = new smbMessageHandler (template, REPLACE);
h.setAutoCreateDirectory(true);
h.setRemoteDirectoryExpression(getExpression("headers['msg'].smbFolder"));
h.setFileNameGenerator(g);
return h;
}
the inbound (starting the flow) is defined like this:
public HttpRequestHandlerEndpointSpec inbound(){
return Http.inboundChannelAdapter ("/notification")
.requestMapping(m->m.methods(POST))
.requestPayloadType(String.class)
.validator(notificationValidator);
}
First of all you should store a digest in the message headers in the beginning of the flow.
Then you need to write a service method to calculate a checksum of the file you got downloaded. And insert a new handle() in between:
.handle(httpDownloader) //here I get file effectively
.handle(smbUploader) //here I upload the file to samba
to call your service method. The input for that method must be a whole Message, so you got access to the downloaded file in the payload and digest in the headers. The result of this method could be just your file to proceed into an SMB handler for uploading.
How to calculate a checksum you can find in this SO thread: Getting a File's MD5 Checksum in Java
I want to get all file under a particular remote directory in a periodic manner. I am able to get the file under that directory only once after application startup. Not sure why the Poller is not working. This is registered in a spring boot project and the version is 2.2.1
#InboundChannelAdapter(value = "sftpReportChannel",
poller = #Poller(fixedDelay = "5000"))
public String filesForGET(){
return "/etl/biq/autoscore/output/report-data/";
}
#Bean
public IntegrationFlow sftpGetFlow(SessionFactory<ChannelSftp.LsEntry> csf) {
return IntegrationFlows.from("sftpReportChannel")
.handle(Sftp.outboundGateway(csf,
AbstractRemoteFileOutboundGateway.Command.LS, "payload")
.options(AbstractRemoteFileOutboundGateway.Option.RECURSIVE, AbstractRemoteFileOutboundGateway.Option.NAME_ONLY)
//Persistent file list filter using the server's file timestamp to detect if we've already 'seen' this file.
.filter(new SftpPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "autoscore-meta-data")))
.split()
.log(message -> "file path -> "+message.getPayload())
.handle(Sftp.outboundGateway(csf, AbstractRemoteFileOutboundGateway.Command.GET, "'/etl/biq/autoscore/output/report-data/' + payload")
.options(AbstractRemoteFileOutboundGateway.Option.STREAM))
.handle(new ReportHandler()) //get the payload and create email content and send eamil to recipients
.get();
}
The .filter(new SftpPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "autoscore-meta-data"))) makes it working the way that it doesn't pick up the same file again and again on the subsequent poll activities.
Make sure you add new files in that remote dir at runtime or modify already processed file. The SftpPersistentAcceptOnceFileListFilter logic relies on the mtime property of the LsEntry to determine that the file has been changed therefore it is good for processing again.
I have some variables which are going to be used by the business logic part of a function. Therefore, instead of adding them inside the appsetting.json file, I have added a separated file as variable.json
Testing on my machine works but after deploy, it seems function can not find it. and I got an error:
The properties for this file is like the below image. (The build action was None before, but nothing has been changed even by content)
and the below image shows how it looks like in root
And because of that reason, any call the response will be "Function host is not running."
The code for reading this file (path = "Variables.json")
private static List<Variable> GetVariables(string path)
{
string json = File.ReadAllText(path);
var variables = JsonConvert.DeserializeObject<List<Variable>>(json);
return variables;
}
Does anyone have any clue why this is happening?
Problem was because when we start Azure Function locally the file varibale.json is available by Directory.GetCurrentDirectory(), but published on azure portal it's Directory.GetCurrentDirectory() + #"\site\wwwroot"
To get the correct folder path you can use following code:
public static HttpResponseMessage Run(HttpRequestMessage req, ExecutionContext context)
{
var path = System.IO.Path.Combine(context.FunctionDirectory, "varibale.json");
// ...
}
For startup.cs, you can use the following code:
var executioncontextoptions = builder.Services.BuildServiceProvider()
.GetService<IOptions<ExecutionContextOptions>>().Value;
var currentDirectory = executioncontextoptions.AppDirectory;
I am using Azure App Configuration Store to store configuration. I am using the following code in startup.cs to load my config from Azure.
var builder = new ConfigurationBuilder();
builder.AddAzureAppConfiguration(options =>
{
options.Connect(this.Values.AppConfigConnectionString);
options.Select(keyFilter: KeyFilter.Any, labelFilter: this.Values.Env);
});
var config = builder.Build();
Now this config variable contains my queue names. I need this dynamic so to create and handle it in 4 different environments. Dev / Stage / QA / Prod.
public async Task Run(
[QueueTrigger("%QueueName%", Connection = "StorageConnection")]VoiceHubEvent item)
This isn't working as my local.settings.json file doesn't contain QueueName entry.
Is it possible to make use of config variable in Run() to resolve queuename? By reloading queue trigger function or something?
Thanks,
Kiran.
Is it possible to make use of config variable in Run() to resolve queuename? By reloading queue trigger function or something?
Yes, you can.
Create an extensions method for the IWebJobsBuilder interface to set up a connection to AzureAppConfiguration.
public static IWebJobsBuilder AddAzureConfiguration(this IWebJobsBuilder webJobsBuilder)
{
//-- Get current configuration
var configBuilder = new ConfigurationBuilder();
var descriptor = webJobsBuilder.Services.FirstOrDefault(d => d.ServiceType == typeof(IConfiguration));
if (descriptor?.ImplementationInstance is IConfigurationRoot configuration)
configBuilder.AddConfiguration(configuration);
var config = configBuilder.Build();
//-- Add Azure Configuration
configBuilder.AddAzureAppConfiguration(options =>
{
var azureConnectionString = config[TRS.Shared.Constants.CONFIGURATION.KEY_AZURECONFIGURATION_CONNECTIONSTRING];
if (string.IsNullOrWhiteSpace(azureConnectionString)
|| !azureConnectionString.StartsWith("Endpoint=https://"))
throw new InvalidOperationException($"Missing/wrong configuration value for key '{TRS.Shared.Constants.CONFIGURATION.KEY_AZURECONFIGURATION_CONNECTIONSTRING}'.");
options.Connect(azureConnectionString);
});
//build the config again so it has the key vault provider
config = configBuilder.Build();
return webJobsBuilder;
}
Where the azureConnectionString is read from you appsetting.json and should contain the url to the Azure App Configuration.
In startup.cs:
public void Configure(IWebJobsBuilder builder)
{
builder.AddAzureConfiguration();
ConfigureServices(builder.Services)
.BuildServiceProvider(true);
}
For more details, you could refer to this SO thread.
I've been trying to use SftpInboundFileSynchronizer with a remote directory that contains a subdir, say /myfiles/mysubdir/lefile.txt, I have set a filter to grab the files inside the dirs:
mysync.setRemoteDirectory("myfiles/");
mysync.setFilter(new SftpRegexPatternFileListFilter(".*\\.txt$"));
And then a SftpInboundFileSynchronizingMessageSource as my InboundChannelAdapter
I have set on the SftpInboundFileSynchronizingMessageSource a RecursiveDirectoryScanner as scanner and i have no set limit to the depth or the amount of files to retrieve. I also set a FOLLOW_LINKS fileVisitOption on the scanner for good measure.
I am only able to pull files into the local directory from the myfiles path, but anything deeper is not copied to the local dir.
I can't for the life of me figure out if there is something I'm not doing.
EDIT:
What would the InboundChannelAdapter contain if I'm only going to send "/" as the directory to check with mget -R?
#Bean
#InboundChannelAdapter(value = "sftpChannel", poller = #Poller(fixedDelay = "10"))
public MessageSource<?> myMessageSource() {
}
#Bean(name = "myGateway")
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handler() {
SftpOutboundGateway gateway =
new SftpOutboundGateway(sftpSessionFactory(), "mget", "'myfiles/*'");
gateway.setOutputChannelName("listSplitter");
gateway.setOptions("-R");
gateway.setAutoCreateLocalDirectory(true);
myLocalPath = Paths.get(myLocalParentDir).toRealPath().toString();
gateway.setLocalDirectory(new File(myLocalPath));
SftpRegexPatternFileListFilter regexFilter = new regexFilter("^.*\\.txt");
regexFilter.setAlwaysAcceptDirectories(true);
regexFilter.setFilter(sftpRegexPatternFileListFilter);
return gateway;
}
Recursion of the remote file system is not supported by the inbound synchronizer; use an SftpOutboundGateway (request/reply) instead, with a recursive mget command.
By default, files existing in the local directory are not re-fetched; you can control that with the FileExistsMode.