Building a Handler to Log over SFTP - groovy

I'm attempting to build a Handler to perform a put operation with every logging event on a log file over Sftp. Currently, I've built a new logger in line of my Groovy script that uses an SftpConnector artifact's ChannelSftp to perform a put and append to a log file. When I run my code I receive an error message saying it's an Invalid Type=105. If I append to a string and at the end of my script upload the strings contents to my log file then there are no issues. I'm guessing that the error I'm receiving is due to multiple put operations on the same file in rapid secession?
def LOG = Logger.getLogger(Logger.GLOBAL_LOGGER_NAME)
def handler = new Handler() {
///String outputBuffer
#Override
void publish(LogRecord record) {
ProgressMonitor monitor = new ProgressMonitor()
String aRecord = new SimpleDateFormat("MMM dd, YYYY hh:mm:ss aaa").format(new Date()).toString() + " " + record.level.toString() + ":" + " " + record.message.toString()
def stream = IOUtils.toInputStream(aRecord,"UTF-8")
connector.getChannelSftp().put(stream ,props.getProperty("sftp.log"),monitor,2)
while(!monitor.isFinished()){
//just pause until logging is done.
}
stream.close()
//outputBuffer = outputBuffer + new SimpleDateFormat("MMM dd, YYYY hh:mm:ss aaa").format(new Date()).toString() + " " + record.level.toString() + ":" + " " + record.message.toString() + "\n"
}
#Override
void flush() {
}
#Override
void close() throws SecurityException {
}
void push(){
connector.getChannelSftp().put(IOUtils.toInputStream(outputBuffer, "UTF-8"),
props.getProperty("sftp.log"), 2)
connector.getChannelSftp().put(IOUtils.toInputStream('\n'),
props.getProperty("sftp.log"),2)
}
}
//SftpHandler handler = new SftpHandler(props)
//def handler = new FileHandler(new
File(props.getProperty("log.location")).absolutePath, true) //provides a writer for log file.
handler.setFormatter(new SimpleFormatter()) //defines the logger file
format. must be declared
LOG.addHandler(handler) //adding the handler.

I figured out the problem:
When attempting a put on the open ChannelSftp it works fine so long as the previous put operation has completed. If there are multiple puts on the same channel over the same session the server will execute and lock the file.
The way to get around this is to open a new channel, execute the put, and close the channel after inside the custom handler. Each put command is executed over a different session and will be queued if the file is locked rather than simply rejected.

Related

Kotlin use variable used in a while loop

I want to use a function in loop condition and reuse it in the loop, like this code in Java, but I want it in kotlin :
while ( (data = in.readLine()) != null ) {
System.out.println("\r\nMessage from " + clientAddress + ": " + data);
}
I tried copying this code in android studio to automatically convert to Kotlin and the code looks like this :
while (reader.readLine().also { line = it } != null) {
Log.d("Line", line)
lines.add(line)
}
So I managed to get the lines with var line = "" before the loop, but the loop doesn't stop when the reader is done getting the message sent from the socket.
My wish is to send 2 messages through the socket, so I try to get all lines of one message, and when the second one arrives, I have to clear my lines variable to get the next message, but I can't.
Thanks !
You can use reader.lineSequence() function:
reader.lineSequence().forEach {
Log.d("Line", it)
lines.add(it)
}
Or as suggested in documentation, you can improve it by using File.useLines() which automatically closes the File reader so you don't have to do it manually

Copy remote file in sftp using Spring Integration

I need to copy/duplicate remote file in Sftp server also rename when copied, I read here that copying remote file in Sftp isn't supported so the only the available option I had is to GET file into Local and then PUT again to Sftp & delete the Local file, I have successfully achieved my goal but the problem is there is a log printing from org.springframework.core.log.LogAccessor: I have no idea from where it is coming.
Code that helps in copying remote file:
#Bean
public IntegrationFlow copyRemoteFile() {
return IntegrationFlows.from("integration.channel.copy")
.handle(Sftp.outboundGateway(sftpSessionFactory(),
AbstractRemoteFileOutboundGateway.Command.GET,
"headers[" + COPY_SOURCE_PATH.value + "]+'/'+" +
"headers[" + COPY_SOURCE_FILENAME.value + "]")
.autoCreateLocalDirectory(true)
.fileExistsMode(FileExistsMode.REPLACE)
.localDirectory(new File(localPath)))
.log(LoggingHandler.Level.INFO, "SftpCopyService")
.handle(Sftp.outboundGateway(sftpSessionFactory(),
AbstractRemoteFileOutboundGateway.Command.PUT,
"payload")
.remoteDirectoryExpression("headers[" + COPY_DEST_PATH.value + "]")
.fileNameGenerator(n -> (String)n.getHeaders().get(COPY_DEST_FILENAME.value))
.fileExistsMode(FileExistsMode.REPLACE))
.log(LoggingHandler.Level.INFO, "SftpCopyService")
.handle((p, h) -> {
try {
return Files.deleteIfExists(
Paths.get(localPath + File.separator + h.get(COPY_SOURCE_FILENAME.value)));
} catch (IOException e) {
e.printStackTrace();
return false;
}
})
.get();
Here is the log.
2021-02-16 18:10:22,577 WARN [http-nio-9090-exec-1] org.springframework.core.log.LogAccessor: Failed to delete C:\Users\DELL\Desktop\GetTest\Spring Integration.txt
2021-02-16 18:10:22,784 INFO [http-nio-9090-exec-1] org.springframework.core.log.LogAccessor: GenericMessage [payload=C:\Users\DELL\Desktop\GetTest\Spring Integration.txt, headers={file_remoteHostPort=X.X.X.X:22, replyChannel=nullChannel, sourceFileName=Spring Integration.txt, file_remoteDirectory=/uploads/, destFileName=Spring Integrat.txt, destPath=uploads/dest, id=5105bdd1-8180-1185-3661-2ed708e07ab9, sourcePath=/uploads, file_remoteFile=Spring Integration.txt, timestamp=1613479222779}]
2021-02-16 18:10:23,011 INFO [http-nio-9090-exec-1] org.springframework.core.log.LogAccessor: GenericMessage [payload=uploads/dest/Spring Integrat.txt, headers={file_remoteHostPort=X.X.X.X:22, replyChannel=nullChannel, sourceFileName=Spring Integration.txt, file_remoteDirectory=/uploads/, destFileName=Spring Integrat.txt, destPath=uploads/dest, id=1bf83b0f-3b24-66bd-ffbf-2a9018b499fb, sourcePath=/uploads, file_remoteFile=Spring Integration.txt, timestamp=1613479223011}]
The more surprising part is, it appears very early even before the flow is executed, though I have handled file deletion at very last. How can i get rid of this log message? though it doesn't effect my process but the log message is misleading
Also is there any better way to copy remote file to another path inside sftp
EDIT
Like you suggested I tried the SftpRemoteFileTemplate.execute() method to copy files in sftp but when the session.write(InputStream stream,String path) method is called the method control never returns it keeps the control forever
I tried debugging, the control is lost when the execution reaches here:
for(_ackcount = this.seq - startid; _ackcount > ackcount && this.checkStatus((int[])null, header); ++ackcount) {
}
This code sits inside _put method of ChannelSftp.class
Here is the sample code that I'm trying
public boolean copy() {
return remoteFileTemplate.execute(session -> {
if (!session.exists("uploads/Spring Integration.txt")){
return false;
}
if (!session.exists("uploads/dest")){
session.mkdir("uploads/dest");
}
InputStream inputStream = session.readRaw("uploads/Spring Integration.txt");
session.write(inputStream, "uploads/dest/spring.txt");
session.finalizeRaw();
return true;
});
}
Would you please point out what mistake I'm doing here?
Instead of writing the whole flow via local file copy, I'd suggest to look into a single service activator for the SftpRemoteFileTemplate.execute(SessionCallback<F, T>). The provided SftpSession in that callback can be used for the InputStream readRaw() and write(InputStream inputStream, String destination). In the end you must call finalizeRaw().
The LogAccessor issue is not clear. What Spring Integration version do you use? Do you override Spring Core version though?
I think we can improve that WARN message and don't call File.delete() if it does not exists().
Feel free to provide such a contribution!
UPDATE
The JUnit test to demonstrate how to perform a copy on SFTP server:
#Test
public void testSftpCopy() throws Exception {
this.template.execute(session -> {
PipedInputStream in = new PipedInputStream();
PipedOutputStream out = new PipedOutputStream(in);
session.read("sftpSource/sftpSource2.txt", out);
session.write(in, "sftpTarget/sftpTarget2.txt");
return null;
});
Session<?> session = this.sessionFactory.getSession();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
FileCopyUtils.copy(session.readRaw("sftpTarget/sftpTarget2.txt"), baos);
assertThat(session.finalizeRaw()).isTrue();
assertThat(new String(baos.toByteArray())).isEqualTo("source2");
baos = new ByteArrayOutputStream();
FileCopyUtils.copy(session.readRaw("sftpSource/sftpSource2.txt"), baos);
assertThat(session.finalizeRaw()).isTrue();
assertThat(new String(baos.toByteArray())).isEqualTo("source2");
session.close();
}

Azure Durable function removes files form local storage after it is downloaded

I am struggling a lot with this task. I have to download files from SFTP and then parse them. I am using Durable functions like this
[FunctionName("MainOrch")]
public async Task<List<string>> RunOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context, ILogger log)
{
try
{
var filesDownloaded = new List<string>();
var filesUploaded = new List<string>();
var files = await context.CallActivityAsync<List<string>>("SFTPGetListOfFiles", null);
log.LogInformation("!!!!FilesFound*******!!!!!" + files.Count);
if (files.Count > 0)
{
foreach (var fileName in files)
{
filesDownloaded.Add(await context.CallActivityAsync<string>("SFTPDownload", fileName));
}
var parsingTasks = new List<Task<string>>(filesDownloaded.Count);
foreach (var downlaoded in filesDownloaded)
{
var parsingTask = context.CallActivityAsync<string>("PBARParsing", downlaoded);
parsingTasks.Add(parsingTask);
}
await Task.WhenAll(parsingTasks);
}
return filesDownloaded;
}
catch (Exception ex)
{
throw;
}
}
SFTPGetListOfFiles: This functions connects to SFTP and gets the list of files in a folder and return.
SFTPDownload: This function is suppose to connect to SFTP and download each file in Azure Function's Tempt Storage. and return the download path. (each file is from 10 to 60 MB)
[FunctionName("SFTPDownload")]
public async Task<string> SFTPDownload([ActivityTrigger] string name, ILogger log, Microsoft.Azure.WebJobs.ExecutionContext context)
{
var downloadPath = "";
try
{
using (var session = new Session())
{
try
{
session.ExecutablePath = Path.Combine(context.FunctionAppDirectory, "winscp.exe");
session.Open(GetOptions(context));
log.LogInformation("!!!!!!!!!!!!!!Connected For Download!!!!!!!!!!!!!!!");
TransferOptions transferOptions = new TransferOptions();
transferOptions.TransferMode = TransferMode.Binary;
downloadPath = Path.Combine(Path.GetTempPath(), name);
log.LogInformation("Downloading " + name);
var transferResult = session.GetFiles("/Receive/" + name, downloadPath, false, transferOptions);
log.LogInformation("Downloaded " + name);
// Throw on any error
transferResult.Check();
log.LogInformation("!!!!!!!!!!!!!!Completed Download !!!!!!!!!!!!!!!!");
}
catch (Exception ex)
{
log.LogError(ex.Message);
}
finally
{
session.Close();
}
}
}
catch (Exception ex)
{
log.LogError(ex.Message);
_traceService.TraceException(ex);
}
return downloadPath;
}
PBARParsing: function has to get the stream of that file and process it (processing a 60 MB file might take few minutes on Scale up of S2 and Scale out with 10 instances.)
[FunctionName("PBARParsing")]
public async Task PBARParsing([ActivityTrigger] string pathOfFile,
ILogger log)
{
var theSplit = pathOfFile.Split("\\");
var name = theSplit[theSplit.Length - 1];
try
{
log.LogInformation("**********Starting" + name);
Stream stream = File.OpenRead(pathOfFile);
i want the download of all files to be completed using SFTPDownload thats why "await" is in a loop. and then i want parsing to run in parallel.
Question 1: Does the code in MainOrch function seems correct for doing these 3 things 1)getting the names of files, 2) downloading them one by one and not starting the parsing function until all files are downloaded. and then 3)parsing the files in parallel. ?
I observed that what i mentioned in Question 1 is working as expected.
Question 2: 30% of the files are parsed and for the 80% i see errors that "Could not find file 'D:\local\Temp\fileName'" is azure function removing the files after i place them ? is there any other approach i can take? If i change the path to "D:\home" i might see "File is being used by another process" error. but i haven't tried it yet. out the 68 files on SFTP weirdly last 20 ran and first 40 files were not found at that path and this is in sequence.
Question3: I also see this error " Singleton lock renewal failed for blob 'func-eres-integration-dev/host' with error code 409: LeaseIdMismatchWithLeaseOperation. The last successful renewal completed at 2020-08-08T17:57:10.494Z (46005 milliseconds ago) with a duration of 155 milliseconds. The lease period was 15000 milliseconds." does it tells something ? it came just once though.
update
after using "D:\home" i am not getting file not found errors
For others coming across this, the temporary storage is local to an instance of the function app, which will be different when the function scales out.
For such scenarios, D:\home is a better alternative as Azure Files is mounted here, which is the same across all instances.
As for the lock renewal error observed here, this issue tracks it but shouldn't cause issues as mentioned. If you do see any issue because of this, it would be best to share details in that issue.

P4API.net: how to use P4Callbacks delegates

I am working on a small tool to schedule p4 sync daily at specific times.
In this tool, I want to display the outputs from the P4API while it is running commands.
I can see that the P4API.net has a P4Callbacks class, with several delegates: InfoResultsDelegate, TaggedOutputDelegate, LogMessageDelegate, ErrorDelegate.
My question is: How can I use those, I could not find a single example online of that. A short example code would be amazing !
Note: I am quite a beginner and have never used delegates before.
Answering my own questions by an example. I ended up figuring out by myself, it is a simple event.
Note that this only works with P4Server. My last attempt at getting TaggedOutput from a P4.Connection was unsuccessful, they were never triggered when running a command.
So, here is a code example:
P4Server p4Server = new P4Server(syncPath);
p4Server.TaggedOutputReceived += P4ServerTaggedOutputEvent;
p4Server.ErrorReceived += P4ServerErrorReceived;
bool syncSuccess = false;
try
{
P4Command syncCommand = new P4Command(p4Server, "sync", true, syncPath + "\\...");
P4CommandResult rslt = syncCommand.Run();
syncSuccess=true;
//Here you can read the content of the P4CommandResult
//But it will only be accessible when the command is finished.
}
catch (P4Exception ex) //Will be caught only when the command has completely failed
{
Console.WriteLine("P4Command failed: " + ex.Message);
}
And the two methods, those will be triggered while the sync command is being executed.
private void P4ServerErrorReceived(uint cmdId, int severity, int errorNumber, string data)
{
Console.WriteLine("P4ServerErrorReceived:" + data);
}
private void P4ServerTaggedOutputEvent(uint cmdId, int ObjId, TaggedObject Obj)
{
Console.WriteLine("P4ServerTaggedOutputEvent:" + Obj["clientFile"]);
}

FileSystemWatcher not firing upon text file change and how to get appended lines into panel

I'm after some help or guidance if possible.
I'm trying to monitor a text file in real time and copy the appended text into panel within windows forms.
The text file is being updated via an exe file so it might be few lines appended in quick sessions or nothing for few minutes.
I have tried FileSystemWatcher but it does not seem to work which I cannot understand and also I'm very new to this event handlers etc and still learning in progress :(
private FileSystemWatcher watcher = new FileSystemWatcher();
public async void StandardOutputHandler(object sender, DataReceivedEventArgs outLine)
{
if (outLine.Data != null && !String.IsNullOrWhiteSpace(outLine.Data)) //checks if line coming from CMD is blank or empty
{
// check if cmd output was redirected into a log file
if (outLine.Data.Contains(">>"))
{
BeginInvoke(new MethodInvoker(() =>
{
//get log path and name
string[] commandLine = outLine.Data.Split('>');
this.logFileFullPath = commandLine[3];
this.logFileFullPath = this.logFileFullPath.Replace('"', ' ').Trim();
string[] split = logFileFullPath.Split('\\');
this.logFileName = split[6];
this.path = split[0] + "\\" + split[1] + "\\" + split[2] + "\\" + split[3] + "\\" + split[4] + "\\" + split[5];
//// Create a new FileSystemWatcher and set its properties.
watcher.Path = this.path + "\\";
watcher.Filter = this.logFileName;
//watch for changes to a a log file
watcher.NotifyFilter = (NotifyFilters.LastWrite | NotifyFilters.Size | NotifyFilters.LastAccess | NotifyFilters.CreationTime);
watcher.Changed += new FileSystemEventHandler(OnChanged);
watcher.Created += new FileSystemEventHandler(OnChanged);
// Begin watching.
watcher.EnableRaisingEvents = true;
}));
}
}
// Define the event handlers.
private void OnChanged(object source, FileSystemEventArgs e)
{
MessageBox.Show("Im here");// not showing
//how to copy appended lines into panel??
}
The message box in OnChanged method is not coming up. I have manually amended the file or delete it and create it but the event is not being fired.
I believe this might be related to file being used by another process which in this case is an exe file updating it when necessary. Also, I think, the risk is that I might get only partial text if exe updates the log at the same time as I read the appended lines.
Is there a better way to monitor the text file updates and copy the content onto the panel within GUI application??
UPDATE:
I have moved the code into click event or initial method and it makes no difference. Message box is not appearing
public BatchRun()
{
InitializeComponent();
watcher.Path = "C:\\Test\\Projects\\99999\\Logs";
watcher.Filter = "*.log";
watcher.NotifyFilter = (NotifyFilters.LastWrite | NotifyFilters.Size | NotifyFilters.LastAccess | NotifyFilters.CreationTime);
watcher.Changed += new FileSystemEventHandler(OnChanged);
watcher.Created += new FileSystemEventHandler(OnChanged);
watcher.EnableRaisingEvents = true;
}
It seems that you only begin watching the file in the call to StandardOutputHandler.
If this method is only called after writing to the file, then the watcher won't notice any changes to the file.
You probably want to have all the code in your MethodInvoker action somewhere it is called when the application starts (or at least when it needs to monitor the file).
Then in your OnChanged method you may have to invoke MessageBox.Show() on the UI thread.

Resources