For one of my Project, I have a certain challenge where I need to take all the reports generated in a certain path, I want this to be an automated process in "Linux". I know the way how to get the file names which have been updated in the past 120 mins, but not the files directly. Now my requirements are in such a way
Take a certain files that have been updated in past 120 mins from the path
/source/folder/which/contains/files
Now do some bussiness logic on this generated files which i can take care of
Move this files to
/destination/folder/where/files/should/go
I know how to achieve #2 and #3 but not sure of #1. Can someone help me how can i achieve this.
Thanks in Advance.
Write a shell script. Sample below. I haven't provided the commands to get the actual list of file names as you said you know how to do that.
#!/bin/sh
files=<my file list>
for file in $files; do
cp $file <destination_dirctory>
done
Related
I am trying to rename a file from multiple directories
I have a file called slave.log in multiple directories like slave1,slave2......slave17. so daily log rotation happens and creates a new file with current dateformat in it whereas the file data contains a previous day data . I want to rename those files with previous date format .
I have written shell script which works fine but the problem here is , I need to pass the path as parameter but likewise I have 17 directories so i cant schedule 17 cron enteries to run ...I have only basic knowledge about scripting . Please help me with best solution for this scenario
I wanted to create a job where I need to consider the latest file available as input file.
File format is as below: FILE1.TEST.TYYMMDD
is there any way to identify latest file based on date present in file name via JCL.
P.S. GDG versions are not created in existing process . Only PS file is created.
Thank you
I wanted to create a job where I need to consider the latest file available as input file. File [name] format is as below: FILE1.TEST.TYYMMDD is there any way to identify latest file based on date present in file name via JCL.
No.
You indicate that GDGs are not created in the existing process. GDGs would be the best way to accomplish your goal. Absent GDGs, you must write code.
You could accomplish your goal by writing (C, clist, COBOL, PL/I, Rexx) code using the LMDINIT and LMDLIST ISPF services. Then you would execute your code by running ISPF in batch. Many mainframe shops have a cataloged procedure to execute ISPF in batch.
Agree with #cschneid that there is not a platform way to handle this. However, I want to point out that GDGs are the platform way of managing PS files for access in a relative form.
Your comment
GDG versions are not created in existing process . Only PS file is
created.
That statement didn't make sense to me. GDGs are not a file type like physical sequential (PS) or partitioned (PO). It's a convention to allow relative reference to files created over time which sounds like what you want. I've only seen the use of GDGs for PS files.
Putting the date in the file name can have its uses but to z/OS its only part of the filename and not meta information that it operates on (like G0000v00's in GDGs.
I am using the file component with an argument quartz scheduler in order to pull some files from a given directory on every hour. Then i transform the data from the files and move the content to other files in other directory. After that I am moving the input files to an archive directory. When a file is moved to this directory it should stay there only a week and then it should be deleted automatically. The problem is that Im not really sure how can i start a new cron job because I dont really know when any of the files is moved to that archive directory. Maybe is something really trivial but I am pretty new to camel and I dont know the solution. Thank you in advance.
Use option "filterFile"
Every file has modified timestamp and you can use this timestamp to filter file that are older than 1 week. Under file component, there exist an option filterFile
filterFile=${date:file:yyyyMMdd}<${date:now-7d:yyyyMMdd}
Above evaluation comes from file language, ${date:file:yyyyMMdd} denote modified timestamp of the file in form (year)(month)(day) and ${date:now-7d:yyyyMMdd} denote current time minus 7 days in form (year)(month)(day).
I wonder if you can configure logstash in the following way:
Background Info:
Every day I get a xml file pushed to my server, which should be parsed.
To indicate a complete file transfer afterwards I get an empty .ctl (custom file) transfered to the same folder.
The files both have the following name schema 'feedback_{year}{yearday}_UTC{hoursminutesseconds}_51.{extention}' (e.g. feedback_16002_UTC235953_51.xml). So they have the same file name but one is with .xml and the other is a .ctl file.
Question:
Is there a way to configure logstash to wait parsing the xml file until the according .ctl file is present?
EDIT:
Is there maybe a way to archiev that with filebeat?
EDIT2:
It would also be enough to be able to configure logstash in a way that it will wait x minutes before starting to process a new file, if that is easier.
Thanks for any help in advance
Your problem is that you don't want to start the parser before the file transfer hasn't been completed. So, why don't push the data to a file (file-complete.xml) when you find your flag file (empty.ctl)?
Here is the possible logic for a script and runs using crontab:
if empty.ctl exists:
Clear file-complete.xml
Add the content of file.xml to file-complete.xml.
Remove empty.ctl
This way, you'd need to parse the data from file-complete.xml. I think is simpler to debug and configure.
Hope it helps,
I have a little script to create a directory listing
You can use it like
./createDiectoryListing.sh SOME_DIRECTORY
It will create a index.html show the list of the file, Here is an example
But i found sometime the file's date modified attribute changed to the time i ran the script.
In my script, i just use md5sum to caculate the hash of the file, I never change the content.
It's really annoying, all the file has the same modified date,after i ran the script
I don't know why, anyone can help me out thanks.
Okay. I figured it out. the reason is i use git to manage the file. when switch between branches, the date attribute is changed.