I have a process that periodically gets files from a server and copy them with SFTP to a local directory. It should not overwrite the file if it already exists. I know with something like Winston I can automatically rotate the log file when it fills up, but in this case I need a similar functionality to rotate files if they already exist.
An example:
The routine copies a remote file called testfile.txt to a local directory. The next time it's run the same remote file is found and copied. But now I want to rename the first testfile.txt to testfile.txt.0 so it's not overwritten. And so on - after a while I'd have a directory of files with the name testfile.txt.N and the most recent testfile.txt.
What you can do is you can append date and time on the file name that gives every filename a unique name and also helps you archive it.
For example you text.txt can be either 20170202_181921_test.txt or test_20170202_181921.txt
You can use a JavaScript Date Object to get date and time.
P.S show your code of downloading files so that I can add more to that.
Related
I'd like my bash script to perform an action every time new file is downloaded to /Downloads (generate hash of downloaded file and send it to API). So far I've been trying to make use of "inotify-tools", but it works only for newly created file and that won't do.
Script should work like this:
I download a file via browser (normal way)
Script notices new file and is executed automatically
Thanks in advance for help :D
You can use /etc/crontab to check ~/Downloads folder at startup and every n minutes. Script that will run every nth minute can do either
Keep the number of files. If number decreases script updates cache. And if number increases then gets the latest created file (or modified) and sends that file's hash to the api via curl.
Keep the name of files. If a file no longer exists, script then updates the cache of file names. If a new file appears again hashes and sends hash to the api via curl.
You can keep cache of files under /tmp.
If you can provide an example scenario I can write a simple script
This question already has answers here:
SSIS - How to loop through files in folder and get path+file names and finally execute stored Procedure with parameter as Path + Filename
(2 answers)
Closed 3 years ago.
I have a xlsx file that will be dropped into a folder on a monthly basis. The filename will change every month (filename_8292019) based on the date, to which I cannot change.
I want to build a foreach loop to pick up the xlsx file and manipulate it (load into SQL server table, the move the file to an archive folder). I cannot figure out how to do this with a dynamic filename (where the date changes.
I was able to successfully run the package when converting the xlsx to CSV, and also when pointing directly to the xlsx filename.
[Flat File Destination [219]] Error: Cannot open the datafile "filename"
OR errors relating to file not found
The Files: entry on the Collection tab of the Foreach Loop container will accept wildcard characters.
The general pattern here is to create a variable, say, FileName. Set your Files: to something like:
Files:
BaseFileName*
or, if you want to be sure to only pick up spreadsheets, maybe:
Files:
BaseFileName*.xlsx
Select either Name and extension or Fully qualified, which will include the full file path. I usually just use Name and extension and put the file path into another variable so when Ops tells me they're moving my drop location, I can change a parameter instead of editing the package. This step tells the container to remember the name of the file it just found so you can use it later for a variable mapping.
On the Variable Mappings tab, select your variable name and assign it to Index 0.
Then, for each spreadsheet, the container will loop, pick up the name of the first file it finds that matches your pattern, and assign the full name, with the date extension (and path, if you go that way), to your variable. Pass the variable as in input parameter to the tasks inside the loop and use that to process the file, including moving it to the archive, or you'll get yourself into an infinite loop, processing the same file(s) over and over. <--Does that sound like the voice of experience? Yeah. Been there, done that.
Edit:
Here, the FullFilePath variable is just the folder name, without a file reference. (Red variable to red entry in the Folder box).
The FileBaseName variable drives what shows up in the Files box. (Blue to blue).
Another variable picks up the actual file name, with the date extension. Later, say in a File System Task, if I need the folder & file name together, I concatenate the variables.
As far as the Excel Connection Manager error you're getting, unfortunately I'm no help. I don't use it. We have SentryOne's Task Factory for SSIS which includes a much more resilient Excel connector.
I have a very specific question. I am using Debian.
I have an FTP folder where an app will upload a pdf-file, and the file will be stored in ftpfolder/EMAIL_ADDRESS, and the name of the file will be CURRENT_DATE_AND_TIME.
What I want to do is whenever a new file is uploaded, in either of the EMAIL_ADDRESS folders, to send the file with mpack. As you might have guessed I want the file sent to the name of the folder, with the file attached.
So to break it down I need to:
Detect whenever a new file is uploaded
Extract the address from the foldername
Extract the filename, and attach it with mpack
Send it
I am stumped on how to approach this problem, so any suggestions will be greatly appreciated!
How about a cron that would launch a script doing all the stuff you need and then archive the files found in another folder?
#!/usr/bin/env bash
cd ftpfolder;
for email in *; do
mpack -s "New PDF file uploaded" $email/* $email;
mv $email /archivefolder;
done
Pros:
simplicity
Cons:
you have to have write permissions to move files
messing up with the original files
Note that the above script assumes only one file appears in the folder between the cron executions. If you cannot assure that (i.e. expect more than one file within a minute) you might have to loop over the folder contents.
I looked to TortoiseSVN and TortoiseHG... bit too much for me and I don't want to setup a structure where the files will be saved. All I want is:
When saving a file, first saves the file.
Then if there isn't in file's folder a sub-folder called "History", it creates it.
Then Copies the file to "History" and gives it the first available number suffix as "_[version]".
Then, if is no txt file, in the file's folder, called "[filename].[extension]_Notes.txt", it creates it and adds a line with:
Version: [version] " of the " [filename] " saved at: " [date]
Now how would this run from any application? Dunno. Maybe have a button in the file save dialog?
I'm asking this because I'm just one guy working and not on huge projects and 99% of the time I would just like to have a localized versioning, set on a local sub-folder and a log to track the versions and dates.
Is this super easy with Tortoise and I'm being a mule?
Cheers
I have been using Node.JS file system module for performing various file related operations. I have a need to verify the file name if exists in a directory and if exists i would need to keep a suffix at the end of the file. Typically how windows does with duplicate file names..
if TestFile.txt already exists and another file with same names comes in during processing the new file should be renamed as TestFile (1).txt and next file with same name should be renamed as TestFile (2).txt.
What could be the best way to achieve this. Do i have to use a temporary array to keep all file names and traverse through for each? This is a multi threaded environment and there could be 50,000+ documents coming for processing.
Thanks a ton.