This question already has answers here:
SSIS - How to loop through files in folder and get path+file names and finally execute stored Procedure with parameter as Path + Filename
(2 answers)
Closed 3 years ago.
I have a xlsx file that will be dropped into a folder on a monthly basis. The filename will change every month (filename_8292019) based on the date, to which I cannot change.
I want to build a foreach loop to pick up the xlsx file and manipulate it (load into SQL server table, the move the file to an archive folder). I cannot figure out how to do this with a dynamic filename (where the date changes.
I was able to successfully run the package when converting the xlsx to CSV, and also when pointing directly to the xlsx filename.
[Flat File Destination [219]] Error: Cannot open the datafile "filename"
OR errors relating to file not found
The Files: entry on the Collection tab of the Foreach Loop container will accept wildcard characters.
The general pattern here is to create a variable, say, FileName. Set your Files: to something like:
Files:
BaseFileName*
or, if you want to be sure to only pick up spreadsheets, maybe:
Files:
BaseFileName*.xlsx
Select either Name and extension or Fully qualified, which will include the full file path. I usually just use Name and extension and put the file path into another variable so when Ops tells me they're moving my drop location, I can change a parameter instead of editing the package. This step tells the container to remember the name of the file it just found so you can use it later for a variable mapping.
On the Variable Mappings tab, select your variable name and assign it to Index 0.
Then, for each spreadsheet, the container will loop, pick up the name of the first file it finds that matches your pattern, and assign the full name, with the date extension (and path, if you go that way), to your variable. Pass the variable as in input parameter to the tasks inside the loop and use that to process the file, including moving it to the archive, or you'll get yourself into an infinite loop, processing the same file(s) over and over. <--Does that sound like the voice of experience? Yeah. Been there, done that.
Edit:
Here, the FullFilePath variable is just the folder name, without a file reference. (Red variable to red entry in the Folder box).
The FileBaseName variable drives what shows up in the Files box. (Blue to blue).
Another variable picks up the actual file name, with the date extension. Later, say in a File System Task, if I need the folder & file name together, I concatenate the variables.
As far as the Excel Connection Manager error you're getting, unfortunately I'm no help. I don't use it. We have SentryOne's Task Factory for SSIS which includes a much more resilient Excel connector.
Related
help my code keeps giving me a name error for my lyrics value, for context the external file contains a single character
I have tried editing the external files as well as alternate ways to open the file in hand
this is the code snippet which is relevant:
with open("lyrics" ,"r") as lyrics:
lyrics=lyrics.read()
while counter>=1:
print(lyrics.read([0])," ",lyrics.read([-0]))
I have a source from which the files are to be processed. Multiple files are coming to that loacation at any time randomly (Package should run every 2 hours). I have to process only the new files, i can not delete, move the already processed files from that location. I can only copy the files to Archive location. How can I achieve this ?
You can achieve this using the following steps.
Use the foreach file enumerator for your incoming folder and save
the filename in "IncomingFile" variable. Configure to select "Name
and Extension"[In my code I have used that otherwise you need to do
some modification to the script]
Create tow SSIS variables Like "ArchivePath" as string and
"IsLoaded" as Boolean[default to false].
Create the SSIS script component and use "IncomingFile" and
"ArchivePath" as the readonly variable. "IsLoaded" should be the
ReadandWrite variable.
Write the following code in the script component. If file is already
exists then it will return true. Otherwise False.
public void Main()
{
var archivePath = Dts.Variables["ArchivePath"].Value.ToString();
var incomingFile = Dts.Variables["IncomingFile"].Value.ToString();
var fileFullPath = string.Format(#"{0}\{1}",archivePath,incomingFile);
bool isLoaded = File.Exists(fileFullPath);
Dts.Variables["IsLoaded"].Value = isLoaded;
Dts.TaskResult = (int)ScriptResults.Success;
}
Use the Precedence constraint to call the Data flow task and evaluation operation should be "Expression" . Set something as follows in your expression box.
#IsLoaded==False
Hope this helps.
Your package should process the files in a given directory, then move them to another directory once processed. That way, each time the package runs, it has to fully process the source directory.
To process each files in a directory, use the ForEach Container. You can specify a folder to look in, and some expressions to filter. If, for instance, your filename contains a timestamp, you could use that timestamp to filter your files in or out.
You use a flat file source to read files, then use the filesystem task to move them around.
To start, take a look at the answer here: Enumerate files in a folder using SSIS Script Task
The SSIS Script Task should enumerate all the files in a given folder, then take a snapshot of the already processed files from a table where you will keep a log of what's processed, ignore the already processed ones and just return the non-processed in an object variable for a for-each task to consume.
I have a process that periodically gets files from a server and copy them with SFTP to a local directory. It should not overwrite the file if it already exists. I know with something like Winston I can automatically rotate the log file when it fills up, but in this case I need a similar functionality to rotate files if they already exist.
An example:
The routine copies a remote file called testfile.txt to a local directory. The next time it's run the same remote file is found and copied. But now I want to rename the first testfile.txt to testfile.txt.0 so it's not overwritten. And so on - after a while I'd have a directory of files with the name testfile.txt.N and the most recent testfile.txt.
What you can do is you can append date and time on the file name that gives every filename a unique name and also helps you archive it.
For example you text.txt can be either 20170202_181921_test.txt or test_20170202_181921.txt
You can use a JavaScript Date Object to get date and time.
P.S show your code of downloading files so that I can add more to that.
I've got a file on my local drive that is being replaced with a newer version of it every 10 minutes.
I'd like to make a function reference to this file in another workbook, but the link is being broken every time the automatically generated file is being replaced (which is perfectly logical).
Is there a way to retain a link to the external file?
Perhaps keep an identical name for the file that keeps being replaced.
i.e instead of "Timesheet 10/08/2016 10:07" give it a custom name which remains immutable; that way the link will remain unbroken as it will be referencing the same file
Unfortutnately the file already has a static name (HPI.xlsx) and the link is getting broken anyway.
I have an SSIS File system Task to Copy an Excel template and create one with a date appended. I get the error saying "the connection is not found. This error is thrown by Connections collection when the specific connection element is not found." It's not found because I'm trying to make it!
The preceding error says "Error at File System Task: Failed to lock variable "c:\Reports\Unregistered_20150915.xlsx" for read access with error 0xC0010001 The variable cannot be found. This occurs when an attempt is made to retrieve a variable from the Variables collection on a container during execution of the package, and the variable is not there. The variable name may have changed or the variable is not being created".
I'm new to this, and all I want to do is run a procedure nightly and put the results in a new Excel file with the date name appended to the end. My plan was to take a blank worksheet/file and copy it into a new file, then execute the procedure on the server to load into this new file. As you can see, the variable is filling in the date just fine, so I don't understand how it says the variable cannot be found.
I've used these pages for guidance, but I am taking some of this and some of that, obviously missing something in between:
Including the Date in a Filename in SSIS using an Expression
and SSIS: Export a Query Result to a File
Any suggestions, articles, or videos would be vastly appreciated.
I found the error. When I set the destination variable, I also had an expression that kept kicking the variable out and using the evaluated result, which is not the name of the variable. I guess in some way that only a newbie can, I created the variable with the expression and created an additional expression which the program was trying to use concurrently.