I have an SSIS File system Task to Copy an Excel template and create one with a date appended. I get the error saying "the connection is not found. This error is thrown by Connections collection when the specific connection element is not found." It's not found because I'm trying to make it!
The preceding error says "Error at File System Task: Failed to lock variable "c:\Reports\Unregistered_20150915.xlsx" for read access with error 0xC0010001 The variable cannot be found. This occurs when an attempt is made to retrieve a variable from the Variables collection on a container during execution of the package, and the variable is not there. The variable name may have changed or the variable is not being created".
I'm new to this, and all I want to do is run a procedure nightly and put the results in a new Excel file with the date name appended to the end. My plan was to take a blank worksheet/file and copy it into a new file, then execute the procedure on the server to load into this new file. As you can see, the variable is filling in the date just fine, so I don't understand how it says the variable cannot be found.
I've used these pages for guidance, but I am taking some of this and some of that, obviously missing something in between:
Including the Date in a Filename in SSIS using an Expression
and SSIS: Export a Query Result to a File
Any suggestions, articles, or videos would be vastly appreciated.
I found the error. When I set the destination variable, I also had an expression that kept kicking the variable out and using the evaluated result, which is not the name of the variable. I guess in some way that only a newbie can, I created the variable with the expression and created an additional expression which the program was trying to use concurrently.
Related
I'm trying to use the Power Query source component in a generic way from SSIS (VS2019).
The idea would be to use a for each loop to load and transform Excel files. At run time, I need to set the connection manager properties for each file as well as the PQY script to be executed on the file.
What I did so far is trying to create a JSON connection string inside a script component and assign the connection string to the connection manager. It keeps on saying that the file requires credentials.
Would someone already experienced that kind of dev? All the files do have the same structure so far, do meta-data need to be refreshed too?
[Edit]
1. In the control flow, I'm retrieving the PQY script I want to apply from a DB.
Before transormations, script starts like this:
let Source = Excel.Workbook(File.Contents("path_to_a_file.xlsx"),null,true),RawData_Sheet = Source{[Item="Table1",Kind="Table"]}[Data]..."
In the C# script task, I'm replacing the path to excel file by the current file variable. M Script is stored in a variable used in the PQY component.
C# Script is then updating the PQY connection manager to target the appropriate file:
ConnectionManager _conn = Dts.Connections["Power Query Connection Manager"];
String _ConnectString = "[{kind:File,path:path_to_a_file.xlss,AuthenticationKind:Windows,Username:myusername,Password:mypassword}]";
_conn.ConnectionString = _ConnectString;
The PQY component is left has it is, connected to ["Power Query Connection Manager"] and getting its script from the variable I set.
PQY configuration screen
Thanks for any tip on this,
Olivier
I can't address the specifics of the PQ but generic anything in a Data Flow will not work.
The Data Flow task works because it makes a strict contract between the source(s) and the destination(s). These columns with these data types will be in play during the run. It's a design-time contract because that allows the run-time engine to allocate resources based on how many buffers of data the system can support. Each row is X bytes, we have Y bytes of memory available, so Z buffers worth of data plus parallelism stuff.
Wish I had a better story to tell you.
This question already has answers here:
SSIS - How to loop through files in folder and get path+file names and finally execute stored Procedure with parameter as Path + Filename
(2 answers)
Closed 3 years ago.
I have a xlsx file that will be dropped into a folder on a monthly basis. The filename will change every month (filename_8292019) based on the date, to which I cannot change.
I want to build a foreach loop to pick up the xlsx file and manipulate it (load into SQL server table, the move the file to an archive folder). I cannot figure out how to do this with a dynamic filename (where the date changes.
I was able to successfully run the package when converting the xlsx to CSV, and also when pointing directly to the xlsx filename.
[Flat File Destination [219]] Error: Cannot open the datafile "filename"
OR errors relating to file not found
The Files: entry on the Collection tab of the Foreach Loop container will accept wildcard characters.
The general pattern here is to create a variable, say, FileName. Set your Files: to something like:
Files:
BaseFileName*
or, if you want to be sure to only pick up spreadsheets, maybe:
Files:
BaseFileName*.xlsx
Select either Name and extension or Fully qualified, which will include the full file path. I usually just use Name and extension and put the file path into another variable so when Ops tells me they're moving my drop location, I can change a parameter instead of editing the package. This step tells the container to remember the name of the file it just found so you can use it later for a variable mapping.
On the Variable Mappings tab, select your variable name and assign it to Index 0.
Then, for each spreadsheet, the container will loop, pick up the name of the first file it finds that matches your pattern, and assign the full name, with the date extension (and path, if you go that way), to your variable. Pass the variable as in input parameter to the tasks inside the loop and use that to process the file, including moving it to the archive, or you'll get yourself into an infinite loop, processing the same file(s) over and over. <--Does that sound like the voice of experience? Yeah. Been there, done that.
Edit:
Here, the FullFilePath variable is just the folder name, without a file reference. (Red variable to red entry in the Folder box).
The FileBaseName variable drives what shows up in the Files box. (Blue to blue).
Another variable picks up the actual file name, with the date extension. Later, say in a File System Task, if I need the folder & file name together, I concatenate the variables.
As far as the Excel Connection Manager error you're getting, unfortunately I'm no help. I don't use it. We have SentryOne's Task Factory for SSIS which includes a much more resilient Excel connector.
I am on Day one in learning Excel Macros. As per the tutorial I recorded the macros and tried to run it. (I have used the same file that they have provided in the tutorial and it is a .csv file)
But it showing me the following Error
Runtime Error
Invalid Procedure call or Argument
When I tried to debug it is showing as follows:
Need Help
-Thanks
You can set the CommandType property only if the value of the QueryType property for the query table or PivotTable cache is xlOLEDBQuery, which is enum 5.
This is not the case for your query. In fact, that line doesn't even exist in the source code. Perhaps you attempted to merge scripts? If so, be careful when merging connection code that the types are the same.
If you add in a debug line to get the QueryType
Debug.Print .QueryType
.Name = "receipts"
you will get the enum 6 back. You can look this up using the object explorer:
As would be expected from the connection specification:
ActiveSheet.QueryTables.Add(Connection:="TEXT;
I have a SSIS package that needs to enumerate Excel files in Sharepoint. When I setup the For Each loop container at the package level and then set up the file path in my Excel Source at the task level everything is fine. When I add my expression variable to the Excel Connection Manager properties I lose the connection and can't view my input table anymore. I have "Delay Validation" set to true on the Excel Connection Manager.
Has anybody experienced this?
EDIT:
The following are screenshots of my ForEach Loop Container configuration:
The following is a screenshot of where I am passing the variable. It is in the Expression box of the Excel Connection Manager:
Do you have a default value set for the variable to point to a sample file? Even though you have set the Delay validation to be True it just means that when the package runs it would not do a validation on the connection. It does not mean that while you are working on the package it would not use the variable to translate to the connection. So you should go ahead and set the default value of your connection variable to the path of the file you are working on.
I had set the DelayValidation at the connection level, at the dataflow level, and at the for-each loop in control flow level but none were working. I had spent 3-4 hours until I found this thread and setting a default value of the variable finally worked.
So setting a default value of the connection variable does solve the problem if you have a similar issue.
I'll explain the task requested from me:
I have two containers in Azure, one called "data" and one called "script". In the "data" container there's a txt file with data, and in the "script" container there's a script file.
Now, I need programatically (with WorkerRole) to execute the script file, with the content of the data file as parameters (Example: a script file that accepts a string 's' and returns to the screen "Hello, 's'", when 's' in the string given, and in the data file there's a string), and save the result of the run into another file which needs to be saved in another container called "result".
How do I do all these? I've already uploaded the files and created the blobs programatically, but I can't seem to understand how to execute the file of how to save its result to another file?
Can I please have some help?
Thanks in advance
Here are the steps in pseudo code:
Retrieve the script from the blob(using DownloadToStream())
Compile the script(I will leave this to you as I have no idea what
format your script is)
Load parameters from blob(same as step 1)
Execute script with those parameters.
If your script's can be written as lambda expressions then this becomes a lot easier as you can turn them into Action's
Edit based on your questiions:
DownloadText() is no longer included in Azure Storage 2.0, you only have access to DownloadToStream(). Even if you are using an older version(say 1.7) I would recommend using DownloadToStream() in the event you ever upgrade in the future. This will prevent having to refactor your code.
In terms of executing your script, depending on what type of script it is(if it is c# code you can use this example: Is it possible to dynamically compile and execute C# code fragments?. If you need to execute a different type of script you would need to run it using Process.Start and you can look at this example: http://www.dotnetperls.com/process-start
I do not have much experience with point number 2 but those are the processes I have heard and seen used.