I have to import some Excel data on a regular basis. Checking the DB2 documentation one can directly access OLE DB datasources via an external function.
However I'm unable to set it up properly. I got the Microsoft Access Database Enginge 2010 plus the fix pack and installed it on the database server.
I placed the excel file in a local directory from the database server. (C:\Temp\test.xls)
The excel has a workbook called TEST1 and two rows ABC and DEF following some numeric data:
ABC | DEF
---------
1 | 5
2 | 6
3 | 7
4 | 8
For creating the table function I used the following statement:
CREATE OR REPLACE FUNCTION MYSCHEMA.test_excel ()
RETURNS TABLE(ABC INTEGER,
DEF INTEGER)
LANGUAGE OLEDB
EXTERNAL NAME '!TEST1!Provider=Microsoft.ACE.OLEDB.12.0;
Data Source=C:\Temp\test.xls;
Extended Properties="Excel 8.0;HDR=Yes"';
It seems to create that function. However when querying the data with:
SELECT * FROM TABLE(MYSCHEMA.test_excel()) AS FUNCTABLE;
I'm getting the following error:
User defined function "MYSCHEMA.TEST_EXCEL" received an OLE DB error from specified OLE DB provider. HRESULT="0x80040e37". Diagnostic text: "The Microsoft Access database engine".. SQLCODE=-1183, SQLSTATE=38506, DRIVER=3.53.71
According to the documentation the error means: 0x80040E37 The specified table does not exist.
The !TEST! should reference the workbook, however I'm unsure if it is correct syntax. How can one access a Excel worksheet from DB2? Is there a way to get a more detailed error message? Does anyone the correct naming scheme?
The naming seems to be incorrect. Looking at other oledb samples I figured a '$' was missing.
!TEST$! works as an external name when referencing a worksheet called TEST and I can access the data.
Related
There are N tables in the DB with the following data types:
Numeric, long text, date and time bigint, boolean.
All of them opens, except one
I'm opening a database
db = QSqlDatabase("QODBC")
db.setDatabaseName(r"DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=C:\Users\...\file.accdb")
db.open(username, password)
I output the tables contained in the db
db.tables()
Output:
["messages", "table1", "table2", ..., "tableN"]
And I'm trying to open the "messages" table
model = QSqlTableModel(db=db)
model.setTable("messages")
model.select()
Output:
False
Then I checked which other tables are not opening
for i in db.tables():
model.setTable(i)
if model.select() == False:
print(i)
Output:
"messages"
This means that the problem is only in this table.
But directly through MS Access the table opens
I have already tried to open it through the cycle. The keyword was found in db.tables(), but QSqlTableModel does not see the 'messages' table specifically.
I tried to change MS Access to the 2016 version. I thought, suddenly some certain type from MS Access 2019 conflicts with the old driver. It didn't help.
I was thinking of downloading a newer driver, but I didn't find one. I tried to dig into the registry... I didn't find anything either.
Please help
So, I figured out the problem by poking. I was initially right about the driver conflict with the bigint data type, but a few additional actions were missing.
Apparently, when you try to set the bigint data type and save it, Access warns you that because of it, databases may not support older versions, and you save anyway, it automatically sets the minimum supported version, or what?
Data separation helped, but before that you need to change the bigint data type to another data type. Database Tools->Move Data->Access Database.
english isn't my native tongue, but I hope I can explain my problem sufficiently.
I made a View in the Oracle DB which only contains the data I need.
Using SQL in my VBScript file, I select the View by using:
"SELECT * FROM TEST_1234"
I have selected the complete view now, that works fine.
Now I need to 'export' or copy the complete View to Excel using VBScript (via UFT [Unified Functional Testing]).
Is there an easy way to just copy the whole thing at once or at least complete rows or columns?
If 1. doesn't work, can I just 'iterate' through the rows and columns using two loops and copy the data from every field to the respective field in Excel?
It would be nice to be able to copy the Data without using the names of the columns in a recordset (is there a way to use numbers until EOC [End of columns]?), because there is a very high amount of columns to be copied and the column names are subject to change.
Thanks for any help!
From a programmer==code writer's point of you the most attractive solution is your very first approach (copy the whole thing with just one SQL statement). Depending on the providers' capabilities this statement could look like
INSERT INTO [DstTable] SELECT * FROM [SrcTable] IN '' 'odbc;dsn=DSNName'
or
SELECT * INTO [DstTable] FROM [SrcTable] IN '' 'odbc;dsn=DSNName'
Look here for a working solution that couldn't be simpler; but I admit that a dsnless connection to the destination database looks more complicated and your drivers may have other incantations to refer to the external Database. Furthermore, your pair of providers may not support an external connection from the source to the destination and the dirty trick of using the Access OLEDB driver (which came/still comes? with ADO) to connect to both Databases externally may not work for you. In all, it's certainly not easy to get "INSERT/SELECT INTO External Database" right. [Look at my (just downvoted) answer to see that people dispair and fall back (and upvote) code that uses single-item-copy-loops.] In your case, you'll have to research whether at least one of the Oracle providers available to you supports external connections to Excel (or vice versa).
From a programmer==hacker's point of view (let's get the job done with minimal fuss) an easy solution could be to export the views/tables to .csv (
I looked at this and was disappointed, but you may know much better) and to import them into Excel (just load .csv and save .xls)
If you can't/won't use the file system, you could go thru memory: Use GetRows to get the data into a two dimensional array and assign that to the desired Excel range.
If all the above fails and you need assignments to single cells in row and column loopings over the recordset, remember that the Fields collection gives you access to not only the data but the meta-info (number of columns, column-names, types, ...) too.
Thanks for the help, and the links you provided, Ekkehard and Bond! After reading them and trying a lot, i got a very simple solution.
Here's some working code, if anybody else faces the same or a similar problem:
Option explicit
Dim conn, rec, xlStat, xlStatW, dbCnnStr, SQLSec, statArt
Set conn = Createobject("ADODB.Connection")
Set rec = CreateObject("ADODB.Recordset")
Set xlStat = CreateObject("Excel.Application")
dbCnnStr = "[your DB-connection]"
conn.open dbCnnStr
'Start Excel XXX
Set xlStatW = xlStat.Workbooks.Add()
xlStatW.Sheets(1).Name = "AAA_123"
xlStatW.Sheets(2).Name = "BBB_123"
xlStatW.Sheets(3).Name = "CCC_123"
SQLSec = "SELECT * FROM XXX_123"
rec.open SQLSec,conn
xlStatW.Sheets(1).cells(2,1).CopyFromRecordset rec
rec.Close
SQLSec = "SELECT * FROM YYY_123"
rec.open SQLSec,conn
xlStatW.Sheets(2).cells(2,1).CopyFromRecordset rec
rec.Close
SQLSec = "SELECT * FROM ZZZ_123"
rec.open SQLSec,conn
xlStatW.Sheets(3).cells(2,1).CopyFromRecordset rec
rec.Close
xlStatW.SaveAs ("C:\test.xlsx")
xlStatW.Close
'Ende Excel XXX
conn.Close
OK, so my latest project requires loading an Excel 2007 spreadsheet into a SQL Server table. I'm working in SSIS 2008R2. Based on some stuff I found on the internet, I opened the Excel source in Advanced editor and changed the datatype of the long column to DT_NTEXT, so that it wouldn't truncate it. Then I made the database column VARCHAR(MAX). This runs correctly in debug mode on my laptop.
Then I deployed it to the development server and attempted to load the same test file. It failed with the following error messages:
Error: Code: 0xC0208265
Source: Main Data Flow Task Get Main Data [1]
Description: Failed to retrieve long data for column "DESCR".
End Error
Error: Code: 0xC020901C
Source: Main Data Flow Task Get Main Data [1]
Description: There was an error with output column "DESCR" (72) on output "Excel Source Output" (9). The column status returned was: "DBSTATUS_UNAVAILABLE".
End Error
Error: Code: 0xC0209029
Source: Main Data Flow Task Get Main Data [1]
Description: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "output column "DESCR" (72)" failed because error code 0xC0209071 occurred, and the error row disposition on "output column "DESCR" (72)" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.
End Error
Searching for information about the error, I found about a million sites offering the same three suggested solutions:
Add 'IMEX=1' to the extended properties of the connection string.
It was already there.
Change the TypeGuessRows key in the registry.
This was set to zero on the server, which I understand to mean that it should look at the entire file. Nevertheless, I changed it to 8 to match my laptop. The same error occurred when I ran it again. Then I changed it to 1,763, which is more than the number of rows in the spreadsheet. It still gave the same error. So, I put it back to zero. (There's a 1,900-character value in the first row of my test file, so it shouldn't really matter how many it checks, in this case.)
Change the datatype to DT_WSTR(4000) in the source.
The column is supposed to have up to 10,000 characters, so I'm not sure this would be a good idea even if it worked. However, I tried it anyway. This time it gave me a truncation error. I changed the truncation error disposition to "ignore failure" and it loaded the data, but truncated the value to 255 characters. I have verified that the length is 4000 and doesn't get changed when I save the file, but it's still truncating at 255 characters.
I have no idea what else to look at. Any help would be appreciated.
UPDATE 1/29: The package, without any changes, works correctly when running on the pre-production server. It still fails when running on the development server. Both servers have the same version of SSIS (including minor version numbers) as well as the same versions of Windows, Access and Excel. I do not know how to explain this, nor do I know how to tell if it would work in production.
I created a new package with similar non-functional requirements (Excel 2007 file, SSIS 2008, SQL Server 2008 R2, VARCHAR(MAX) target column) and it worked just fine after deployment into the database server. My package:
Metadata at the Excel Source component's output (checked using Advanced Editor): DT_NTEXT
Derived Column component between source and destination to cast to non-unicode from unicode using (DT_TEXT,1252)
Metadata at the OLE DB Destination component's input (checked using Advanced Editor): DT_TEXT
Target Column data type: VARCHAR(MAX)
I do not explicitly use the extended property IMEX in the connection
Executed by right-clicking on the package at the database server, and loaded a file with a few thousand characters per record into the table without truncation. Hope this helps
I have faced this issue while importing an excel file with a field containing more than 255 characters. I solved the issue using Python.
Simply, import the excel in a pandas data frame and then calculate the length of each of those string values per row.
Then, sort the dataframe in descending order. This will enable SSIS to allocate maximum space for that field as it scans the first 3 rows to allocate storage:
df = pd.read_excel(f,sheet_name=0,skiprows = 1)
df = df.drop(df.columns[[0]], axis = 1)
df['length'] = df['Item Description'].str.len()
df.sort_values('length', ascending=False, inplace=True)
writer = ExcelWriter('Clean/Cleaned_'+f[5:])
df.to_excel(writer,sheet_name='Billing',index=False)
writer.save()
ALL,
I am trying to store the value of two variables:
Variable1Value: 2014-05-10 00:00:00.000
Variable2Value: 2014-05-08 00:00:00.000
Into an Object type variable so that I loop it with a FOREACH LOOP in SSIS but, I don't know how to do it.
Normally with a OLE DB connection, I would create a variable of type object and stored my result set there, but with an ODBC connection, that's not the case, because the step always fails.
Can somebody help me.
Thank you
I would replace the Execute SQL Task with a Data Flow Task. Inside that, I would start with an ODBC Source component with your ODBC SQL statement. Then I would connect that to a Recordset Destination, and configure that for the Object type Variable.
This design also exposes the SSIS Data Types of the returned columns (e.g. in the Recordset Destination), avoiding guesswork when you come to use them downstream.
I installed Rocket Software for accessing an Unidata Db through SQL Server 2008. The idea is to write SQL Procedures for populating SQL Tables, but the problem I am getting is retrieving wrong filenames i. e. Select * from MyDb_Members. I got the field names as Member{Name, Phone{number. In my unidata core these fields are named as Member Name, Phone Number.
Do you know if there is way to run sql queries with those field names without getting sql query errors. It looks sql server does not like to use that name convention:
Select Member{Name from MyDb_Members
Error near '{'
Thanks for your help
Try formatting the query using NATIVE keyword. I don't use Unidata, but in UniVerse this works well. I have a lot of columns that contain periods and those are illegal column names in standard SQL.
{ NATIVE "Select * from MyDb_Members" }