I am trying to open a file that i exported from TOAD which has 1.4 million records. It has 4 columns.
I tried opening the file in notepad++, deleting half records there and copying them into other file and saving them. But when i open the 2 files in excel, all the columns are combined as one column.
Could someone give me a solution on how to divide the 1.4million records excel file into 2 files without messing up the columns or data.
If you are married to Excel, skip Toad completely and bring the data in directly with MS Query.
If this isn't SQL Server, you will need to set up an ODBC connection, and from there you can bind the query to the spreadsheet as follows:
From the "Data" tab, select "From Other Sources"
Pick SQL Server if it's SQL Server or MS Query if it's anything else
Skip all of the menus and paste in your SQL once you get to MS Query
Close MS Query, and your live query (and its results) will be dynamically linked to an Excel table (aka ListObject)
The great thing about this is when you want to refresh the query, right-click and refresh. Done.
Better still, you can set up the ODBC connection for your boss and he can do it himself.
Oh, and if you want this split into two datasets, change your SQL to pull the first half and the second half and have them each in a different worksheet. How you do this depends on your DBMS, which I'd encourage you to tag in your question.
Related
I am supposed to optimize the performance of an old Access DB in my company. It contains several tables with about 20 columns and 50000 rows. The speed is very slow, because the people work with the whole table and set the filters afterwards.
Now I want to compose a query to reduce the amount of data in Excel before transfering the complete rows, but the speed is still very slow.
First I tried the new power query editor from Excel. I first reduced the rows by selecting only the last few ones (by date). Then I made an inner join with the 2nd table.
Finally I got less than 20 rows returned, and I thought I was fine.
But when I started Excel to perform the query, it took 10 - 20 seconds to read the data. I could see, Excel loads the complete tables, before setting the filters.
My next try was to create the same query direcly inside the Access DB, same setting. Then I opened this query in Excel, and the time to load the rows is nearly zero. You select "refresh", and the result is shown instantly.
My question is: Is there any way to perform a query in Excel only (without touching the Access file), that is nearly as fast as a query in Access itself?
Best regards,
Stefan
Of course.
Just run an SQL query from MS Query in Excel. You can create the query in Access, and copy-paste the SQL in MS Query. They're executed by the same database engine, and should run at exactly the same speed.
See this support page on how to run queries using MS Query in Excel.
More complex solutions using VBA are available, but shouldn't be needed.
I have introduced several Excel files in the company, which use Power Query to consolidate different Excel files and prepare them accordingly. The structure is as follows:
1st file:
- Query from Access
- Querys from 3 Excel files
2.-5.Datei:
Various Excel files.
Some of these files are updated daily with new records.
That means I have to daily in the excel files with the queries and these update.
Is it possible to outsource this somehow?
Sharepoint, server?
What ideas do you have?
Except a VBA macro which opens and refreshes the data accordingly?
Best regards
If you have one file with Power Query that get data from multiple files with daily refreshed data without any Power Query in them then:
In file with Power Query when you add result of Power Query to sheet you will get connection.
You can customize this connection to refresh on workbook opening or every N minutes.
Is this is what you want ?
Here:
http://www.excel2013.info/power-query/automatic-update/
you can see in "Refresh the connection" section how to customize connection.
Refresh the connection
Because your data is in a table, your table is link to your source of data. When you will receive a new csv file with the same name, in the same folder, you just have to refresh the query to update your workbook or you can refresh automatically your table when you open your workbook. To do that, you have go to the menu Data > Connections
In the dialog box Workbook connections, you select one of the connections of your workbooks (here there is only one connection) and you click on Properties …
Select the option Refresh data when opening the file
Don’t forget to save and close your file.
If you have multiple files with Power Query and one file get data from another files results of Power Query then you can schedule some code to refresh connections. You can create Console App (C# language), add some library to work with Excel, get target files, open its workbook, refresh connections.
Like here:
https://social.msdn.microsoft.com/Forums/vstudio/en-US/b288098d-d4e8-4845-ae3f-38ad235e22aa/how-to-execute-a-quotrefresh-allquot-for-excel-programmatically-in-c?forum=csharpgeneral
Open Excel File, Refresh Query and Save C#
This way multiple files with Power Query will have always updated results of Power Query.
And your one main file with Power Query that get data from results of Power Query from this multiple files will have always updated data.
I have an excel sheet with a single workbook with data in it. The data is around 1000 rows and 50 columns. I need to import these data to an Oracle DB every week. Here comes the problem, the columns in the sheet belongs to different tables with some columns go in multiple tables. I use SQL Developer V.18.1.0.095. Thanks in advance for the help.
Note: I created a temp table and copied all data to it, then wrote the query to push each column to its respective tables. But, I feel its complex and think it won't work. Is there any better way.
PL/SQL Developer has special tool for tasks like this, calls ODBC Importer (Menu 'Tools'-> ODBC Importer).
For use it you have to set Excel File in USER / System DSN field and your domain user and password, and push Connect after.
After connection developer will ask you path of excel file, and after you can create table in heiborhood tab for your dataset.
Or, you can use sql loader. Ask google how to. It's easy.
I am using the Toad Automation Designer to export data of table to Excel.
Unfortunately my table contains more then 65000 rows (100k) and every single Excel file
can only contain 65k entries.
My workaround was to write two SQL statements to create two Excel files
select * from my_table offset 0 rows FETCH NEXT 65000 ROWS ONLY
select * from my_table offset 64999 rows
That is ok for the moment but I am looking for a more dynamic way to export the whole
table into several Excel files without writing multiple SQL statement because in future
I will have maybe 300k entries in my database table.
So I am looking for a possibility to write something like a little script in the
Automation Designer or something similar.
Any Ideas?
Had similar issue: Toad .xls download populates one tab (64K rows). Only difference is that my Toad (11) would then populate another tab with residual data (the next 64K rows), then another tab and so on until all data was downloaded... it sometimes took many tabs.
I changed the download file type to .xlsx and then all the data downloaded to one tab.
safety tip: After download, Excel will have no problem opening the downloaded xlsx - but I have to open/save/close the downloaded .xlsx in Excel prior to using an ms access import spec. (else access' import spec complains the file's unreadable).
Let's say I have an Access database, "ADB", and an Excel workbook, "EWB".
ADB has a table called "ATable" which contains columns including a column called "A_ID"
EWB has a worksheet called "EWorksheet" which contains columns "E_ID" and "ECol"
Now, I want to know how I can create an Excel worksheet that combines ATable.A_ID from ADB and EWorksheet.ECol from EWB, where if A_ID = E_ID then return ECol.
So in SQL, it should look something like this:
SELECT ATable.A_ID, EWorksheet.ECol
FROM ADB.ATable, EWB.EWorksheet
WHERE ATable.A_ID = EWorksheet.ECol
Of course I want the data to be dynamic, so that data will be updated when refreshed.
You need something called PowerQuery, in Excel 2016 comes by default. In previous versions you will have to install it.
Once you have it, it´s pretty straightforward. Select a connection as seen below selecting an Access database or an Excel workbook.
Once you have both queries loaded in PowerQuery you need to merge them through the column you specified, and that will make the join for you.
What PowerQuery does is record a set of steps such as connecting to an Access database or merging 2 tables. Those steps are saved within the file, so when you click RefreshAll in Excel it will reproduce those same steps you specified and bring you the latest data.