I am having memory issue when I execute DAX query inside the code in ref picture. If its like 10000 row it works however more than that it create memory issue. My query may return up to 50 Millions of data.
Question 1: What should be the efficient way to execute the query.
Question 2: What settings or properties might change to adjust huge amount of data.
Question 3: Is that possible to use partition and split data to fill into data table?
I am new in python coding. Please suggest me if code needs to change or any other efficient way to pull the data and send to data frame? my end goal is to send all data to CSV format into data lake. Its currently working however, for smaller amount of rows. I have tested till 10k is working in few min. its super inefficient seems to me.
Thanks in Advance!
I'm not sure how I would automate it, but the Export Data function from DAX Studio works really fast. I just did 500k rows in under a minute. It works against tables in the model, so if you are working with a DAX Expression, you would have to create it as a DAX Table in the model first.
Related
I want to create a pretty big table in excel.
I have 4000 rows and 70 columns, I have images that are locked into cells as well.
Do I have a chance this table creation process will work, or will excel be stuck?
Is there a workaround maybe?
any help will be much appreciated.
Excel doesn't have a hard cap on it's rows / column count (well it does but it's astronomical), although once reaching certain thresholds, it might be preferable to move towards a database solution. Have you considered creating a small MySql server instead of using excel files?
For excel limitations see : https://support.office.com/en-us/article/data-model-specification-and-limits-19aa79f8-e6e8-45a8-9be2-b58778fd68ef
I am supposed to optimize the performance of an old Access DB in my company. It contains several tables with about 20 columns and 50000 rows. The speed is very slow, because the people work with the whole table and set the filters afterwards.
Now I want to compose a query to reduce the amount of data in Excel before transfering the complete rows, but the speed is still very slow.
First I tried the new power query editor from Excel. I first reduced the rows by selecting only the last few ones (by date). Then I made an inner join with the 2nd table.
Finally I got less than 20 rows returned, and I thought I was fine.
But when I started Excel to perform the query, it took 10 - 20 seconds to read the data. I could see, Excel loads the complete tables, before setting the filters.
My next try was to create the same query direcly inside the Access DB, same setting. Then I opened this query in Excel, and the time to load the rows is nearly zero. You select "refresh", and the result is shown instantly.
My question is: Is there any way to perform a query in Excel only (without touching the Access file), that is nearly as fast as a query in Access itself?
Best regards,
Stefan
Of course.
Just run an SQL query from MS Query in Excel. You can create the query in Access, and copy-paste the SQL in MS Query. They're executed by the same database engine, and should run at exactly the same speed.
See this support page on how to run queries using MS Query in Excel.
More complex solutions using VBA are available, but shouldn't be needed.
I have around 200,000 data in excel which is separated in per 15min for each day for two years. Now, I want to add for each day(including all the 15mins data at once) eg. 01/01/2014 to 12/31/2016. I did it using a basic formula (=sum(range)) but it is very time consuming. Can anyone help me find a easy way to resolve this problem?
It's faster and more reliable to work with big data sets using Ms Power Query in which you can perform data analitics and process with single queries, or using Power View of the dataset loaded into the datamodel that really works really fast.
I have an Excel that collects product data from an Oracle database. This process is executed once a day and one of its columns is the execution date.
The problem is that every time I run the query data is updated. What I want is to append the result below the existing data, so I can generate graphs showing some product information through the time.
How could I do that?
Thanks is advance!
Can you not just change the date in your query to a specific date that you would like to set as the begin date for your data pool? You would probably have to batch it occasionally as the file would probably get rather large, but just a suggestion if I'm understanding the question correctly. Otherwise, you could take #Dank advice and just copy the data into another worksheet to create a "master file" to populate your graphs.
Here is the situation we have:
a) I have an Access database / application that records a significant amount of data. Significant fields would be hours, # of sales, # of unreturned calls, etc
b) I have an Excel document that connects to the Access database and pulls data in to visualize it
As it stands now, the Excel file has a Refresh button that loads new data. The data is loaded into a large PivotTable. The main 'visual form' then uses VLOOKUP to get the results from the form, based on the related hours.
This operation is slow (~10 seconds) and seems to be redundant and inefficient.
Is there a better way to do this?
I am willing to go just about any route - just need directions.
Thanks in advance!
Update: I have confirmed (due to helpful comments/responses) that the problem is with the data loading itself. removing all the VLOOKUPs only took a second or two out of the load time. So, the questions stands as how I can rapidly and reliably get the data without so much time involvement (it loads around 3000 records into the PivotTables).
You need to find out if its the Pivot Table Refresh or the VLOOKUP thats taking the time.
(try removing the VLOOKUP to see how long it take just to do the Refresh).
If its the VLOOKUP you can usually speed that up.
(see http://www.decisionmodels.com/optspeede.htm for some hints)
If its the Pivot table Refresh then it depends on which method you are using to get the data (Microsoft Query, ADO/DAO, ...) and how much data you are transferring.
One way to speed this up is to minimize the amount of data you are reading into the pivot cache by reducing the number of columns and/or predefining a query to subset the rows.