Here is the situation we have:
a) I have an Access database / application that records a significant amount of data. Significant fields would be hours, # of sales, # of unreturned calls, etc
b) I have an Excel document that connects to the Access database and pulls data in to visualize it
As it stands now, the Excel file has a Refresh button that loads new data. The data is loaded into a large PivotTable. The main 'visual form' then uses VLOOKUP to get the results from the form, based on the related hours.
This operation is slow (~10 seconds) and seems to be redundant and inefficient.
Is there a better way to do this?
I am willing to go just about any route - just need directions.
Thanks in advance!
Update: I have confirmed (due to helpful comments/responses) that the problem is with the data loading itself. removing all the VLOOKUPs only took a second or two out of the load time. So, the questions stands as how I can rapidly and reliably get the data without so much time involvement (it loads around 3000 records into the PivotTables).
You need to find out if its the Pivot Table Refresh or the VLOOKUP thats taking the time.
(try removing the VLOOKUP to see how long it take just to do the Refresh).
If its the VLOOKUP you can usually speed that up.
(see http://www.decisionmodels.com/optspeede.htm for some hints)
If its the Pivot table Refresh then it depends on which method you are using to get the data (Microsoft Query, ADO/DAO, ...) and how much data you are transferring.
One way to speed this up is to minimize the amount of data you are reading into the pivot cache by reducing the number of columns and/or predefining a query to subset the rows.
Related
I am having data which is reported in excel format and the same is created manually. There are lots of typos expected in the report as the person feeding the data in excel sheet is highly unskilled. The way the person listens to the data entry, as per his understanding the data is entered in excel eg Shree/Shri/Sri etc. sounds almost similar while they are pronounced but all 3 are different as the data as whole.
At present I am solving the problem by cleaning the data with Open Refine java based localhost solution using various clustering methods available in the software. The data is added to the excel pool in incremental manner hence the excel rows are increasing after each update. Open refine is consuming heavy resources as it works on localhost.
It will be really helpful if there is any other way round to solve the problem.
Thanks in advance!
I am wondering, if I make a pivottable in Excel from a recordset with about 50000 lines, it takes about 30 seconds to produce a running total in a date field. Yet when I want to achieve the same result in an Access table, the DSUM takes over 30 minutes. Same data... Why is there so much performance difference? What does Excel do in the background?
You might find this article helpful:
http://azlihassan.com/apps/articles/microsoft-access/queries/running-sum-total-count-average-in-a-query-using-a-correlated-subquery
Here's what it says about Dsum, DLookup, etc.
They involve VBA calls, Expression Service calls, and they waste
resources (opening additional connections to the data file.)
Particularly if JET must perform the operation on each row of a query,
this really bogs things down.
Alternatives include looping through the recordset in VBA or creating a subquery. If you need to use DSUM, make sure your field is indexed and avoid text fields.
I am supposed to optimize the performance of an old Access DB in my company. It contains several tables with about 20 columns and 50000 rows. The speed is very slow, because the people work with the whole table and set the filters afterwards.
Now I want to compose a query to reduce the amount of data in Excel before transfering the complete rows, but the speed is still very slow.
First I tried the new power query editor from Excel. I first reduced the rows by selecting only the last few ones (by date). Then I made an inner join with the 2nd table.
Finally I got less than 20 rows returned, and I thought I was fine.
But when I started Excel to perform the query, it took 10 - 20 seconds to read the data. I could see, Excel loads the complete tables, before setting the filters.
My next try was to create the same query direcly inside the Access DB, same setting. Then I opened this query in Excel, and the time to load the rows is nearly zero. You select "refresh", and the result is shown instantly.
My question is: Is there any way to perform a query in Excel only (without touching the Access file), that is nearly as fast as a query in Access itself?
Best regards,
Stefan
Of course.
Just run an SQL query from MS Query in Excel. You can create the query in Access, and copy-paste the SQL in MS Query. They're executed by the same database engine, and should run at exactly the same speed.
See this support page on how to run queries using MS Query in Excel.
More complex solutions using VBA are available, but shouldn't be needed.
High-Level Problem: Excel doesn't seem to be able to handle some of the more complex dashboards I am try to build, and I am hoping for some suggestions on how to alleviate this issue.
My job/requirements: I build executive and sales facing dashboards and reports for our organization in excel through an OLAP cube connection - the tool has been great for running quick analysis - but some of the larger projects that I run through this connection often crash or take painstakingly long to refresh. Here are the requirements I have, and why I am using excel:
Need to manipulate/alter our data freely - for example, if I am pulling customer data, where I have all 50 states in the columns, and all 10,000+ customers in the rows, I need to be able to run a formula adjacent to the data where I am comparing that customer/state data to company wide customer/state data.
End product (dashboard/table) needs to be visually pleasing as the end users are our sales team/executives.
Need to be able to re-run/refresh the same analysis easily. Most of the dashboards I run get sent out on a daily or weekly cadence, so each refresh of the dashboard needs to take no more than a few minutes, ideally seconds.
- Need built in time intelligence functionality: I need the dashboard I am building to know what day it is. Excel has the = today() formula.
Symptoms of issue: Right now, when I run more complex queries, data refreshes can take upwards of 15 minutes. Some of these files are approaching north of 200kb. For these problem files, further development is painfully slow and inefficient, because even the simplest of formulas (like a vlookup column off of the OLAP table) forces my computer to use all 8 processors and hogs about 95% of the 32gb of RAM.
Specifics of the analysis I am running: Much of this analysis requires pulling tens (and sometimes hundreds) of thousands of rows, and these tables are sometimes dozens of columns wide, as I might be pulling sales data by state or by month. Some dashboards often have half a dozen of these OLAP tables - each with its own purpose. There is no way to reduce the amount of data that I am pulling - I need all of the data to reach the conclusions I am trying to reach.
For example, one of the projects I am currently working on requires me to take 24 months of sales data, broken out by state (as columns, on the x axis of the OLAP table) and broken out by individual customer (over 10,000) on the rows. This calculation alone takes forever to process and refresh, and causes all other subsequent formulas to run extremely slow, even simple vlookup columns adjacent to the OLAP cube.
Research I have done: I have looked into alternatives to excel, but the largest issue I face is that I still have a set of requirements that need to be met, and the obvious tools at my disposal don't seem to meet those requirements at first glance. Again, the end product of what I am building must be easily consumable for our end users (sales reps and execs). If I were to run a sql query through Microsoft SQL Server to pull the data I am looking for, I would need to be able to refresh the data on a daily basis - but more importantly, I would need to be able to easily manipulate and build out extra calculations and formulas off of the query. Excel seems like it is the only tool that checks those boxes, and yet it also feels like a dead end.
I have around 200,000 data in excel which is separated in per 15min for each day for two years. Now, I want to add for each day(including all the 15mins data at once) eg. 01/01/2014 to 12/31/2016. I did it using a basic formula (=sum(range)) but it is very time consuming. Can anyone help me find a easy way to resolve this problem?
It's faster and more reliable to work with big data sets using Ms Power Query in which you can perform data analitics and process with single queries, or using Power View of the dataset loaded into the datamodel that really works really fast.