I have one table in Excel 365 that I transform using a few queries to output as 2 pivot tables and 1 table. One of these queries is also loaded to the data model as I need to create measures.
The operation is taking extremely and erratically long - approximately 25 to 70 minutes just to load a query involving 14 columns and 100 rows. The same operation in Power BI consistently takes 2.5 minutes in comparison. After spending 2 weeks on diagnostics, I eventually isolate the problem to
Output query to pivot table and Load to data model.
Row-wise operations in query (if loaded to data model).
I had spent 2 week refactoring my code. This includes buffering what can be buffer and optimising every operation as best as I can. Also tried "Fast Data Load" but doesn't help.
Not loading the query to data model means everything only takes 2-3 minutes. But I cannot not load the query to the data model because I need measures.
Is there anything else I can do to fix this extremely long load to data model operation?
Related
I'm trying to create a column in a pivot table I've created that calculates the difference between a forecast and actual cost column. Under 'Fields, Items, & Sets', 'calculated field' is grayed out/unavailable. From other forum posts as far back as 2010 (all with dead links to websites meant to explain it more), I've heard that this has something to do with using data that comes from an OLAP table.
My 'actual cost' data is being pulled from a server and funneled into a group of tables on separate sheets. I also have a manual entry sheet for the forecasts (also type table) that's being connected to it. that I'm The cell count is in the millions and steadily increasing, so it's understandable that the system uses OLAP to refresh manually (it takes about a minute to 'refresh all'). That said, I don't know why this would prevent me from doing math with these columns. I've looked up some videos on OLAP and I figure it's just above my head. I've looked into converting it into a "dynamic range" per a recommendation from about 10 years ago, but there isn't much about what that actually is online and I'm not sure I'd be able to do that with how I'm pulling it from the database. That, and I'm doing about 2 weeks worth of operations that are probably dependent on it being a table. I'm not even sure if you can feed data into power pivot if it's not a table.
I have a way that I think I can do this, but it requires about a week's worth of work manually building out a massive, un-iterative sheet. It would pull all of the values for a specific job of each activity type, sum them, and then compile them into another table that can be used to find the difference between the values of forecast and actual. But, at that point I'm basically just making a worse pivot table infrastructure from scratch.
Is there a way that I can just side-step this issue? I've only been using power pivot for a few weeks and this is the first time I've had to do math between columns, so it may be that I'm missing something that most people know.
EDIT1: download file with 2 days of real data
My home automation controller collects data from several 4-in-1 motion sensors in different rooms of my house. The sensor prioritizes motion, sending motion reports every few seconds, but also independently reports temperature, humidity, and illuminance. I am trying to determine if the temp and humidity reports are sent frequently enough to automate control of heaters and exhaust fans.
Sensors independently report each category to the controller, which sends data to excel. Sample data below, but without motion reports that clutter up the real data.
A pivot table generated from the raw data:
Answering the question of frequency takes me several manual steps. Sorting/filtering the dataset for temp/humidity by room, then manually adding a time diff column
where time diff = (<current Date-Time cell> - <prev Date-Time cell>)*24*60. I then calculate the average and stdev of minutes between reports by manually selecting, in turn, each room/category subset in the time diff column; once for the average and once for the stdev.
After a few more manual steps, I end up with this desired result:
BUT I have to do it all over every time new data is added to the table. I'm certain excel can do this automatically, but I didn't find a solution through pivots, power pivots, slicing, or queries. I'm hoping one of you excel gurus can help. Thanks!
Im trying to build an excel sheet that calculates synthetic options prices and greeks for time series data to model intraday options pricing, input is simply intraday price data, say Tick level to 5 minute interval. I found this https://www.thebiccountant.com/2021/12/28/black-scholes-option-pricing-with-power-query-in-power-bi/ which provides for powerBI and Black Scholes but possibly not very accurately. I prefer the Binomial method (I have used this excellent tutuorial to build a manual version for a large number of strikes but it takes a long time to calculate and is very very complex and also inaccurate due to not being able to calculate many steps before topping excel out: https://www.macroption.com/binomial-option-pricing-excel/).
Does anyone have any idea if this is possible to create an entire column in Power Query that will calculate bionomially derived options pricing using >100 even up to 1000 steps? The reason is intraday pricing using high resolution data 5min, 1min, Seconds and Tick I think needs a large number of steps to properly converge. This is just about doing a good enough model that can be used for visualising the progress of a trade on a given day.
Any pointers on how this could be done and calculated using M Language would be much appreciated and useful!
I want to do some data analysis using Excel and this tasks I need to answer.
*data clustering
*margine
*seasonality/forecast
Can somebody give me an input maybe?
I am not coming from financial/statistical/analytical field, and these things are more less new to time, but I would like to learn how to do this in Excel.
Here is my data set, where time axis is non consecutive (for forecast prediction for instance).
I wanted to see if anyone can assist. I have been using power pivot a fair amount over the last 2 years. I have created a report within it which uses 9 columns measuring 3 metrics. (Essentially sales over the last 2 years with a variance).
The Pivot table pulling through uses no sub-totals and is in tabular form pulling from only the one source file which is an excel workbook.
The issue I am receiving when I pulled in the 6th to 9th column the query got notably slower. Now taking up to 15 minutes to simply add a filter.
Can anyone assist in making this data refresh at a normal rate? I have tried making the source file smaller, making it now so that there is only one source file and not 3 however this seems to have a minimal effect.
I have other reports with much larger amounts of data and more connections where I am not encountering this problem at all so this is a little beyond my comprehension.