I have sensor data that storing in table storage and this data is used as the data source for Powerbi. The data contains 6 data for a DateTime hour. For example 50 for 2019-10-24 12:10 Am, 65 for 2019-10-24 12:20 Am etc
I want to show the graph as the following format where Y axis shows 10th mint, 20th mint to 50th mint of an hour
Is it possible to create such graph in PowerBi? I am new to PowerBi
I tried one and got the following result
The query is like
let
Source = AzureStorage.Tables("https://xxxxx.table.core.windows.net"),
AveragedResult1 = Source{[Name="AveragedResult"]}[Data],
#"Expanded Content" = Table.ExpandRecordColumn(AveragedResult1, "Content", {"DayID", "DeviceDayID", "Leq", "max", "min", "date"}, {"Content.DayID", "Content.DeviceDayID", "Content.Leq", "Content.max", "Content.min", "Content.date"}),
#"Filtered Rows" = Table.SelectRows(#"Expanded Content", each ([PartitionKey] = "2019-10-21" ))
in
#"Filtered Rows"
create a 10 min bin.. and then have that column as your y axis and hour for x axis.
Related
The Question (How do I make it faster)
I have been playing around with Power Query in Excel for over a year now but for the first time, I have a query that takes 20+ minutes to run.
I am sure there is something here I can learn!
While it does currently work I believe if it was well-written it would run much faster.
Data Structure
There are two databases here
Database of Company (Aka attendees) - About 400 rows
Company Title
Rita Book
Paige Turner
Dee End
etc
Database of Events - About 500 rows
An Event can have many Company (Attendees). The database exports this as a comma-separated list in the column [#"Export CSV - Company"]
Event Title
Export CSV - Company
Date
Year
Event 1
Rita Book, Dee End
1/1/2015
2015
Event 2
Paige Turner
2/1/2015
2015
Event 3
Dee End
3/1/2015
2015
Event 4
Rita Book, Paige Turner, Dee End
1/1/2016
2016
etc
...
...
...
Note that I also have a separate query called #"Company Event Count - 1 Years List" which is a list of all years that events have been run.
The Goal
For a visualization, I need to get the data into the following structure:
Company Title
2015
2016
etc
John Smith
10
20
...
Jane Doe
5
14
...
etc
...
...
...
The Code
I have done my best to comment on my code below. Feel free to ask any questions.
let
// This is a function. It was the only way I could figure out how to use [Company Title] from #"Keep only names column" and "currentColumnTitleYearStr" from the dynamically created columns in the same scope
count_table_year_company = (myTbl, yearStr, companyStr) =>
Table.RowCount(
Table.SelectRows(
myTbl,
each Text.Contains([#"Export CSV - Company"], companyStr)
)
),
Source = #"Company 1 - Loaded CSV From Folder", // Grab a list of all Company
#"Keep only names column" = Table.SelectColumns(Source,{"Company Title"}), // Keep only the [Company Title] field
// Dynamically create columns for each year. Example Columns: [Company Title], [2015], [2016], [2017], etc
#"Add Columns for each year" =
List.Accumulate(
#"Company Event Count - 1 Years List", // Get a table of all events
#"Keep only names column",
(state, currentColumnTitleYearStr) => Table.AddColumn(
state,
currentColumnTitleYearStr, // The Year becomes the column title and is also used in filters
let // I hoped that filting the table by Year at this point would mean it only has to do it once per column, instead of once per cell.
eventsThisYearTbl = Table.SelectRows(
#"Event 1 - Loaded CSV From Folder",
each ([Year] = Number.FromText(currentColumnTitleYearStr))
)
in(
// Finally for each cell, calculate the count of events. E.g How many events did 'John Smith' attend in 2015
each count_table_year_company(eventsThisYearTbl, currentColumnTitleYearStr, [Company Title]) //CompanyTitleVar
)
)
),
FinalStep = #"Add Columns for each year"
in
FinalStep
My Theries
I believe one of a few things may be making it slow
I am using "List.Accumulate(" to dynamically create a column for each year. While this does work I think it may be the wrong formula for the job. Especially because the state field which is like a running total of each cell must be a huge number.
I worry that I have an 'each' where I dont need it but I cant seem to remove any. Its my understanding that every 'each' is effectively a nested loop so removing one may have a dramatic impact on performance.
In Conclusion
While it does currently work I know there is something for me to learn here.
Thank you so much any guidance or suggested readings you can provide :)
Does this do what you want? Converts from left to right. If not please explain more clearly
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
SplitNames = Table.TransformColumns(Source,{{"Names", each Text.Split(_,", ")}}),
#"Expanded Names" = Table.ExpandListColumn(SplitNames, "Names"),
#"Removed Columns" = Table.RemoveColumns(#"Expanded Names",{"Event Title", "Date"}),
#"Added Custom" = Table.AddColumn(#"Removed Columns", "Count", each 1),
#"Pivoted Column" = Table.Pivot(Table.TransformColumnTypes(#"Added Custom", {{"Year", type text}}, "en-US"), List.Distinct(Table.TransformColumnTypes(#"Added Custom", {{"Year", type text}}, "en-US")[Year]), "Year", "Count", List.Sum)
in #"Pivoted Column"
this is my first time asking a question on stackoverflow, let's see how this goes!
I am trying to connect a scenario manager for different sizes of assets to their belonging cost timeseries so that I can calculate the cashflow of the asset configuration belonging to a certain scenario.
This is what (in short version) both my tables that need to be connected look like:
ScenarioManager:
Scenario
SG
HOB
Scenario1
SG280
HB200
Scenario2
SG320
HOB160
CostSheet:
Config
Subtype
2021
2022
2023
2024
SG280
SG
-500
-180
-250
-680
SG320
SG
-600
-700
-345
-880
HOB200
HOB
-300
-680
-500
-320
HOB160
HOB
-250
-300
-260
-700
I load the data to the Data Model in Excel and unpivoted SG and HOB in the Scenario Manager as Config and then connected Config from the ScenarioManager to Config in the CostSheet. However that way I can't see the data in a PivotTable the way I want to. To achieve the view I need I need to create a second CostSheet where the Years columns are unpivoted as well. However when I go forward like this the PivotTable doesn' show me the correct values, instead it shows the sum of all Configs for the specific year.
The table I want to create should look like this:
Filter: Scenario1
Config
Subtype
2021
2022
2023
2024
SG280
SG
-500
-180
-250
-680
HOB200
HOB
-300
-680
-500
-320
etc. for every scenario that I have.
I uploaded the original file with the data here
I hope you guys can help me and thank you in advance!
Best
Julia
In Power Query, once you have unpivoted the ScenarioManager table, you can merge it with the CostSheet table on the Config column. Here is the Power Query M code, where ScenarioManager and CostSheet are queries that import the sample tables given above:
let
Source = ScenarioManager,
#"Unpivoted Columns" = Table.UnpivotOtherColumns(Source, {"Scenario"}, "Attribute", "Value"),
#"Removed Columns" = Table.RemoveColumns(#"Unpivoted Columns",{"Attribute"}),
#"Merged Queries" = Table.NestedJoin(#"Removed Columns", {"Value"}, CostSheet, {"Config"}, "CostSheet", JoinKind.LeftOuter),
#"Expanded CostSheet" = Table.ExpandTableColumn(#"Merged Queries", "CostSheet", {"Subtype", "2021", "2022", "2023", "2024"}, {"Subtype", "2021", "2022", "2023", "2024"}),
#"Renamed Columns" = Table.RenameColumns(#"Expanded CostSheet",{{"Value", "Config"}})
in
#"Renamed Columns"
Then you can load this query as a table and filter it using the Scenario column.
If you prefer having a PivotTable, you can add a step to unpivot the table and then load it to a PivotTable instead:
...
#"Unpivoted Columns2" = Table.UnpivotOtherColumns(#"Renamed Columns", {"Scenario", "Config", "Subtype"}, "Year", "Value")
in
#"Unpivoted Columns2"
The values might be aggregated by count by default when applying this to your full dataset. Aggregating by sum will produce the desired outcome seeing as there are only single values to sum for each Config-Year pair.
I have a table in Power query, which besides other fields has the following key fields:
SKU | Year | Week | Customer | Transaction | Type | Value
As an example, some rows would be:
AB587 | 2019 | 12 | Tom | Purchase | Forecast |200
AB587 | 2019 | 12 | Tom | Sale | Forecast |15
AB587 |2019 | 11 | Tom | Stock | Actual |1455
This is a table with about 300,000 rows with all the SKUs and a couple of year's worth of transactions for all customers, and this gets into a very very useful pivot table that is used extensively. I now need to add something to the data to make the table even more useful.
I have the forecast for purchases and sales for the whole year along with the actuals of course and they follow the above pattern. I also have the stock for all the weeks but only the one in the past i.e. actuals only. I don't have the stock forecast, which is what I want to add. The calculation is as simple as:
Stock from previous week + Purchase forecast from this week - Sale forecast from this week
The end result which I am expecting is that there will now be more rows added which will have as an example:
AB587 |2019 | 12 | Tom | Stock | Forecast |1640
(I am using numbers from above to calculate)
This will now enable me not only to pivot Purchase and Sales but also stock levels which will be game changing.
I would love for anyone to help me with this in Power Query (I have tried a number of methods over weeks but have not cracked it)
To try and solve it myself:
I appended more rows essentially appending Week-1 data for all actual weeks from my source reducing potentially some calculation time. Then I pivoted my "Transaction column" leading to new columns i.e. Purchase, Sale, Stock and Stock-1, which made the Stock forecast calculation easy (that's what it appears to be).
The thing which I did not think about is: this is only good to calculate the first week stock forecast, but then there is no way that I know to use that just calculated stock forecast to calculate the next week's stock forecast.
Basically there is no way to save that stock forecast that I just calculated to be used for the next week's calculation.
I'm not clear on what you are asking when you say you say you want to use the "calculated stock forecast to calculate the next week's stock forecast". If you just want to generate the formula and result you gave as an example as a component of your dataset though, that is pretty simple.
Starting from this as a sample table of your data loaded into PQ that I'm calling "Data Table":
I create two reference queries based off it called StockForecast and CombinedDataTable
In the "StockForecast" query we will add three custom columns. Two are the CalcYear and CalcWeek columns that take "Stock Actual" records and increase the week by one. The third is a CalcValue column that takes "Sale Forecast" records and makes the value in those negative. The code in the editor looks like this:
Source = DataTable,
#"Added Custom" = Table.AddColumn(Source, "CalcYear", each
if [Transaction] = "Stock" and [Type] = "Actual" then
(if [Week] = 52 then [Year] + 1 else [Year])
else [Year]),
#"Added Custom1" = Table.AddColumn(#"Added Custom", "CalcWeek", each
if [Transaction] = "Stock" and [Type] = "Actual" then
(if [Week] = 52 then 1 else [Week] + 1)
else [Week]
),
#"Added Custom2" = Table.AddColumn(#"Added Custom1", "CalcValue", each
if [Transaction] = "Sale" and [Type] = "Forecast" then
[Value] * -1
else [Value]
),
Then you use the Group function and aggregate by Stock, Customer, CalcYear and CalcWeek, with a Sum on the CalcValue function. This gets the Stock Forecast value you are looking for. After that it's just a matter of adding a couple columns for identification and some cleanup.
#"Grouped Rows" = Table.Group(#"Added Custom2", {"Stock", "Customer", "CalcYear", "CalcWeek"}, {{"Value", each List.Sum([CalcValue]), type number}}),
#"Added Custom3" = Table.AddColumn(#"Grouped Rows", "Transaction", each "Stock"),
#"Added Custom4" = Table.AddColumn(#"Added Custom3", "Type", each "Forecast"),
#"Renamed Columns" = Table.RenameColumns(#"Added Custom4",{{"CalcYear", "Year"}, {"CalcWeek", "Week"}})
in
#"Renamed Columns"
Then end result of the data looks like this:
Then just go to the CombinedDataTable query, append the StockForecast query, and you have Stock Forecast values in your dataset.
I'm using PowerBI desktop and I'm creating a chart to display average sales per weekday:
My data is in the format below:
(sampled in Excel to remove sensitive information, added colors to facilitate visualization)
My problem is: since each day is broken in 24 rows (hours), my average is wrong by a factor of 24.
For example, if I select January-2019 in the slicer, which has five Tuesdays (weekday code: 2), I want to see on the bar number 2:
(sum of amount where weekday = 2) / 5
Instead, I'm calculating:
(sum of amount where weekday = 2) / (24 * 5)
I can think of some ways to get this right, but they involve custom columns or auxiliary tables. I'm sure there is a simpler answer using DAX and measures, but I'm still learning it.
How can I correctly calculate this?
Let's assume your table name is "Data". Create 3 DAX measures (not calculated columns):
Measure 1:
Total Amount = SUM(Data[Amount])
Measure 2:
Number of Days = DISTINCTCOUNT(Data[Date])
Measure 3:
Average Amount per Day = DIVIDE( [Total Amount], [Number of Days])
Drop the last measure into a chart, it should give you the expected result.
As I understand from your excel you are working with 3 different columns. You can better combine this to a datetime and let power-bi handle it.
Below m-language will do this for you:
let
Source = Excel.Workbook(File.Contents("C:\....\Test.xlsx"), null, true),
Sheet1_Sheet = Source{[Item="Sheet1",Kind="Sheet"]}[Data],
#"Promoted Headers" = Table.PromoteHeaders(Sheet1_Sheet, [PromoteAllScalars=true]),
#"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"date", type datetime}, {"hour", type time}, {"amount", type number}}),
#"Added Custom" = Table.AddColumn(#"Changed Type", "Date", each [date]+ Duration.FromText(Time.ToText([hour]))),
#"Removed Other Columns" = Table.SelectColumns(#"Added Custom",{"amount", "Date"}),
#"Filtered Rows" = Table.SelectRows(#"Removed Other Columns", each ([amount] <> 0))
in
#"Filtered Rows"
The trick is in the added column: #"Added Custom" = Table.AddColumn(#"Changed Type", "Date", each [date]+ Duration.FromText(Time.ToText([hour])))
Here I add the time to the date.
I also removed the empty (zero amount) rows, you do not need them.
I added the Date & weekday to the Axis so a user can now drill down from year, month, day to weekday.
Be aware you need to do the SUM of the amount, not the average.
I am trying to calculate number of days for particular year based on calendar table that i have created.
For Example: I have 3 columns.
Event, number of days and Date when this event started
Event DaysLost
Injury 30 25/12/2016
Injury 588 06/08/2012
Days in 2016 - 6
Days in 2017 - 24
For the second case:
Days in 2012 - 146
Days in 2013 - 365
Days in 2014 - 77
Now for above case there are only 6 days which need to be counted in 2016 and the rest of the days should automatically be counted in 2017. But i cannot figure out how to do it.
In my output i would like to put years in one column and days lost for year in front of that particular year.
I have a calendar table and i want sum of days to populate for a particular year.
I tried calculating it by getting end date, by adding number of days to First start date and then if days were more that remaining days in that year. subtract remaining days from total days and remaining days should move to next year. But i cannot figure out how to keep adding days for next years if days extends for many years and list them after words.
Sept 4, 2017
Please see the excel solution below
Excel solution of the problem
0) Importing the data from your Excel screenshot into Power BI results in this.
1) Create a new column in that table using the following formula for end date (to help with future formulas).
EndDate = Injuries[First Start Date] + Injuries[Days]
You stated that you have a calendar table, so you can skip to step 3
2) Create a new table by clicking on Modeling -> New Table and entering the following formula. This gives a single column table with a list of years.
Years = GENERATESERIES(2000, 2020, 1)
3) Create another new table using the following formula. This gives a table with all of the fields from the initial data table crossjoined with the Year table that was just created. The formula also filters the resulting table to only return rows where the value in the Year column is between the First Start Date and the First Start Date plus Days. To learn more about the CROSSJOIN function, check of the documentation here.
InjuriesByYear = FILTER(
CROSSJOIN(Years, Injuries),
Years[Year] >= Injuries[First Start Date].[Year] &&
Years[Year] <= Injuries[EndDate].[Year]
)
4) Create relationships from the InjuriesByYear table back to the initial data table and the Year table. This will help facilitate nicer reporting efforts.
5) In the InjuriesByYear table, create a new column by clicking on Modeling -> New Column and entering the following formula. The first IF checks if all of the days lost are in a single year. The second IF handles when the days are spread across multiple years, with the True clause handling the first year, and the False clause handling all other years.
DayPerYear = IF(
InjuriesByYear[Year] = InjuriesByYear[First Start Date].[Year] && InjuriesByYear[Year] = InjuriesByYear[EndDate].[Year], InjuriesByYear[Days],
IF(
InjuriesByYear[Year] = InjuriesByYear[First Start Date].[Year], DATEDIFF(InjuriesByYear[First Start Date], DATE(InjuriesByYear[First Start Date].[Year], 12, 31), DAY),
DATEDIFF(DATE(InjuriesByYear[Year], 1, 1), MIN(InjuriesByYear[EndDate], DATE(InjuriesByYear[Year], 12, 31)), DAY) + 1
)
)
6) To test it all out, create a pivot table as configured in below. Following these steps, the pivot table should match your Excel solution.
This is a Power Query based approach...
I started with this:
Then I added a custom column by clicking the Add Column tab and Custom Column button and completing the pop-up window like this:
...and clicking OK.
Then I changed the type for that new column by selecting it and then clicking the Transform tab and then Data Type and Date.
Then I added another custom column, completing the pop-up like this:
Then I added another custom column, completing the pop-up like this:
Then I added yet another custom column, completing the pop-up like this:
Then I expanded that last column I added by clicking on the at the top of the column and Expand to New Rows.
Then I added a final custom column, completing the pop-up like this:
Finally, I grouped by the Event, DaysLost, Started, and Year columns and summed the DaysLostForYear column by clicking the Transform tab and Group By button and completing the pop-up like this:
I end up with this:
You might want a different grouping, but this should get you close. It shows how many days were lost in the years associated with each instance of an injury's total days lost. For instance, the first injury, which was 30 days in duration, started on 12/25/2016: 7 of those days occurred in 2016 and 23 in 2017. The second injury was 588 days, started on 8/6/2012: 148 days were in 2012, 365 in 2013, and 75 in 2014.
Note that I count the started date as a lost day.
Note also that I account for leap years.
I hope this helps.
Here's the query code:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Event", type text}, {"DaysLost", Int64.Type}, {"Started", type date}}),
#"Added Custom" = Table.AddColumn(#"Changed Type", "Ended", each Date.AddDays([Started],[DaysLost]-1)),
#"Changed Type1" = Table.TransformColumnTypes(#"Added Custom",{{"Ended", type date}}),
#"Added Custom3" = Table.AddColumn(#"Changed Type1", "DaysYearStarted", each Number.From(Date.From(Text.From(Date.Year([Started]))&"/12/31")-[Started])+1),
#"Added Custom4" = Table.AddColumn(#"Added Custom3", "DaysYearEnded", each Number.From([Ended]-Date.From(Text.From(Date.Year([Ended])-1)&"/12/31"))),
#"Added Custom5" = Table.AddColumn(#"Added Custom4", "Year", each List.Numbers(Date.Year([Started]),Date.Year([Ended])-Date.Year([Started])+1)),
#"Expanded Custom" = Table.ExpandListColumn(#"Added Custom5", "Year"),
#"Added Custom1" = Table.AddColumn(#"Expanded Custom", "DaysLostForYear", each if [Year]=Date.Year([Started]) then [DaysYearStarted] else
if [Year]=Date.Year([Ended]) then [DaysYearEnded] else
if Date.IsLeapYear([Year]) then 366 else 365),
#"Grouped Rows" = Table.Group(#"Added Custom1", {"Event", "DaysLost", "Started", "Year"}, {{"DaysLostForYear", each List.Sum([DaysLostForYear]), type number}})
in
#"Grouped Rows"