Using PowerQuery I import data from a CSV that looks like this:
Report Title,,,,,
,Date,Type,USER_ID,PICKED_QTY,No of Hours
,31/10/2021,Type A,User_1,300,3
,31/10/2021,Type A,User_3,250,8
,01/11/2021,Type B,User_1,167,5
,01/11/2021,Type C,User_2,988,2
,02/11/2021,Type A,User_1,1113,4
Date,Type,USER_ID,PICKED_QTY,No of Hours,
03/11/2021,Type C,User_1,1500,5,
04/11/2021,Type A,User_1,200,8,
sometimes it looks like this (no second page) - which is where the problem is:
Report Title,,,,,
,Date,Type,USER_ID,PICKED_QTY,No of Hours
,31/10/2021,Type A,User_1,300,3
,31/10/2021,Type A,User_3,250,8
,01/11/2021,Type B,User_1,167,5
,01/11/2021,Type C,User_2,988,2
,02/11/2021,Type A,User_1,1113,4
I get the data into a readable format using this PQ (this source would be different, but references a table here for simplicity):
DataSource:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
SplitData = Table.SplitColumn(Source,"Column1", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv))
in
SplitData
I then use two queries to line the data up, so the Date columns are both in the same column, etc.
Query1:
let
Source = DataSource,
RemoveTopRows = Table.Skip(Source,1),
PromoteHeaders = Table.PromoteHeaders(RemoveTopRows, [PromoteAllScalars=true]),
FilterRows = Table.SelectRows(PromoteHeaders, each ([#""] = "")),
RemoveOtherColumns = Table.SelectColumns(FilterRows,{"Date", "Type", "USER_ID", "PICKED_QTY", "No of Hours"}),
ChangeType = Table.TransformColumnTypes(RemoveOtherColumns,{{"Date", type date}, {"Type", type text},
{"USER_ID", type text}, {"PICKED_QTY", Int64.Type},
{"No of Hours", Int64.Type}})
in
ChangeType
Query2:
let
Source = DataSource,
RemoveTopRows = Table.Skip(Source,1),
FilterRows = Table.SelectRows(RemoveTopRows, each ([Column1.1] <> "")),
PromoteHeaders = Table.PromoteHeaders(FilterRows, [PromoteAllScalars=true]),
RemoveOtherColumns = Table.SelectColumns(PromoteHeaders,{"Date", "Type", "USER_ID", "PICKED_QTY", "No of Hours"}),
FilterRows2 = Table.SelectRows(RemoveOtherColumns, each ([Date] <> "Date")),
ChangeType = Table.TransformColumnTypes(FilterRows2,{{"Date", type date}, {"Type", type text}, {"USER_ID", type text},
{"PICKED_QTY", Int64.Type}, {"No of Hours", Int64.Type}})
in
ChangeType
Finally, I join the previous two queries together and group to get my final table.
Query3:
let
Source = Query1,
AppendQueries = Table.Combine({Source, Query2}),
SortRows = Table.Sort(AppendQueries,{{"Date", Order.Ascending}}),
GroupRows = Table.Group(SortRows, {"Date", "Type"}, {{"Picked Qty", each List.Sum([PICKED_QTY]), type nullable number},
{"Total Hours", each List.Sum([No of Hours]), type nullable number}}),
AddDivision = Table.AddColumn(GroupRows, "Rate", each [Picked Qty] / [Total Hours], type number)
in
AddDivision
Question
Sometimes my raw data doesn't include a second page of data, so there's no need for Query2.
When this happens, if I don't manually add the headers for the second page I get an error: [Expression Error] The column 'Date' of the table wasn't found.
How do I avoid this? The error appears in Query2 with RemoveOtherColumns - without column headers it can't find the correct column, and in Query3 as it can't append a query that's returning an error.
Without re-writing all of it, you could just change the last line of Query2 to be
in try ChangeType otherwise Table.FromRecords({[Date = null, Type = null, USER_ID=null, PICKED_QTY=null, No of Hours = null]})
or
in try ChangeType otherwise Table.Skip(Table.FromRecords({[Date = null, Type = null, USER_ID=null, PICKED_QTY=null, No of Hours = null]}),1)
creating:
or just do the whole thing in one query
let Source = Csv.Document(File.Contents("C:\temp2\data.csv"),[Delimiter=",", Encoding=1252, QuoteStyle=QuoteStyle.None]),
#"Removed Top Rows" = Table.Skip(Source,1),
#"Filtered Rows" = Table.PromoteHeaders(Table.SelectRows(#"Removed Top Rows", each [Column1] = ""), [PromoteAllScalars=true]),
#"Filtered Rows2" = Table.PromoteHeaders(Table.SelectRows(#"Removed Top Rows", each [Column1] <> ""), [PromoteAllScalars=true]),
AppendQueries = Table.Combine({#"Filtered Rows",#"Filtered Rows2"}),
SortRows = Table.Sort(AppendQueries,{{"Date", Order.Ascending}}),
#"Changed Type1" = Table.TransformColumnTypes(SortRows,{{"PICKED_QTY", type number}, {"No of Hours", type number}}),
GroupRows = Table.Group(#"Changed Type1", {"Date", "Type"}, {{"Picked Qty", each List.Sum([PICKED_QTY]), type number}, {"Total Hours", each List.Sum([No of Hours]), type number}}),
AddDivision = Table.AddColumn(GroupRows, "Rate", each [Picked Qty] / [Total Hours], type number)
in AddDivision
Related
I wish to expand a table that contains data from an invoked column pulling data from the web.
The issue is that not every row finds a desired result on the web and returns an error. Although I don't mind the result failing for this row it causes an issue when trying to expand the table because the table relies on all rows having the same captured headers for the expansion.
Below is an image showing the errors and the result of the expansion.
M Code:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"No.", Int64.Type}, {"CAS Number", type text}, {"Chemical name", type text}}),
#"Invoked Custom Function" = Table.AddColumn(#"Changed Type", "Fx GetBriefProfileLink", each #"Fx GetBriefProfileLink"([CAS Number])),
#"Expanded Fx GetBriefProfileLink" = Table.ExpandTableColumn(#"Invoked Custom Function", "Fx GetBriefProfileLink", {"Name", "Cas Number", "exported-column-briefProfileLink"}, {"Name", "Cas Number.1", "exported-column-briefProfileLink"})
in
#"Expanded Fx GetBriefProfileLink"
Fx GetBriefProfileLink: Data Public
(CAsNumberorName as text) =>
let
Source = Excel.Workbook(Web.Contents("https://echa.europa.eu/search-for-chemicals?p_p_id=disssimplesearch_WAR_disssearchportlet&p_p_lifecycle=2&p_p_state=normal&p_p_mode=view&p_p_resource_id=exportResults&p_p_cacheability=cacheLevelPage&_disssimplesearch_WAR_disssearchportlet_sessionCriteriaId=dissSimpleSearchSessionParam101401654440118533&_disssimplesearch_WAR_disssearchportlet_formDate=1654440118558&_disssimplesearch_WAR_disssearchportlet_sskeywordKey="&CAsNumberorName&"&_disssimplesearch_WAR_disssearchportlet_orderByCol=relevance&_disssimplesearch_WAR_disssearchportlet_orderByType=asc&_disssimplesearch_WAR_disssearchportlet_exportType=xls"))[Data]{0},
#"Removed Top Rows" = Table.Skip(Source,2),
#"Promoted Headers" = Table.PromoteHeaders(#"Removed Top Rows", [PromoteAllScalars=true]),
#"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"Name", type text}, {"EC / List Number", type text}, {"Cas Number", type text}, {"Substance Information Page", type text}, {"exported-column-briefProfileLink", type text}, {"exported-column-obligationsLink", type text}}),
#"Removed Other Columns" = Table.SelectColumns(#"Changed Type",{"Name", "EC / List Number", "Cas Number", "exported-column-briefProfileLink"}),
#"Kept First Rows" = Table.FirstN(#"Removed Other Columns",1),
#"Removed Other Columns1" = Table.SelectColumns(#"Kept First Rows",{"Name", "Cas Number", "exported-column-briefProfileLink"})
in
#"Removed Other Columns1"
Sample Data:
No. CAS Number Chemical name
43 3380-30-1 5-chloro-2-(4-chlorphenoxy)phenol
44 03228-02-2 4-isopropyl-m-cresol
45 89-83-8 Thymol
46 60207-90-1 Propiconazole
47 5395-50-6 Tetrahydro-1,3,4,6-tetrakis(hydroxymethyl)imidazo[4,5-d]imidazole-2,5(1H,3H)-dione
48 15630-89-4 Sodium percarbonate
49 027176-87-0 Dodecylbenzenesulfonic acid
50 001344-09-8 Sodium silicate
It appears to be a glitch in PQ looking online however I am wondering if there are any workarounds.
My desired output is simply the same expansion but rows with errors just appear empty in the expanded section.
You just replace errors with null before your expansion.
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"No.", Int64.Type}, {"CAS Number", type text}, {"Chemical name", type text}, {"Column1", type text}}),
#"Merged Columns" = Table.CombineColumns(#"Changed Type",{"Chemical name", "Column1"},Combiner.CombineTextByDelimiter(" ", QuoteStyle.None),"Merged"),
#"Renamed Columns" = Table.RenameColumns(#"Merged Columns",{{"Merged", "Chemical name"}}),
#"Added Custom" = Table.AddColumn(#"Renamed Columns", "Custom", each #"Fx GetBriefProfileLink"([CAS Number])),
#"Replaced Errors" = Table.ReplaceErrorValues(#"Added Custom", {{"Custom", null}}),
#"Expanded Custom" = Table.ExpandTableColumn(#"Replaced Errors", "Custom", {"Name", "Cas Number", "exported-column-briefProfileLink"}, {"Custom.Name", "Custom.Cas Number", "Custom.exported-column-briefProfileLink"})
in
#"Expanded Custom"
I'm working on a simplification of some of my reports by using Power Query.
I have a table with this structure:
In the first column we can see the Sales Order Number.
In the second column we can see the Message type. If it is a XR or PR Message.
And in the last column we can find the Key for the status:
A = Active
B = Active
C = Closed
With this logic I would like to get this result table:
This way I can see as a cross table the type and the state per sales order.
How is this possible by using Power Query?
Are you sure about your results for Sales Order 103?
Try the following M code, amending where necessary (Source step, for example):
let
Source = Excel.CurrentWorkbook(){[Name = "Table1"]}[Content],
XRCol = Table.AddColumn(
Source,
"XR",
each Text.BeforeDelimiter([Message], "-") = "XR",
type logical
),
PRCol = Table.AddColumn(
XRCol,
"PR",
each Text.BeforeDelimiter([Message], "-") = "PR",
type logical
),
XRActCol = Table.AddColumn(PRCol, "XR Active", each [XR] and [Key] <> "C", type logical),
PRActCol = Table.AddColumn(XRActCol, "PR Active", each [PR] and [Key] <> "C", type logical),
RemoveCols = Table.RemoveColumns(PRActCol, {"Message", "Key"}),
Group = Table.Group(
RemoveCols,
{"Sales Order"},
{
{"XR", each List.Max([XR]), type logical},
{"PR", each List.Max([PR]), type logical},
{"XR Active", each List.Max([XR Active]), type logical},
{"PR Active", each List.Max([PR Active]), type logical}
}
)
in
Group
If I see it correctly your output PR and XR is mixed up in Sales order 103.
Try if this works for you:
let
Source = Excel.Workbook(File.Contents("C:\Users\maitting\Documents\Mappe1.xlsx"), null, true),
Tabelle1_Sheet = Source{[Item="Tabelle1",Kind="Sheet"]}[Data],
#"Promoted Headers" = Table.PromoteHeaders(Tabelle1_Sheet, [PromoteAllScalars=true]),
#"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"Sales Order", Int64.Type}, {"Message", type text}, {"Key", type text}}),
#"Split Column by Delimiter" = Table.SplitColumn(#"Changed Type", "Message", Splitter.SplitTextByDelimiter("-", QuoteStyle.Csv), {"Message"}),
#"Changed Type1" = Table.TransformColumnTypes(#"Split Column by Delimiter",{{"Message", type text}}),
#"Replaced Value" = Table.ReplaceValue(#"Changed Type1","A","Active",Replacer.ReplaceText,{"Key"}),
#"Replaced Value1" = Table.ReplaceValue(#"Replaced Value","B","Active",Replacer.ReplaceText,{"Key"}),
#"Replaced Value2" = Table.ReplaceValue(#"Replaced Value1","C","Closed",Replacer.ReplaceText,{"Key"}),
#"Merged Columns" = Table.CombineColumns(#"Replaced Value2",{"Message", "Key"},Combiner.CombineTextByDelimiter(" - ", QuoteStyle.None),"Merged"),
#"Unpivoted Columns" = Table.UnpivotOtherColumns(#"Merged Columns", {"Sales Order"}, "Attribute", "Value"),
#"Pivoted Column" = Table.Pivot(#"Unpivoted Columns", List.Distinct(#"Unpivoted Columns"[Value]), "Value", "Attribute", List.Count),
#"Added Conditional Column" = Table.AddColumn(#"Pivoted Column", "XR",
each if [#"XR - Active"] = 1 then 1
else if [#"XR - Closed"] = 1 then 1
else 0),
#"Added Conditional Column1" = Table.AddColumn(#"Added Conditional Column", "PR",
each if [#"PR - Active"] = 1 then 1
else if [#"PR - Closed"] = 1 then 1
else 0),
#"Removed Columns" = Table.RemoveColumns(#"Added Conditional Column1",{"XR - Closed", "PR - Closed"}),
#"Reordered Columns" = Table.ReorderColumns(#"Removed Columns",{"Sales Order", "XR", "PR", "XR - Active", "PR - Active"}),
#"Changed Type2" = Table.TransformColumnTypes(#"Reordered Columns",{{"XR", type logical}, {"PR", type logical}, {"XR - Active", type logical}, {"PR - Active", type logical}})
in
#"Changed Type2"
Output:
I've just added Sales order 104 with PR Closed to handle all possible cases.
My version
let Source = Excel.CurrentWorkbook(){[Name = "Table1"]}[Content],
RemoveEndingColumn = Table.TransformColumns(Source,{{"Message", each Text.BeforeDelimiter(_, "-"), type text}}),
#"Create MessageKey" = Table.AddColumn(RemoveEndingColumn, "MessageKey", each [Message]&" "&[Key]),
StackColumns = Table.RenameColumns(Table.SelectColumns(#"Create MessageKey",{"Sales Order", "Message"}),{{"Message", "Column"}})& Table.RenameColumns(Table.SelectColumns(#"Create MessageKey",{"Sales Order", "MessageKey"}),{{"MessageKey", "Column"}}),
#"Added Custom1" = Table.AddColumn(StackColumns, "TrueFalse", each true, type logical),
#"Pivoted Column" = Table.Pivot(#"Added Custom1", List.Distinct(#"Added Custom1"[Column]), "Column", "TrueFalse")
in #"Pivoted Column"
I have the following code:
let
Source = AnalysisServices.Databases("external\database", [TypedMeasureColumns=true, Implementation="2.0"]),
ProfiBI = Source{[Name="DB"]}[Data],
Profi1 = ProfiBI{[Id="DBFolder"]}[Data],
Profi2 = Profi1{[Id="DBFolder2"]}[Data],
#"Added Items" = Cube.Transform(Name2,
{
{Cube.AddAndExpandDimensionColumn, "[Calendar]", {"[Calendar].[Month Name].[Month Name]", "[Calendar].[Week No Retail].[Week No Retail]", "[Calendar].[Year No Retail].[Year No Retail]"}, {"Calendar.Month Name", "Calendar.Week No Retail", "Calendar.Year No Retail"}},
{Cube.AddAndExpandDimensionColumn, "[Merchandise Structure]", {"[Merchandise Structure].[Level 02 Desc EN].[Level 02 Desc EN]"}, {"Merchandise Structure.Level 02 Desc EN"}},
{Cube.AddAndExpandDimensionColumn, "[Sites]", {"[Sites].[Area Director].[Area Director]", "[Sites].[Opening Date].[Opening Date]", "[Sites].[Opening Year].[Opening Year]", "[Sites].[Regional Director].[Regional Director]", "[Sites].[Site Name].[Site Name]", "[Sites].[Site No].[Site No]", "[Sites].[Site Type 02].[Site Type 02]"}, {"Sites.Area Director", "Sites.Opening Date", "Sites.Opening Year", "Sites.Regional Director", "Sites.Site Name", "Sites.Site No", "Sites.Site Type 02"}},
{Cube.AddMeasureColumn, "Sales Net Amt AY", "[Measures].[Sales Net Amt AY]"}
}),
#"Filtered Rows" = Table.SelectRows(#"Added Items", each [Sites.Opening Year] = "2020"),
#"Filtered Rows1" = Table.SelectRows(#"Filtered Rows", each (Cube.AttributeMemberId([Calendar.Year No Retail]) = "[Calendar].[Year No Retail].&[2020]" meta [DisplayName = "2020"])),
#"Changed Type" = Table.TransformColumnTypes(#"Filtered Rows1",{{"Sites.Site No", Int64.Type}, {"Sites.Opening Year", type date}, {"Sites.Opening Date", type date}, {"Calendar.Year No Retail", type date}, {"Calendar.Week No Retail", Int64.Type}})
in
#"Changed Type"
When the query runs it will load 92785 rows, but the problem is I have to wait 5-7 minutes for that. Is there a way to load rows faster into my table? Also is checked Add to Data Model.
Everything works as it should be, but it takes too slow to load data. Data is retrieved from an OLAP Cube. I'm new with PowerQuery so please, I need help.
Thanks!
I have data in an excel table, generated by some software we use on site, the report is timestamped like this
6/05/2018 6:23:00 AM
As we are a 24 hour operation, I need PowerQuery to be able to recognise that the timestamps from 12:00:00 AM to 7:00:00 AM belong to the night shift of the previous day.
The problem I am facing is that although PowerQuery can handle Date/Time as a data type, it is seems to be truncating the time when importing data, so that the result for the above example within my query (and obviously in my output) is
6/05/2018 12:00:00 AM
Most of the stuff I can find on the net is about how to strip the time away - I want to keep it!!!
The purpose of this is so I can display records in chronological order for a night shift of production in a pivot table. At the moment I am having to add another column with the time alone, which causes data from midnight to 7:00am to preceed data from 7:00pm to midnight - when in fact it occurred after.
Cheers,
Mat
Edit: adding pictures of my problem, as I type I cannot see the images in the thread so I hope they are in the correct spots!
Example of my source data, timestamp is in the "Time" column.The other date columsn on the right are me getting something working so at least I have the shift date and the actual time together.
Source data
The following is my query, there is a lot in here that appends the source location to a predefined set of attributes, basically what I am struggling with is that the "Time" field gets imported, but loses all the data after the decimal point so I just get a date. I want to keep that time appended to that date, as well as have another field which is the shift date as described above.
let
// Removes unwanted characters.
CharsToRemove = List.Transform({33..45,47,58..126}, each Character.FromNumber(_)),
// The query is all based on the current month's portion of the current 13wk.
Source = Location_Data,
// Set some data fields, not all are changed here as it affects later calculations.
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Group Name", type text}, {"Name", type text}, {"Waste tonnes", type number}, {"Total Ore Tonnes", Int64.Type}, {"Dil cu_pct", type number}, {"Dil au", type number}, {"Dil ag", type number}, {"Dil fe_pct", type number}, {"Dil zn_pct", type number}, {"Density", type number}, {"Material", type text}, {"Type", type text}, {"Active from", Int64.Type}, {"Active to", Int64.Type}, {"Comments", type text}}),
// This steps add the MTD trucking data, and correlates it with our claim grades and density based on two fields, "Name" and "Material".
#"Merged Queries" = Table.NestedJoin(Source,{"Name", "Material"},LoadTrak_Data,{"Load Origin", "Material"},"NewColumn",JoinKind.LeftOuter),
// This step expands the trucking data so we can work with the each column individually, such as truck ID, Date/time, Load Volume, etc.
#"Expanded NewColumn" = Table.ExpandTableColumn(#"Merged Queries", "NewColumn", {"Record", "Time", "Dir.", "Operator", "Truck ID", "Load (m3)", "Truck Operator", "Crew", "Shift", "Material", "Load Origin", "Dumped At", "Day", "Shift Time", "Calc Shift"}, {"Record", "Time", "Dir.", "Operator", "Truck ID", "Load (m3)", "Truck Operator", "Crew", "Shift", "Material.1", "Load Origin", "Dumped At", "Day", "Shift Time", "Calc Shift"}),
#"Changed Type3" = Table.TransformColumnTypes(#"Expanded NewColumn",{{"Time", type number}}),
#"Sorted Record low to high" = Table.Sort(#"Changed Type3",{{"Record", Order.Ascending}}),
#"Added Error Volume Column" = Table.AddColumn(#"Sorted Record low to high", "Error Volume", each if [#"Load (m3)"] = null then "28.2" else null ),
#"Changed Error Volume to decimal number" = Table.TransformColumnTypes(#"Added Error Volume Column",{{"Error Volume", type number}}),
#"Added Custom1" = Table.AddColumn(#"Changed Error Volume to decimal number", "DMT", each if [#"Dir."] = null then null else if [#"Load (m3)"] = null then ([Error Volume]*[Density] * 0.7) else [#"Load (m3)"] * [Density] * 0.7),
#"Added Custom2" = Table.AddColumn(#"Added Custom1", "Level Loaded", each if [Load Origin] = "Empty" then 0 else (Text.Start([Load Origin],4))),
#"Creates shift date" = Table.AddColumn(#"Added Custom2", "Shift Date", each if [Shift Time] is null then null else if [Shift Time] < 0.2916 then [Time] -1 else [Time]),
#"Added Custom" = Table.AddColumn(#"Creates shift date", "Correct location", each if [#"Dir."] = null then null else if ([Shift Date]) < ([Active from]) or ([Shift Date]) > (([Active to])+0.999999) then "No" else "Yes"),
#"Changed Type1" = Table.TransformColumnTypes(#"Added Custom",{{"Level Loaded", type number}}),
#"Level Dumped" = Table.AddColumn(#"Changed Type1", "Level Dumped", each if [Dumped At] = "ROM" then 5270 else if [Dumped At] = "PAF" then 5170 else if [Dumped At] = "Paste" then 5270 else if [Dumped At] = "Waste" then 5270 else Text.Start([Dumped At], 4)),
#"Change Level Dumped to decimal number" = Table.TransformColumnTypes(#"Level Dumped",{{"Level Dumped", type number}}),
#"Added TKMs column" = Table.AddColumn(#"Change Level Dumped to decimal number", "TKMs", each if [Dumped At] = "Waste" then (((([Level Dumped] - [Level Loaded])*7)+300)/1000 * [DMT]) else if [Dumped At] = "ROM" then (((([Level Dumped] - [Level Loaded])*7)+150)+300)/1000 * [DMT] else if [Dumped At] = "PAF" then (((([Level Dumped] - [Level Loaded])*7)+150)+300)/1000 * [DMT] else if [Dumped At] = "Paste" then (((([Level Dumped] - [Level Loaded])*7)+150)+300)/1000 * [DMT] else (([Level Dumped] - [Level Loaded])*7)/1000 * [DMT]),
#"Changed TKMs to decimal number" = Table.TransformColumnTypes(#"Added TKMs column",{{"TKMs", type number}}),
#"Filtered Correct Location to remove incorrect duplicates" = Table.SelectRows(#"Changed TKMs to decimal number", each [Correct location] <> "No"),
#"Added load count helper column" = Table.AddColumn(#"Filtered Correct Location to remove incorrect duplicates", "Load", each if [Correct location] = "Yes" then 1 else ""),
#"Filtered Rows1" = Table.SelectRows(#"Added load count helper column", each true),
#"Filtered non-null Group Name rows" = Table.SelectRows(#"Filtered Rows1", each [Group Name] <> null and [Group Name] <> ""),
#"Converts date fields to type date" = Table.TransformColumnTypes(#"Filtered non-null Group Name rows",{{"Active from", type date}, {"Active to", type date}, {"Time", type date}, {"Shift Date", type date}}),
#"Merged with Sched_13wk" = Table.NestedJoin(#"Converts date fields to type date",{"Name", "Material"},Sched_13wk,{"Name", "Material"},"Sched_13wk",JoinKind.LeftOuter),
#"Expanded Sched_13wk" = Table.ExpandTableColumn(#"Merged with Sched_13wk", "Sched_13wk", {"Dil cu_pct", "Material", "Scheduled Tonnes"}, {"Sched_13wk.Dil cu_pct", "Sched_13wk.Material", "Sched_13wk.Scheduled Tonnes"}),
#"Changed Type2" = Table.TransformColumnTypes(#"Expanded Sched_13wk",{{"Shift Time", type time}, {"Day", type date}, {"Time", type date}})
in
#"Changed Type2"
Urgghhh that looks horrible, not sure how better to include it though. Lol the notes in there are to remind me what I am doing when I have to edit it. Still not finished, wanting to add a load more so that if I were to change position, the nest person sitting in my chair will have an idea of what is happening in the query!
And this is how the data is in the query output, see that all time values are gone.
Output with no time values.
Okay I really hope that helps. As you have guessed I am not a programmer by trade!
Report data which is copied into my excel workbook, with additional info typed in and then queried
Report generated by software which is copied and pasted into workbook containing query.
The problem may be resolved by using Power Query to import from the original text file. You should be able to resolve the timestamp issues, as well as more easily format the results.
Copy/Paste is frequently NOT the best way to handle this sort of problem.
My problem:
Through New Query -> From Other Sources -> From Web, I entered a static URL that allowed me to load approximately 60k "IDs" from a webpage in JSON format.
I believe each of these IDs corresponds to an item.
So they're all loaded and organised in a column, with one ID per line, inside a Query tab.
For the moment, no problem.
Now I need to import information from a dynamic URL that depends on the ID.
So I need to import from URL in this form:
http://www.example.com/xxx/xxxx/ID
This imports the following for each ID:
name of correspond item,
average price,
supply,
demand,
etc.
After research I came to the conclusion that I had to use the "Advanced Editor" inside the query editor to reference the ID query tab.
However I have no idea how to put together the static part with the ID, and how to repeat that over the 60k lines.
I tried this:
let
Source = Json.Document(Web.Contents("https://example.com/xx/xxxx/" & ID)),
name1 = Source[name]
in
name1
This returns an error.
I think it's because I can't add a string and a column.
Question: How do I reference the value of the cell I'm interested in and add it to my string ?
Question: Is what I'm doing viable?
Question: How is Excel going to handle loading 60k queries?
Each query is only a few words to import.
Question: Is it possible to load information from 60k different URLs with one query?
EDIT : thank you very much for answer Alexis, was very helpful. So to avoid copying what you posted I did it without the function (tell me what you think of it) :
let
Source = Json.Document(Web.Contents("https://example.com/all-ID.json")),
items1 = Source[items],
#"Converted to Table" = Table.FromList(items1, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Renamed Columns" = Table.RenameColumns(#"Converted to Table",{{"Column1", "ID"}}),
#"Inserted Merged Column" = Table.AddColumn(#"Renamed Columns", "URL", each Text.Combine({"http://example.com/api/item/", Text.From([ID], "fr-FR")}), type text),
#"Added Custom" = Table.AddColumn(#"Inserted Merged Column", "Item", each Json.Document(Web.Contents([URL]))),
#"Expanded Item" = Table.ExpandRecordColumn(#"Added Custom", "Item", {"name"}, {"Item.name"})
in
#"Expanded Item"
Now the problem I have is that it takes ages to load up all the information I need from all the URLs.
As it turns out it's possible to extract from multiple IDs at once using this format : http://example.com/api/item/ID1,ID2,ID3,ID4,...,IDN
I presume that trying to load from an URL containing all of the IDs at once would not work out because the URL would contain way too many characters to handle.
So to speed things up, what I'm trying to do now is concatenate every Nth row into one cell, for example with N=3 :
205
651
320165
63156
4645
31
6351
561
561
31
35
would become :
205, 651, 320165
63156, 4645, 31
6351, 561, 561
31, 35
The "Group by" functionnality doesn't seem to be what I'm looking for, and I'm not sure how to automatise that throught Power Query
EDIT 2
So after a lot of testing I found a solution, even though it might not be the most elegant and optimal :
I created an index with a 1 step
I created another costum column, I associated every N rows with an N increasing number
I used "Group By" -> "All Rows" to create a "Count" column
Created a costum column "[Count][ID]
Finally I excracted values from that column and put a "," separator
Here's the code for N = 10 000 :
let
Source = Json.Document(Web.Contents("https://example.com/items.json")),
items1 = Source[items],
#"Converted to Table" = Table.FromList(items1, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Renamed Columns" = Table.RenameColumns(#"Converted to Table",{{"Column1", "ID"}}),
#"Changed Type" = Table.TransformColumnTypes(#"Renamed Columns",{{"ID", Int64.Type}}),
#"Added Index" = Table.AddIndexColumn(#"Changed Type", "Index", 0, 1),
#"Added Conditional Column" = Table.AddColumn(#"Added Index", "Custom", each if Number.RoundDown([Index]/10000) = [Index]/10000 then [Index] else Number.IntegerDivide([Index],10000)*10000),
#"Reordered Columns" = Table.ReorderColumns(#"Added Conditional Column",{"Index", "ID", "Custom"}),
#"Grouped Rows" = Table.Group(#"Reordered Columns", {"Custom"}, {{"Count", each _, type table}}),
#"Added Custom" = Table.AddColumn(#"Grouped Rows", "Custom.1", each [Count][ID]),
#"Extracted Values" = Table.TransformColumns(#"Added Custom", {"Custom.1", each Text.Combine(List.Transform(_, Text.From), ","), type text})
in
#"Extracted Values"
I think what you want to do here is create a custom function that you invoke with each of your ID values.
Let me give a similar example that should point you in the right direction.
Let's say I have a table named ListIDs which looks like this:
ID
----
1
2
3
4
5
6
7
8
9
10
and for each ID I want to pull some information from Wikipedia (e.g. for ID = 6 I want to lookup https://en.wikipedia.org/wiki/6 and return the Cardinal, Ordinal, Factorization, and Divisors of 6).
To get this for just one ID value my query would look like this (using 6 again):
let
Source = Web.Page(Web.Contents("https://en.wikipedia.org/wiki/6")),
Data0 = Source{0}[Data],
#"Changed Type" = Table.TransformColumnTypes(Data0,{{"Column1", type text}, {"Column2", type text}, {"Column3", type text}}),
#"Filtered Rows" = Table.SelectRows(#"Changed Type", each ([Column2] = "Cardinal" or [Column2] = "Divisors" or [Column2] = "Factorization" or [Column2] = "Ordinal")),
#"Removed Columns" = Table.RemoveColumns(#"Filtered Rows",{"Column1"}),
#"Renamed Columns" = Table.RenameColumns(#"Removed Columns",{{"Column2", "Property"}, {"Column3", "Value"}}),
#"Pivoted Column" = Table.Pivot(#"Renamed Columns", List.Distinct(#"Renamed Columns"[Property]), "Property", "Value")
in
#"Pivoted Column"
Now we want to convert this into a function so that we can use it as many times as we want without creating a bunch of queries. (Note: I've named this query/function WikiLookUp as well.) To do this, change it to the following:
let
WikiLookUp = (ID as text) =>
let
Source = Web.Page(Web.Contents("https://en.wikipedia.org/wiki/" & ID)),
Data0 = Source{0}[Data],
#"Changed Type" = Table.TransformColumnTypes(Data0,{{"Column1", type text}, {"Column2", type text}, {"Column3", type text}}),
#"Filtered Rows" = Table.SelectRows(#"Changed Type", each ([Column2] = "Cardinal" or [Column2] = "Divisors" or [Column2] = "Factorization" or [Column2] = "Ordinal")),
#"Removed Columns" = Table.RemoveColumns(#"Filtered Rows",{"Column1"}),
#"Renamed Columns" = Table.RenameColumns(#"Removed Columns",{{"Column2", "Property"}, {"Column3", "Value"}}),
#"Pivoted Column" = Table.Pivot(#"Renamed Columns", List.Distinct(#"Renamed Columns"[Property]), "Property", "Value")
in
#"Pivoted Column"
in
WikiLookUp
Notice that all we did is wrap it in another set of let...in and defined the parameter ID = text which gets substituted into the Source line near the end. The function should appear like this:
Now we can go back to our table which we've imported into the query editor and invoke our newly created function in a custom column. (Note: Make sure you convert your ID values to text type first since they're being appended to a URL.)
Add a custom column with the following definition (or use the Invoke Custom Function button)
= WikiLookUp([ID])
Expand that column to bring in all the columns you want and you're done!
Here's what that query's M code looks like:
let
Source = Excel.CurrentWorkbook(){[Name="ListIDs"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"ID", type text}}),
#"Added Custom" = Table.AddColumn(#"Changed Type", "Custom", each WikiLookUp([ID])),
#"Expanded Custom" = Table.ExpandTableColumn(#"Added Custom", "Custom", {"Cardinal", "Ordinal", "Factorization", "Divisors"}, {"Cardinal", "Ordinal", "Factorization", "Divisors"})
in
#"Expanded Custom"
The query should look like this: