I wish to extract the data from this website:
https://apps.who.int/food-additives-contaminants-jecfa-database/search.aspx?fl=%2b
When I do so in Power query I only get the Kind, Name, Children, Text table.
However when doing the exact same on PowerBI this recognises the list as desired.
Can I get Power Query to recognise the data in the same way? Or is there a way to export the Query to Excel?
It seems that Power Query in Excel does not have the Html.Table function which is used in Power Query in Power BI.
But you can export the data as a csv, then import into Excel.
From the PQ Editor:
Close and Apply
Visualize all the columns
Click in the visualized area
At the bottom right, you will see an ellipsis
Click there and you will be able to select Export to CSV
Bit of a convoluted solution but I have subsequently found how to achieve this using Power Query in Excel. It is possible to navigate the HTML within power query to get to the raw data. Once here the data may not be adjacent to one another but this can be cleaned up easily using power query.
M Code:
let
Source = Web.Page(Web.Contents("https://apps.who.int/food-additives-contaminants-jecfa-database/search.aspx?fcc=4")),
Data0 = Source{0}[Data],
Children0 = Data0{0}[Children],
Children1 = Children0{1}[Children],
Children2 = Children1{2}[Children],
Children3 = Children2{2}[Children],
Children6 = Children3{6}[Children],
Children4 = Children6{6}[Children],
Children5 = Children4{1}[Children],
Children19 = Children5{19}[Children],
Children7 = Children19{1}[Children],
#"Expanded Children" = Table.ExpandTableColumn(Children7, "Children", {"Kind", "Name", "Children", "Text"}, {"Children.Kind", "Children.Name", "Children.Children", "Children.Text"}),
#"Expanded Children.Children" = Table.ExpandTableColumn(#"Expanded Children", "Children.Children", {"Kind", "Name", "Children", "Text"}, {"Children.Children.Kind", "Children.Children.Name", "Children.Children.Children", "Children.Children.Text"}),
#"Removed Columns" = Table.RemoveColumns(#"Expanded Children.Children",{"Kind", "Name", "Children.Kind", "Children.Name", "Children.Children.Kind", "Children.Children.Name", "Children.Children.Children", "Text"}),
#"Filled Up" = Table.FillUp(#"Removed Columns",{"Children.Text"}),
#"Filtered Rows" = Table.SelectRows(#"Filled Up", each ([Children.Children.Text] <> null and [Children.Children.Text] <> "Flavouring Agent"))
in
#"Filtered Rows"
Related
I'm trying to extract the data from another excel file, but I tried to make it dinamyc.
This is the context so I'm extractig data with a survey for different cities and the questions are the same but the data is different in each city, so I'm trying to create visualisations for each city but just replacing the data in the file, so this data is exported as a file we can call it "results-city.xlsx" and my goal is just placing this document with another with the same name and columns but obviusly different responses in each column so I'm trying to use power query and the advanced editor and this is my formula but is not getting success also the path will be dynamic that's the reason I included folder in the formula. Help please to achive this
let
Source = Excel.Workbook(File.Contents("C:\Users\iotal\OneDrive\Desktop\stack\folder\results-city.xlsx"), null, true),
Sheet1_Sheet = Source{[Item="Sheet1",Kind="Sheet"]}[Data],
#"Changed Type" = Table.TransformColumnTypes(Sheet1_Sheet,{{"Column1", type text}, {"Column2", type text}, {"Column3", type text}})
in
#"Changed Type"
Update 2:
Example of data:
File with data city 1
So here is an example with the data from two cities File with data city 2, when I tried to use the formula the power query don't import the data to excel.
My desired output is have the data imported in a sheet in excel that doesn't mind which of the two is in the folder but can be updated.
This is my desired output with images:
Image 1 is the first data that I will import
And the second image is when I replace the data for city2 should look like this just replacing the file for another one with the same name
And what is not working is the formula that doesn't import to a sheet the data as a table
When I replace the file from city1 for the one from city2 I got this error:
In powerquery, something like this will combine all tabs in all xlsx files in a specified hardcoded directory
let Source = Folder.Files("C:\subdirectory\directory"),
#"Filtered Rows" = Table.SelectRows(Source, each ([Extension] = ".xlsx")),
#"Removed Other Columns" = Table.SelectColumns(#"Filtered Rows",{"Name", "Content"}),
#"Added Custom" = Table.AddColumn(#"Removed Other Columns", "GetFileData", each Excel.Workbook([Content],true)),
#"Expanded GetFileData" = Table.ExpandTableColumn(#"Added Custom", "GetFileData", {"Data", "Hidden", "Item", "Kind", "Name"}, {"Data", "Hidden", "Item", "Kind", "Sheet"}),
#"Removed Columns" = Table.RemoveColumns(#"Expanded GetFileData",{"Content", "Hidden", "Item", "Kind"}),
List = List.Union(List.Transform(#"Removed Columns"[Data], each Table.ColumnNames(_))),
#"Expanded Data" = Table.ExpandTableColumn(#"Removed Columns", "Data", List,List)
in #"Expanded Data"
Alternatively, give a cell a range name, like path in excel and put your filepath in that cell, like C:\temp\a.xlsx
Then in powerquery, use that range name in place of hardcoding the file name like
let location= Excel.CurrentWorkbook(){[Name="path"]}[Content]{0}[Column1],
Source = Excel.Workbook(File.Contents(location), null, true),
...
this second alternative assumes that the tab name is constant. Otherwise I recommend the first method
You can combine the two alternatives if you want
To make the folder location dynamic with respect to a cell with named range "file_path" (B2, below/ here), modify an ordinary power query data import for one of the files/locations (here, "C:\temp\Folder1\File1.txt") as follows:
ordinary power query:
let
Source = Csv.Document(File.Contents("C:\temp\Folder1\File1.txt"),[Delimiter=" ", Columns=3, Encoding=1252, QuoteStyle=QuoteStyle.None]),
#"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),
in
#"Promoted Headers"
updated power query:
let
MyFolder = Excel.CurrentWorkbook(){[Name="file_path"]}[Content][Column1]{0},
Source = Csv.Document(File.Contents(MyFolder),[Delimiter=" ", Columns=3, Encoding=1252, QuoteStyle=QuoteStyle.None]),
#"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),
in
#"Promoted Headers"
i.e. file name XYZ in [Name ="XYZ"] replaced with variable MyFolder, defined as follows:
MyFolder = Excel.CurrentWorkbook(){[Name="file_path"]}[Content][Column1]
(optional)
Include VB code to refresh query every time the drop down list changes in the cell with named range ("file_path") (i.e. cell B2 above gif) - as follows:
(you don't have to do this - you could just as easily make the query refreshable periodically via the properties UI associated with the query, for instance)..
Private Sub Worksheet_Change(ByVal Target As Range)
If Not Intersect(Target, Range("file_path")) Is Nothing Then
Application.ScreenUpdating = False
Calculate
ActiveWorkbook.Connections("Query - dynamic_file").Refresh
Application.ScreenUpdating = True
Calculate
End If
End Sub
Kudos to Cristiano Galvão (here) for more details RE: dynamic links per above.
ta
Oi,
So i'm having issues with the List.Contains (x,x)=false function.
Context
I have 32 Excel files where i retrieve data from in a first query, this data gets filtered so only the columns needed (Each of those 32 Excel files is about 2MB) - This query then gets transformed into a "list" (ListofJustifWBS) so i only have the WBS's of that particular Query.
I also have another query, where i import a huge data excel file including WBS's - Actuals - Best estimates,...
What i want to do is : only keep the WBS's from the second Query that are not included in the first query.
The code i'm using is = Table.SelectRows(#"Changed Type", each (List.Contains(ListOfJustifWBS,[WBS])=false))
Whenever i run the query in the editor, the data get processed.. However, when i track the "progress" in the bottom right cornor i see all 32 excel file getting progressed, but excel sometimes "retrieved" (?) 20MB worth of data in each excel file while the excel file itself is only 2MB?
Whenever i try to run the query in an Excel Sheet tabl, Excel goes "Not responding".
Any idea how to fix this?
If you replace
= Table.SelectRows(#"Changed Type", each (List.Contains(ListOfJustifWBS,[WBS])=false))
with
#"Merged Queries" = Table.NestedJoin(#"Changed Type", {"WBS"}, Table.FromList(ListOfJustifWBS ), {"Column1"}, "ListOfJustifWBS", JoinKind.LeftOuter),
#"Expanded ListOfJustifWBS" = Table.ExpandTableColumn(#"Merged Queries", "ListOfJustifWBS", {"Column1"}, {"Column1.1"}),
#"Filtered Rows" = Table.SelectRows(#"Expanded ListOfJustifWBS", each ([Column1.1] = null))
is this any faster ?
I have a REST endpoint that queries the data in JIRA and returns the data for activeSprint and previousSprint so the user can build a burndown chart, similar to this:
{
activeSprint: ['..burndown data array..'],
previousSprint: ['..burndown data array..']
}
So in power query I setup the first query which is only representing the connection-only to the source
JSON
let
Source = Json.Document(Web.Contents("https://.../burndown"))
in
Source
Then I make a reference of the query called JSON and create a new query called Active Sprint
let
Source = JSON,
activeSprint = Source[activeSprint],
burndownData = activeSprint[burndownData],
#"Converted to Table" = Table.FromList(burndownData, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"date", "dateFormatted", "storyPoints", "issueCount", "issues", "plannedStoryPoints"}, {"date", "dateFormatted", "storyPoints", "issueCount", "issues", "plannedStoryPoints"}),
#"Changed Type" = Table.TransformColumnTypes(#"Expanded Column1",{{"dateFormatted", type datetime}, {"date", type datetime}})
in
#"Changed Type"
then I do the same as per above for the Previous Sprint query.
I group all the queries together in XLS into one group called TEST1 so it looks like this:
When I right click on the group Test1 and click refresh, it makes 2 requests to the API. Can I make it just one call since the source is the same for both ??
The dependency in the Power Query Editor is correct, but I don't get it why it sends 2 separate requests ??
Thanks
I've recently run into an issue which hopefully is solvable.
Currently, I have power query pointing at a folder containing several CSV files. This is normally no issue, however, in this instance not all of the files have the same columns.
Is there a way to have power query return every unique column found in the folder populating empty data observations with null values?
Assume that my folder has csv files similar to the following (note that the rows are indexed using letters for easy reference):
I would like my final table to look something like:
This seems like it should be pretty simple, but I can't figure it out for the life of me! Any help would be greatly appreciated!
Assuming you're using Folder.Files, I think you can:
Grab the Content column of the table returned by Folder.Files -- which should give you a list of binary values.
Parse each item in the list as a CSV document using List.Transform and Csv.Document -- which should give you a list of tables.
Then merge your list of tables with Table.Combine -- which should give you one single table. Table.Combine should take care of the details (like aligning column names).
You've not provided any code in your question, so it's hard to give a relevant example, but I think the code below gives me your expected output.
I've turned the row indexes into an ID column, just to make the final table easier to verify/follow.
let
firstCsv =
"ID,one,two,three
A,1,4,7
B,2,5,8
C,3,6,9",
secondCsv =
"ID,one,two,three,four
D,1,6,11,16
E,2,7,12,17
F,3,8,13,18
G,4,9,14,19
H,5,10,15,20",
thirdCsv =
"ID,one,two,yes,no,maybe
I,1,1,1,1,1
J,2,2,2,2,2
K,3,3,3,3,3
L,4,4,4,4,4
M,5,5,5,5,5",
// For example's sake, let's suppose that the contrived table below was
// returned by calling Folder.Files
filesInFolder = Table.FromColumns({
List.Transform({firstCsv, secondCsv, thirdCsv}, Text.ToBinary),
List.Transform({"1".."3"}, each "CSV file " & _ & ".csv"),
List.Repeat({"someFolderPath"}, 3)
}, type table [Content = binary, Name = text, Folder = text]),
parsed = List.Transform(filesInFolder[Content], each
let
csv = Csv.Document(_, [Delimiter = ",", QuoteStyle = QuoteStyle.Csv]),
promoted = Table.PromoteHeaders(csv, [PromoteAllScalars = true])
in promoted
),
// The step below should match the expected output in your question.
combined = Table.Combine(parsed)
in
combined
Obviously, you'll need to adjust for your own folder path and actually call Folder.Files as you presumably already are in your own code.
I've always used something like this
//read all files in specified directory you fill in here
let Source = Folder.Files("C:\directory\subdirectory"),
//filter only csv files
#"Filtered Rows" = Table.SelectRows(Source, each ([Extension] = ".csv")),
//Pull contents of each file into table with an index
#"Added Custom1" = Table.AddColumn(#"Filtered Rows", "Custom", each Table.AddIndexColumn(Csv.Document(File.Contents([Folder Path]&"\"&[Name]),[Delimiter=",", Encoding=1252, QuoteStyle=QuoteStyle.None]),"Index")),
#"Expanded Custom" = Table.ExpandTableColumn(#"Added Custom1", "Custom", {"Column1", "Index"}, {"Column1", "Index"}),
#"Removed Other Columns" = Table.SelectColumns(#"Expanded Custom",{"Column1", "Index", "Name"}),
#"Pivoted Column" = Table.Pivot(#"Removed Other Columns", List.Distinct(#"Removed Other Columns"[Name]), "Name", "Column1"),
#"Removed Columns" = Table.RemoveColumns(#"Pivoted Column",{"Index"})
in #"Removed Columns"
I recently switched to PowerQuery to fetch data from various sources. I have loaded my existing data to a table called "masterEntries".
The query I have calls a function to check the last record for each source in "masterEntries" and fetches only newer records.
let
Source = Excel.CurrentWorkbook(){[Name="formsMaster"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"FormName", type text}, {"Form", type text}, {"LastEntry", Int64.Type}}),
#"Added Custom" = Table.AddColumn(#"Changed Type", "Custom", each formEntries([FormName],[LastEntry])),
#"Expanded Custom" = Table.ExpandTableColumn(#"Added Custom", "Custom", {"EntryId", "Field1", "Field2", "Field3", "Field5", "DateCreated"}, {"EntryId", "Field1", "Field2", "Field3", "Field5", "DateCreated"}),
#"Removed Columns" = Table.RemoveColumns(#"Expanded Custom",{"Form", "LastEntry"}),
in
#"Removed Columns"
This query loads the data to a new table. Instead I want to append the data to "masterEntries".
I am trying to do this with PowerQuery and not VBA. PowerQuery has Append Query feature where two or more queries/results can be combined to a new table.
Even a new query to append the resulting table from above query ("latestEntries") to existing table ("masterEntries") will do.
Any ideas on how it can be done with PowerQuery?
EDIT
My original data ("masterEntries") was loaded manually. It is a big table with 400K+ records. I can load it using a query if that is going to help.
Every run of "latestEntries" checks what records are already in "masterEntries" and fetches only newer entries from different sources.
The Append Query method in Power Query is just a connection. It does not append records permanently. That is, when "latestEntries" brings a new set of records, the "masterEntries" loses the records that were in the earlier run of "latestEntries".
This sounds a bit like a request for "incremental load". This is currently not supported by Power Query in Excel. The workaround is to go via a "linkback"-table like described here: http://ms-olap.blogspot.de/2015/05/incremental-data-loads-in-microsoft.html
If your linkback-table exceeds 1,1 Mio rows, you can use JSON-compression like described here: http://www.thebiccountant.com/2016/12/06/how-to-store-tables-longer-than-11-mio-rows-in-excel/
But be aware, that this costs performance.
Both methods "cost" performance, so this technique only makes sense, if you "save" repetitive execution of really heavy transformations (or long loads from the web).
You should add something like this, just change name of Your_Table into table you want to use:
#"Append Query" = Table.Combine({#"Removed Columns", Your_Table})
in
#"Append Query"
Assuming you have some kind of ID, and it is integer, here is the query for the masterEntries table (this is important!):
let
Source = Excel.CurrentWorkbook(){[Name="masterEntries"]}[Content],
Types = Table.TransformColumnTypes(Source,{{"ID", Int64.Type}, {"Value", type number}}),
//Assuming you have integer-type IDs.
//Otherwise you have to order and index records in a view, and query that view.
MaxID = List.Max(Types[ID]),
//if you have ordered index, List.Max() can be substituted with Table.LastN(Types, 1)[ID]{0}
//it may perform better.
TableFromDB = Excel.CurrentWorkbook(){[Name="source"]}[Content], //Replace with database table
GetNewRows = Table.SelectRows(TableFromDB, each [ID] > MaxID),
MergeTables = Table.Combine({Types, GetNewRows})
in
MergeTables