I'm trying to make a dynamic oData URL. This works, but it applies the filter after downloading the data set. I need to use query folding, so it applies filter in the oData Query.
Can someone point me in right direction
let
// Get Parameters
Params = Excel.CurrentWorkbook(){[Name="tParams"]}[Content],
ItemNoValue = Params{0}[Value],
// Get D365 Data
Source = OData.Feed("https://xxxxxxxx&$filter=ItemNumber eq '11111111' &$select=dataAreaId,ItemNumber &$top=100", null, [Implementation="2.0"]),
#"Filter by Item Number" = Table.SelectRows(Source, each ([ItemNumber] = ItemNoValue))
in
#"Filter by Item Number"
What's also very strange is if I construct the URL in an excel cell, it STILL returns 30MB of data before displaying one row
let
// Get Parameters
UrlParam = Excel.CurrentWorkbook(){[Name="URL"]}[Content],
UrlValue = UrlParam[Column1]{0},
Source = OData.Feed(UrlValue, null, [Implementation="2.0"])
in
Source
Related
This is my first try at using the Power Query... I've build a "dynamic" query in which I can change the retrieved fields as well as the filtering fields and values to be used by the query.
It's working perfectly on my computer but as soon as I try to execute it on another computer, I get the "Please rebuild this data combination" error. I saw some post saying I'll have to kind of split my query but I have not been able to figure it out.
Here is what my 2 tables look like:
Condition and fields selection
and here is my Query with the error:
Query
This might not be very elegant, but it allow me, thru a VBA script, to generate the list of fields to be retrieved and to generate the condition to be used by the SQL.
Any idea why it's not working on the other computers or how to improved the solution I'm using?
Thank you!
Notes:
Hi, all my Privacy Level are already set to 'None'.
I've tried to parametrize my code but I can't figure how. The Where condition is dynamic: it could be Where Number = "1234" but in other condition, the where might be like: 'Where Assignee = "xyz"'.
Here is a simplified example of my code:
let
Source = Sql.Database("xxxx", "yyyy", [Query=
"Select network, testid
from CM3T1M1 "
& paramConditions[Conditions]{0} &
" "])
in
Source
rebuild query, Formula.Firewall
That's a feature to prevent prevent accidentally leaking data. You can change the privacy level to ignore it
See also: docs.microsoft/dataprivacyfirewall
Is the dynamic query inserting those cells into the SQL query ? Report Parameters are nice for letting the user change variables without having to re-edit the query.
Parameterized native SQL queries
from: https://blog.crossjoin.co.uk/2016/12/11/passing-parameters-to-sql-queries-with-value-nativequery-in-power-query-and-power-bi/
let
Source = Sql.Database("localhost", "Adventure Works DW"),
Test = Value.NativeQuery(
Source,
"SELECT * FROM DimDate
WHERE EnglishMonthName=#MonthName AND
EnglishDayNameOfWeek=#DayName",
[
MonthName = "March",
DayName = "Tuesday"
]
)
in
Test
Dynamic Power Query version of SQL Query
To dynamically generate this SQL Query
select NUMBER, REQUESTED_BY from SourceTable
where NUMBER = 404115
Table.SelectRows is your Where.
SelectColumns is your select
let
Source = ...,
filterByNum = 404115,
columnNames = {"NUMBER", "REQUESTED_BY"},
removedColumns = Table.SelectColumns(
Source, columnNames, MissingField.Error
),
// I used 'MissingField.Error' so you know right away
// if there's a typo or bug
// assuming you are comparing Source[NUMBER]
filteredTable = Table.SelectRows(
Source, each [NUMBER] = filterByNum
)
in
filteredTable
I'm not sure if even possible but the goal is to dynamically update a query based on the user selecting a date. I have a table in my Excel file while updates a value which feeds to PeriodString variable (below)
/*Parameter name = PeriodString */
let
Source = Excel.CurrentWorkbook(){[Name="PeriodString"]}[Content],
StrPeriod = Source[Value]{0}
in
StrPeriod
The part of the code I want to update is the [ ..months selected ].
=List.Sum({[FYOpening],[January],[February],[March],[April],[May]})
With the below variable
=List.Sum({PeriodStr})
I tried using Table.Column as I realize I have to convert the value to a list of selectable columns but I cant' get it to work.
=List.Sum({Table.Column(PeriodString{0},PeriodString[0])})
Expression.Error: We cannot convert the value "[FY Opening],[Januar..." to type List.
Details:
Value=[FY Opening],[January],[February],[March],[April],[May]
Type=[Type]
Let me know if possible / alternatives.
If you need exactly value like "[Col1],[Col2],[Col3]" for PeriodString, then use such code:
let
Source = #table({"a".."e"},{{1..5}, {6..10}}),
PeriodString = "[b],[d],[e]",
sum = Table.AddColumn(Source, "sum", each List.Sum(Expression.Evaluate("{"&PeriodString&"}", [_=_])))
in
sum
I'd prefer to use PQ list instead:
let
Source = #table({"a".."e"},{{1..5}, {6..10}}),
list = {"b","d","e"},
sum = Table.AddColumn(Source, "sum", each List.Sum(Record.ToList(Record.SelectFields(_, list))))
in
sum
I am using power query to load some json data in a table (matches). I want to use a specific part of that data (fixture_id) as a parameter for another json request in another query (predictions), and then combine that output in my main (matches) table. Anyone can point me in the right direction on how to do this ?
So here is my matches table:
And then in my fixtures table i can maybe i have:
apiKey = Excel.CurrentWorkbook(){[Name="ApiKey"]}[Content]{0}[Column1],
fixtureID = "?",
Source = Json.Document(Web.Contents("https://v2.api-football.com/predictions/" & fixtureID, [Headers=[#"X-RapidAPI-Key"=apiKey]])),
If i hardcode the fixtureID, i get this output:
But i want to calculate it dynamically, and then merge the output to the matches table.
The first step is to turn your request into a function that accepts parameters. Put your request on a new blank query:
let
fnGetData = (fixtureID as text) =>
let
apiKey = Excel.CurrentWorkbook(){[Name="ApiKey"]}[Content]{0}[Column1],
fixtureID = "?",
Source = Json.Document(Web.Contents("https://v2.api-football.com/predictions/"
& fixtureID, [Headers=[#"X-RapidAPI-Key"=apiKey]]))
in
Source
in
fnGetData
Rename it to fnGetData.
Then, go to your table and click on Add Column/Add Custom Function. Select fnGetData and the input parameter is your fixtureID column. This should make all the requests and you'll just have to expand the new column results.
I am importing financial data using JSON from the web into excel, but as the source uses pagination (giving 50 results per page I need to implement pagination in order to import all the results.
The data source is JSON:
https://localbitcoins.com//sell-bitcoins-online/VES/.json?page=1
or https://localbitcoins.com//sell-bitcoins-online/VES/.json?page=2
?page=1, ?page=2, ?page=3
I use the following code to implement pagination, but receive an error:
= (page as number) as table =>
let
Source = Json.Document(Web.Contents("https://localbitcoins.com//sell-bitcoins-online/VES/.json?page=" & Number.ToText(page) )),
Data1 = Source{1}[Data],
RemoveBottom = Table.RemoveLastN(Data1,3)
in
RemoveBottom
When I envoke a parameter (1 for page 1) to test it I get the following error and I can't seem to find out why?
An error occurred in the ‘GetData’ query. Expression.
Error: We cannot convert a value of type Record to type List.
Details:
Value=Record
Type=Type
For the record, I try to include page handling using ListGenerate:
= List.Generate( ()=>
[Result= try GetData(1) otherwise null, page = 1],
each [Result] <> null,
each [Result = try GetData([page]+1) otherwise null, Page = [Page]+1],
each [Result])
What is the default way to implement pagination using Power Query in MS Excel?
I realise you asked this nearly a month ago and may have since found an answer, but will respond anyway in case it helps someone else.
This line Data1 = Source{1}[Data] doesn't make sense to me, since I think Source will be a record and you can't use {1} positional lookup syntax with records.
The code below returns 7 pages for me. You may want to check if it's getting all the pages you need/expect.
let
getPageOfData = (pageNumber as number) =>
let
options = [
Query = [page = Number.ToText(pageNumber)]
],
url = "https://localbitcoins.com/sell-bitcoins-online/VES/.json",
response = Web.Contents(url, options),
deserialised = Json.Document(response)
in deserialised,
responses = List.Generate(
() => [page = 1, response = getPageOfData(page), lastPage = null],
each [lastPage] = null or [page] <= [lastPage],
each [
page = [page] + 1,
response = getPageOfData(page),
lastPage = if [lastPage] = null then if Record.HasFields(response[pagination], "next") then null else page else [lastPage]
],
each [response]
)
in
responses
In List.Generate, my selector only picks the [response] field to keep things simple. You could drill deeper into the data either within selector itself (e.g. each [response][data][ad_list]) or create a new step/expression and use List.Transform to do so.
After a certain amount of drilling down and transforming, you might see some data like:
but that depends on what you need the data to look like (and which columns you're interested in).
By the way, I used getPageOfData in the query above, but this particular API was including the URL for the next page in its responses. So pages 2 and thereafter could have just requested the URL in the response (rather than calling getPageOfData).
The API call looks like this:
https://data.edmonton.ca/resource/3pdp-qp95.json?house_number=10008&street_name=103%20STREET%20NW
and returns data in json:
[{"account_number":"3070208","garage":"N","house_number":"10008","latitude":"53.539158992619","longitude":"-113.497760691896","neighbourhood":"DOWNTOWN","street_name":"103 STREET NW","tax_class":"Non Residential","total_asmt":"1717000"}]
I have an excel table with specific house_number and street_name pairs and I want to capture the total_asmt column for each pair.
I've been able to create a power query which pulls the very first data point into a new sheet:
let
Parameter = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Removed Other Columns" = Table.SelectColumns(Parameter,{"house_number", "street_name"}),
X = #"Removed Other Columns"[house_number]{0},
Y = #"Removed Other Columns"[street_name]{0},
Source = Json.Document(Web.Contents("https://data.edmonton.ca/resource/3pdp-qp95.json?house_number="& X &"&street_name=" & Y)),
in
Source
I can't figure out how to iterate through all the value I have in X and Y or how to capture specific rows from the JSON data. Any help would be appreciated!
Thanks,
Aaleem
I think your best best is to not do it.
Why are you wasting your time scraping this data one address at a time when you could have the entire city's data in under a minute.
JSON: https://data.edmonton.ca/resource/3pdp-qp95.json
CSV: https://data.edmonton.ca/api/views/q7d6-ambg/rows.csv?accessType=DOWNLOAD
XML: https://data.edmonton.ca/api/views/q7d6-ambg/rows.xml?accessType=DOWNLOAD
...among others. Heck, they even have !
And when you're done with that one, they have a few hundred other interesting datasets.
The trick was to create a function inside powerquery, and then use the query as part of a table. Create the function as below and then under the data tab select your table using "From Table/Range" from there it is pretty straight forward.
let a_value= (x as number,y as text)=> //this creates the function
let //this is essentially the query I wanted with some minor changes from above
x_text = Number.ToText(x, "D", ""),
Source = Json.Document(Web.Contents("https://data.edmonton.ca/resource/3pdp-qp95.json?house_number="&x_text&"&street_name="&y)),
Source1 = Source{0},
total_asmt = Source1[total_asmt]
in
total_asmt
in a_value //closes the function