Problem Summary
I have a column in my Power Query table which contains a custom linked data type. Creating a custom linked data type filled with all null values is not desired. Instead, if all the values contained in the custom data type is null, I would like the value in the column to be null.
Background
I have a table which holds API response JSON text. This JSON text contains a list of search results (also in JSON), representing movies which match search criteria delivered in the request. There can be any number of search results, including zero. Using Power Query M, I parse these JSON texts with the built-in parser, which generates a list containing one record per search result. I then extract the first record in the list, expand that record into new columns, and combine those new columns into a custom data type.
Example
Here is an example query simulating only the problem area of my query. This example is fully contained and can be used to reproduce my issue exactly.
let
// These two variables holds the API response JSON text obtained from calls to Web.Contents().
// I've eliminated the actual calls in this example because that part of my query works fine.
Search_Fast_and_Furious_Response =
"{ ""total-results"":""2"", ""results"":[
{ ""title"":""Fast & Furious"", ""year"":""2009"" },
{ ""title"":""The Fast and the Furious"", ""year"":""2001"" } ] }",
Search_mmmmm_Response =
"{ ""total-results"":""0"", ""results"":[] }",
// Create the table to hold the response text.
Source = Table.FromRecords( { [#"API Response"=Search_Fast_and_Furious_Response],
[#"API Response"=Search_mmmmm_Response] }),
// Parse the JSON and put the output (a record) in a new column.
#"Insert Parsed JSON" = Table.AddColumn(Source, "JSON", each Json.Document([API Response])),
// Expand the record in the parsed JSON column. Each field in the record becomes a new column.
#"Expand JSON" = Table.ExpandRecordColumn(#"Insert Parsed JSON", "JSON",
{"total-results", "results"}, {"Result Count", "Results List"}),
// Add a new column to hold the first search result in the responses results list.
// This is also a record, like the parsed JSON two steps ago.
#"Add Result #1 Column" = Table.AddColumn(#"Expand JSON", "Result #1", each
try _[Results List]{0}
otherwise null), // In case the list is empty
// Expand the record in the Result #1 column.
#"Expand Result #1" = Table.ExpandRecordColumn(#"Add Result #1 Column", "Result #1",
{"title", "year"}, {"Title", "Year"}),
// Combine the newly expanded columns into a single column.
// Make the Display Name be the value in the Title field/column,
// and make the Type Name be "Excel.DataType."
// This is what creates the custom linked data type.
#"Combine Result #1" = Table.CombineColumnsToRecord(#"Expand Result #1", "Result #1",
{"Title", "Year"}, [ DisplayNameColumn = "Title", TypeName="Excel.DataType" ])
in
#"Combine Result #1"
The list in the very last line before the in statement, i.e. the fourth parameter to the Table.CombineColumnsToRecord function, allows the record to be used as a custom data type used with Excel's new linked data feature. I'm not certain, but I believe Power Query/Excel stores them as records with additional metadata, such as DisplayNameColumn and TypeName (the latter of which I'm sure is the most important part).
Problem and Goal
Here is the resulting table created by the example query. The bottom-right cell is selected. Its contents are shown at the bottom of the image. The cell itself contains a value, specifically a record with all values set to null. Because the Title field is null, the record's display text is "null."
This next picture shows my desired output. Notice again the bottom-right cell. This time, the cell is empty. It no longer contains a record with all values being null; now it contains nothing, so the display shown in this view is null, italicized so as to indicate a null value as opposed to the word "null." (Note: I've been unable to change the "null" cell in the first image to a literal null value, so to demonstrate, I simply added a new column of null values.)
Unfortunately, because of my otherwise clause after the try, the column "Result #1" may be null if the API returned zero search results. If this value is null in any row, then all of the new columns created by #"Expand Result #1" will contain null in that row, also. Finally, when all the null values are combined in the last step, I'm left with a record with all null values. Instead, what I hope to achieve is to have a single null value (of type null) in that cell.
Efforts So Far
I have tried the Table.ReplaceValues function, passing null as the new value and many different values as the old value (the one to be replaced), such as a new record with all null values. All those attempts have either been syntactically incorrect or resulted in expected and unwanted behavior. I have also tried using the "Replace Values" option in the Power Query GUI, but the same result occurs. In case ReplaceValues didn't like nulls, I've also tried using a different value in the otherwise clause, such as "N/A" of type text, then doing a ReplaceValues on that different value. This yielded the same result.
Conclusion
Is there any way I can replace a record—which is filled with null values and is stored in a column containing records—with a singular null value? The linked data type feature is a high priority in this situation, so I would prefer a solution that retains that feature (though of course all solutions are welcome).
I have "solved" my problem. While not technically a solution to the question I posted, I've achieved the desired result using a workaround.
Instead of dealing with the object full of null fields, I ensure that object is not converted to the custom object to begin with. I achieve this by moving all records with a null value after extracting the first List item in the Results List column; this is done before I expand that extracted item. After putting the nulls in a new table (which I call the Null Table), I delete those nulls from the first table (which I call the Non-Null Table). I perform the regular operations on the Non-Null Table to create the custom linked data type for only those rows that were not null. Afterward, I merge the two tables together again.
The full code containing the solution with my representative example is below, with new steps "highlighted" with non-indented comments.
let
// These two variables holds the API response JSON text obtained from calls to Web.Contents().
// I've eliminated the actual calls in this example because that part of my query works fine.
Search_Fast_and_Furious_Response =
"{ ""total-results"":""2"", ""results"":[
{ ""title"":""Fast & Furious"", ""year"":""2009"" },
{ ""title"":""The Fast and the Furious"", ""year"":""2001"" } ] }",
Search_mmmmm_Response =
"{ ""total-results"":""0"", ""results"":[] }",
// Create the table to hold the response text.
Source = Table.FromRecords( { [#"API Response"=Search_Fast_and_Furious_Response],
[#"API Response"=Search_mmmmm_Response] }),
// Parse the JSON and put the output (a record) in a new column.
#"Insert Parsed JSON" = Table.AddColumn(Source, "JSON", each Json.Document([API Response])),
// Expand the record in the parsed JSON column. Each field in the record becomes a new column.
#"Expand JSON" = Table.ExpandRecordColumn(#"Insert Parsed JSON", "JSON",
{"total-results", "results"}, {"Result Count", "Results List"}),
// Add a new column to hold the first search result in the responses results list.
// This is also a record, like the parsed JSON two steps ago.
#"Add Result #1 Column" = Table.AddColumn(#"Expand JSON", "Result #1", each
try _[Results List]{0}
otherwise null), // In case the list is empty
// New step
// Filter down to only rows with null in the new column. Save this new table for later.
#"Filter In Null" = Table.SelectRows(#"Add Result #1 Column", each _[#"Result #1"] = null),
// New step
// Filter down to only rows with NOT null in the new column.
#"Filter Out Null" = Table.SelectRows(#"Add Result #1 Column", each _[#"Result #1"] <> null),
// Expand the record in the Result #1 column.
#"Expand Result #1" = Table.ExpandRecordColumn(#"Filter Out Null", "Result #1",
{"title", "year"}, {"Title", "Year"}),
// Combine the newly expanded columns into a single column.
// Make the Display Name be the value in the Title field/column,
// and make the Type Name be "Excel.DataType."
// This is what creates the custom linked data type.
#"Combine Result #1" = Table.CombineColumnsToRecord(#"Expand Result #1", "Result #1",
{"Title", "Year"}, [ DisplayNameColumn = "Title", TypeName="Excel.DataType" ]),
// New step
// Convert the Null Table into a list of records.
#"Convert Table" = Table.ToRecords(#"Filter In Null"),
// New step
// Append the Null Table from earlier to the main table.
#"Combine Tables" = Table.InsertRows(#"Combine Result #1", Table.RowCount(#"Combine Result #1"),
#"Convert Table")
in
#"Combine Tables"
Related
I'm trying to send a LookUp activity output values as part of a body parameter in a POST request using LogicApp, which uses three parameters: "to", "email_body", "subject".
The LookUp activity depends on a query, and it may return from 2 rows up to 10 rows.
According to Azure, the output of the activity should look like this:
{
"count": 2,
"value": [
{
"column1":value1,
"column2":value2,
"column3":value3
},
{
"column1":value4,
"column2":value5,
"column3":value6
}
]
}
In this case, the query returned 2 rows, but how can I attach every output value to the POST body without having to use #activity('lookup_act').output.value[0].column1 and so on for every value?
The POST body is the following:
{
"email_body": "Hi, the following tables have been updated:
#{activity('lookup_act').output.value[0].column1}
#{activity('lookup_act').output.value[1].column1}",
"subject": "Update on tables",
"to": "email#domain.com"
}
I've tried using #activity('lookup_act').output.value to bring every value but it won't work.
Is there a way to call every single output value? If so, how can it be done and paste into a table?
Thanks beforehand.
There are two ways to get all values in mail:
1. Get whole lookup output array in mail.
First get the results from Lookup activity and then pass the output of this activity by converting it into a string otherwise you will get error regarding deserialization.
{"message":"#string(activity('Lookup1').output.value)",
"dataFactoryName":"#{pipeline().DataFactory}",
"pipelineName":"#{pipeline().Pipeline}",
"receiver":"#{pipeline().parameters.receiver}"}
OUTPUT
2. Get all the respective values column wise.
First get the results from Lookup activity then take a foreach loop and create append variable for every column to store every column value in single array.
ForEach activity setting:
Took append variable activity and created Idarray variable. and gave item().id as value to store all id values in a single array.
Then in web activity passed below body for getting all arrays.
{"message":"#{string(variables('Idarray'))} as Id, #{string(variables('Namearray'))} as Name, #{string(variables('ProfessionArray'))} as Profession",
"dataFactoryName":"#{pipeline().DataFactory}",
"pipelineName":"#{pipeline().Pipeline}",
"receiver":"#{pipeline().parameters.receiver}"}
OUTPUT
I'm trying to insert data row to bigQuery table as follows:
await bigqueryClient
.dataset(DATASET_ID)
.table(TABLE_ID)
.insert(row);
But I get a PartialFailureError when deploying the cloud function.
The table schem has a name (string) and campaigns (record/repeated) fields which I created manually from the console.
hotel_name STRING NULLABLE
campaigns RECORD REPEATED
campaign_id STRING NULLABLE
platform_id NUMERIC NULLABLE
platform_name STRING NULLABLE
reporting_id STRING NULLABLE
And the data I'm inserting is an object like this:
const row = {
hotel_name: hotel_name,//string
campaigns: {
id: item.id,//string
platform_id: item.platform_id,//int
platform_name: item.platform_name,//string
reporting_id: item.reporting_id,//string
},
};
The errors logged don't give much clue about the issue.
These errors suck. The actual info about what went wrong can be found in the errors property on the PartialFailureError. In https://www.atdatabases.org we reformat the error to make this easier using: https://github.com/ForbesLindesay/atdatabases/blob/0e1a033264aac33deaff2ab753796049e623ab86/packages/bigquery/src/implementation/BigQueryDriver.ts#L211
According to my test it seems that there are 2 errors here. First is that you have campaign_id in schema while id in JSON.
2nd thing is related with format of REPEATED mode data in JSON. The documentation mentions following:
. Notice that the addresses column contains an array of values (indicated by [ ]). The multiple addresses in the array are the repeated data. The multiple fields within each address are the nested data.
It's not so straight in mentioned document (probably can be found somewhere else) however when you use REPEATED mode you should use brackets [].
I tested it shortly on my side and it seems that it should work like this:
const row = {
hotel_name: hotel_name,//string
campaigns: [ {
campaign_id: item.id,//string
platform_id: item.platform_id,//int
platform_name: item.platform_name,//string
reporting_id: item.reporting_id,//string
}, ]
};
I am using power query to load some json data in a table (matches). I want to use a specific part of that data (fixture_id) as a parameter for another json request in another query (predictions), and then combine that output in my main (matches) table. Anyone can point me in the right direction on how to do this ?
So here is my matches table:
And then in my fixtures table i can maybe i have:
apiKey = Excel.CurrentWorkbook(){[Name="ApiKey"]}[Content]{0}[Column1],
fixtureID = "?",
Source = Json.Document(Web.Contents("https://v2.api-football.com/predictions/" & fixtureID, [Headers=[#"X-RapidAPI-Key"=apiKey]])),
If i hardcode the fixtureID, i get this output:
But i want to calculate it dynamically, and then merge the output to the matches table.
The first step is to turn your request into a function that accepts parameters. Put your request on a new blank query:
let
fnGetData = (fixtureID as text) =>
let
apiKey = Excel.CurrentWorkbook(){[Name="ApiKey"]}[Content]{0}[Column1],
fixtureID = "?",
Source = Json.Document(Web.Contents("https://v2.api-football.com/predictions/"
& fixtureID, [Headers=[#"X-RapidAPI-Key"=apiKey]]))
in
Source
in
fnGetData
Rename it to fnGetData.
Then, go to your table and click on Add Column/Add Custom Function. Select fnGetData and the input parameter is your fixtureID column. This should make all the requests and you'll just have to expand the new column results.
Creating saved search in suitescript using nlapiSearchRecord. All the column value returns except one column which is type is custom list.
How could I get value of custom list?
To get the value I'm using code lines below.
columns[0] = new nlobjSearchColumn( 'customlist' );
var searchresults = nlapiSearchRecord( 'customrecord', null, filters, columns );
To get the column value
var listValue = searchresult.getListValue( 'customlist' );
I assume you've simplified your code in trying to be clear or confidential but there will never be fields or records with those ids.
from a search you would do:
var searchResult = searchResults[0];
searchResult.getValue(fieldId, joinName, summary)
// or in your case
searchResult.getValue('customlist'); //returns id of list value or simple result of non-list/record fields
or (and I think this is the one you want)
searchResult.getText('customlist'); // returns the display value of the list/record field.
I have a dynamodb table which has following columns,
id,name,events, deadline
events is a list which contain number of events.
I want to scan/query for all the rows with following items as the result,
id, name, number of events.
I tried following way but didn't receive any value for number of events. Can someone show me where am I wrong.
var params = {
TableName: 'table_name',
ExpressionAttributeNames: {"#name": "name",
"#even": "events.length"
},
ProjectionExpression: 'id, #name, #even'
}
You cannot achieve what you want in this way. The entries in "ExpressionAttributeNames" are not evaluated as expressions.
The definition of "#even": "events.length" in "ExpressionAttributeNames" does not evaluate the expression event.length and assign it to the variable "#even". Instead it specifies "#even" as referring to a column named "events.length" or a table where "events" is an object that has a "length" attribute. Since your table has neither, you get nothing back.
From the DynamoDB documentation:
In an expression, a dot (".") is interpreted as a separator character in a document path. However, DynamoDB also allows you to use a dot character as part of an attribute name.
To achieve what you want, you will have to return the "events" column and calculate the length outside of the query, or define a new "eventsLength" column and populate and maintain that value yourself if you are concerned about returning "events" in each query.