Problem Summary
I have a column in my Power Query table which contains a custom linked data type. Creating a custom linked data type filled with all null values is not desired. Instead, if all the values contained in the custom data type is null, I would like the value in the column to be null.
Background
I have a table which holds API response JSON text. This JSON text contains a list of search results (also in JSON), representing movies which match search criteria delivered in the request. There can be any number of search results, including zero. Using Power Query M, I parse these JSON texts with the built-in parser, which generates a list containing one record per search result. I then extract the first record in the list, expand that record into new columns, and combine those new columns into a custom data type.
Example
Here is an example query simulating only the problem area of my query. This example is fully contained and can be used to reproduce my issue exactly.
let
// These two variables holds the API response JSON text obtained from calls to Web.Contents().
// I've eliminated the actual calls in this example because that part of my query works fine.
Search_Fast_and_Furious_Response =
"{ ""total-results"":""2"", ""results"":[
{ ""title"":""Fast & Furious"", ""year"":""2009"" },
{ ""title"":""The Fast and the Furious"", ""year"":""2001"" } ] }",
Search_mmmmm_Response =
"{ ""total-results"":""0"", ""results"":[] }",
// Create the table to hold the response text.
Source = Table.FromRecords( { [#"API Response"=Search_Fast_and_Furious_Response],
[#"API Response"=Search_mmmmm_Response] }),
// Parse the JSON and put the output (a record) in a new column.
#"Insert Parsed JSON" = Table.AddColumn(Source, "JSON", each Json.Document([API Response])),
// Expand the record in the parsed JSON column. Each field in the record becomes a new column.
#"Expand JSON" = Table.ExpandRecordColumn(#"Insert Parsed JSON", "JSON",
{"total-results", "results"}, {"Result Count", "Results List"}),
// Add a new column to hold the first search result in the responses results list.
// This is also a record, like the parsed JSON two steps ago.
#"Add Result #1 Column" = Table.AddColumn(#"Expand JSON", "Result #1", each
try _[Results List]{0}
otherwise null), // In case the list is empty
// Expand the record in the Result #1 column.
#"Expand Result #1" = Table.ExpandRecordColumn(#"Add Result #1 Column", "Result #1",
{"title", "year"}, {"Title", "Year"}),
// Combine the newly expanded columns into a single column.
// Make the Display Name be the value in the Title field/column,
// and make the Type Name be "Excel.DataType."
// This is what creates the custom linked data type.
#"Combine Result #1" = Table.CombineColumnsToRecord(#"Expand Result #1", "Result #1",
{"Title", "Year"}, [ DisplayNameColumn = "Title", TypeName="Excel.DataType" ])
in
#"Combine Result #1"
The list in the very last line before the in statement, i.e. the fourth parameter to the Table.CombineColumnsToRecord function, allows the record to be used as a custom data type used with Excel's new linked data feature. I'm not certain, but I believe Power Query/Excel stores them as records with additional metadata, such as DisplayNameColumn and TypeName (the latter of which I'm sure is the most important part).
Problem and Goal
Here is the resulting table created by the example query. The bottom-right cell is selected. Its contents are shown at the bottom of the image. The cell itself contains a value, specifically a record with all values set to null. Because the Title field is null, the record's display text is "null."
This next picture shows my desired output. Notice again the bottom-right cell. This time, the cell is empty. It no longer contains a record with all values being null; now it contains nothing, so the display shown in this view is null, italicized so as to indicate a null value as opposed to the word "null." (Note: I've been unable to change the "null" cell in the first image to a literal null value, so to demonstrate, I simply added a new column of null values.)
Unfortunately, because of my otherwise clause after the try, the column "Result #1" may be null if the API returned zero search results. If this value is null in any row, then all of the new columns created by #"Expand Result #1" will contain null in that row, also. Finally, when all the null values are combined in the last step, I'm left with a record with all null values. Instead, what I hope to achieve is to have a single null value (of type null) in that cell.
Efforts So Far
I have tried the Table.ReplaceValues function, passing null as the new value and many different values as the old value (the one to be replaced), such as a new record with all null values. All those attempts have either been syntactically incorrect or resulted in expected and unwanted behavior. I have also tried using the "Replace Values" option in the Power Query GUI, but the same result occurs. In case ReplaceValues didn't like nulls, I've also tried using a different value in the otherwise clause, such as "N/A" of type text, then doing a ReplaceValues on that different value. This yielded the same result.
Conclusion
Is there any way I can replace a record—which is filled with null values and is stored in a column containing records—with a singular null value? The linked data type feature is a high priority in this situation, so I would prefer a solution that retains that feature (though of course all solutions are welcome).
I have "solved" my problem. While not technically a solution to the question I posted, I've achieved the desired result using a workaround.
Instead of dealing with the object full of null fields, I ensure that object is not converted to the custom object to begin with. I achieve this by moving all records with a null value after extracting the first List item in the Results List column; this is done before I expand that extracted item. After putting the nulls in a new table (which I call the Null Table), I delete those nulls from the first table (which I call the Non-Null Table). I perform the regular operations on the Non-Null Table to create the custom linked data type for only those rows that were not null. Afterward, I merge the two tables together again.
The full code containing the solution with my representative example is below, with new steps "highlighted" with non-indented comments.
let
// These two variables holds the API response JSON text obtained from calls to Web.Contents().
// I've eliminated the actual calls in this example because that part of my query works fine.
Search_Fast_and_Furious_Response =
"{ ""total-results"":""2"", ""results"":[
{ ""title"":""Fast & Furious"", ""year"":""2009"" },
{ ""title"":""The Fast and the Furious"", ""year"":""2001"" } ] }",
Search_mmmmm_Response =
"{ ""total-results"":""0"", ""results"":[] }",
// Create the table to hold the response text.
Source = Table.FromRecords( { [#"API Response"=Search_Fast_and_Furious_Response],
[#"API Response"=Search_mmmmm_Response] }),
// Parse the JSON and put the output (a record) in a new column.
#"Insert Parsed JSON" = Table.AddColumn(Source, "JSON", each Json.Document([API Response])),
// Expand the record in the parsed JSON column. Each field in the record becomes a new column.
#"Expand JSON" = Table.ExpandRecordColumn(#"Insert Parsed JSON", "JSON",
{"total-results", "results"}, {"Result Count", "Results List"}),
// Add a new column to hold the first search result in the responses results list.
// This is also a record, like the parsed JSON two steps ago.
#"Add Result #1 Column" = Table.AddColumn(#"Expand JSON", "Result #1", each
try _[Results List]{0}
otherwise null), // In case the list is empty
// New step
// Filter down to only rows with null in the new column. Save this new table for later.
#"Filter In Null" = Table.SelectRows(#"Add Result #1 Column", each _[#"Result #1"] = null),
// New step
// Filter down to only rows with NOT null in the new column.
#"Filter Out Null" = Table.SelectRows(#"Add Result #1 Column", each _[#"Result #1"] <> null),
// Expand the record in the Result #1 column.
#"Expand Result #1" = Table.ExpandRecordColumn(#"Filter Out Null", "Result #1",
{"title", "year"}, {"Title", "Year"}),
// Combine the newly expanded columns into a single column.
// Make the Display Name be the value in the Title field/column,
// and make the Type Name be "Excel.DataType."
// This is what creates the custom linked data type.
#"Combine Result #1" = Table.CombineColumnsToRecord(#"Expand Result #1", "Result #1",
{"Title", "Year"}, [ DisplayNameColumn = "Title", TypeName="Excel.DataType" ]),
// New step
// Convert the Null Table into a list of records.
#"Convert Table" = Table.ToRecords(#"Filter In Null"),
// New step
// Append the Null Table from earlier to the main table.
#"Combine Tables" = Table.InsertRows(#"Combine Result #1", Table.RowCount(#"Combine Result #1"),
#"Convert Table")
in
#"Combine Tables"
I have an array with different properties. The title property will be duplicated but the other properties in that item will not. I need to display the title property in a table however only if they are not duplicates.
my code below
data-bind="id: 'criteria_category_item_' + Criteria_Category_ID, text: Criteria_Title"
the above will display text for all titles even if they already exist so i now have duplicates in my table.
How would i do this in a foreach ?
there is no data-bind="id
you probably look for attr-binding:
data-bind="attr:{id: ...},text: ..."
I have a requirement wherein I need to filter the List lookup dynamically. My list has a column called category that can either contain value 'A' or 'B'. Then there is a field -'Selection' on the Content type that can either take value 'A' or 'B' or 'All' . If its 'A' I need the List lookup to take rows where category = 'A' , same goes for'B'. However if the 'Selection' is - 'All' then I need to display all the items from the list.
I was thinking of filtering the List lookup for the column - 'Category' . But the problem is as I am on a content type form , I do not have any variables that can be set dynamically.
I can not use the Filter by control mapped to 'Selection' as the it does not work when the selection is 'All' (There is no value called 'All' under category in the list).
I tried using a calculated value that operates on a formula and tried using it in Filter by specific value in List lookup , but the filter doesn't work as the list lookup loads before the calculated value on form load and hence the calculated value is always empty for the filtering.
Is there any way I can achieve this functionality.
Thanks in advance
I have thought of two solutions to this problem.
Have 3 separate list lookup controls that are laid over top of each other. Filter one by A, one by B, and let one have no filter. Then create rules to show the one with the filter you want to show, and hide the others. To save the value, you'd have to use JavaScript to copy the value from the list lookup to a hidden text box when one of the controls changes value. This solution isn't great and gets worse if you have more options...but it works.
You can use JavaScript to filter the list based off the Selection. This is a little more tricky, but you wouldn't need more lookup controls for more options. You would need only 2 list lookups no mater how many options you have for Category/Selection. You need one that shows the info that you want the user to chose from (unfiltered) and the other is from the same list, and same view, but the column should be the Category column ( you can hide this lookup with javascript). This is the code I used to get what you are describing.
//get the original html to 'reset' the dropdown after a change
var originalTitle = NWF$("#" + title).html();
//when the selection changes
NWF$("#" + selection).change(function () {
//put the original html in the dropdown to check all the options
NWF$("#" + title).html(originalTitle);
//get the new value of the selection
var choice = NWF$("#" + selection + " :checked")[0].value
//if choice is all then we are done because the original html is in the dropdown again with all the options
if (choice == "All") {
return;
}
//create the array where you will store the ids of the options that match the choice
var filteredIds = [];
//for each option in the category drop down, see if the text matches the choice (this is your filtering)
NWF$("#" + categoryDD + " option").each(function (i, n) {
//if the text of the option matches the choice add the id to the array.
if (n.text == choice) {
filteredIds.push(n.value);
}
});
//initialize string of html
var filteredTitlesHTML = ""
//for each of the ids in the list, get the option html with that id from the title dropdown and add it to the resulting HTML string
NWF$(filteredIds).each(function (i, n) {
filteredTitlesHTML += NWF$("#" + title + " option[value = '" + n + "']")[0].outerHTML
})
//put the html in the dropdown to show only filtered values
NWF$("#" + title).html(filteredTitlesHTML);
})
You can see in the picture the javascript variable names I gave the controls to use the javascript I have provided.
I am dynamically generating a selectlist in Drupal and I want to create an associative array to populate the node ID as the value, and the title of the node as the option.
By default, it makes the value for each option the index of the select list. This is no good because the select list is dynamic, meaning the values won't be in the same order.
I used drupal_map_assoc to make the value the same as the option, but I have queries based on the value stored in this field, so if someone updates the value stored, the queries won't match.
<option value="Addison Reserve Country Club Inc.">Addison Reserve Country Club Inc.</option>
I want to replace the value with the Node ID also pulled with the query.
$sql = 'SELECT DISTINCT title, nid FROM {node} WHERE type = \'accounts\' ';
$result = db_query($sql);
while ($row = db_fetch_array($result)) {
$return[] = $row['title'];
//Trying to do something like 'Addison Reserve Country Club' => '2033' - where 2033 is the nid
}
$return = drupal_map_assoc($return);
I think you just want to do this inside the loop:
$return[$row['nid']] = $row['title'];
Based on your comment, you would also want to do an array_flip() right after the loop, but I think your comment may just have it backwards.
I'm trying something very simple - accessing my SharePoint list's items and their properties.
However, SPListItem.Properties count is zero for all normal lists. Everything works as expected for document and pages libraries. So, if the list items are based on document type, all is good. If they are based on item, properties are not returned.
I've tried in two environments, with new sites created from OOTB publishing templates, with new lists which are based on OOTB content types etc. Always the same thing.
Right amount of SPListItems is always returned. Title and Name are fine. It's just that the .Properties hashtable is totally empty.
In desperation, I wrote a web part that outputs the following (ugly!) diagnostics.
foreach (SPList list in SPContext.Current.Web.Lists)
{
foreach (SPListItem item in list.Items)
{
Label label = new Label();
label.Text = "Name: " + item.Name + "Property count: " + item.Properties.Count;
this.Controls.Add(label);
}
}
The only observation is that it works exactly as I described earlier. I just share the code to show that this is the most basic operation imaginable.
Here is sample output - I've added the line breaks for readability ;-)
Name: Test Property count: 0
Name: default.aspx Property count: 21
Obviously item "Test" is an item based list item and default.aspx is a page.
Has anyone ever encountered anything like this? Any ideas?
item["FieldName"] is the canonical way to get a value of a metadata column in a SharePoint list. If you need to get the list of available fields for a SharePoint list, check the parent SPList object's Fields property which is a collection of the fields in this list. Each of those field objects will have a property called InternalName and that is what you should use to access its value when you are working with an instance of SPListItem.
Are you trying to get the field Values? Sadly, they are not strongly typed:
string ModifiedBy = (string)item["Author"];
To get the proper names of the fields (they have to be the internal names), go to the List and then List Settings. You will find the List of Columns there. Click on any Column Name to go to the Edit Page, and then look at the URL in the Address Bar of your Browser. At the very end, there should be a parameter "Field=Editor" or similar - that's your internal field name.
If you wonder why a field like "Test Field" looks strange, that is because Sharepoint encodes certain characters. A Space would be encoded to x0020 hence the Internal Name for "Test Field" is "Test_x0020_Field".
In order to get the proper field type to cast to:
string FieldType = item["Author"].GetType().FullName;
(The Intermediate Window in Visual Studio is tremendously helpful for this!)
I have found the following extension to the SPListItem class to be very helpful.
internal static class SharePointExtensions
{
internal static Dictionary<string, object> PropertiesToDictionary(this SPListItem item)
{
// NOTE: This code might be temping - but don't do it! ...itdoes not work as there are often multiple field definitions with the same Title!
// return item.Fields.Cast<SPField>().ToDictionary(fld => fld.Title, fld => item[fld.Title]);
Dictionary<string, object> dict = new Dictionary<string, object>();
var fieldNames = item.Fields.Cast<SPField>().Select(fld => fld.Title).Distinct().OrderBy(sz => sz).ToArray();
foreach (fieldName in fieldNames)
dict.Add(sz, item[fieldName]);
return dict;
}
}
with it you can simply do:
item.PropertiesToDictionary()
...and you will have a nice Linq dictionary object that will work just the way you'd expect. Is it a little heavy-weight / low-performance? Sure. Are most people willing to make that trade-off? I think so...
Do you run SPListItem.Update() or .SystemUpdate() after setting the properties?
If you want to get an object out of a SPField of a SPListItem you've got to do like this:
SPField field = ((SPList)list).Fields.GetField("FieldName");
object fieldValue = field.GetFieldValue(((SPListItem)item)[field.Title].ToString());
The ListItem.Properties hashtable will be empty unless you assign to it.
SPListItem item = properties.ListItem;
item.Properties["Key"] = "value";
int total = item.Properties.Count;
Answer:
"total == 1"
SPList yourList = web.Lists["Your list name"];
string sColumnValue = oSPListItem[yourList.Fields["yourSiteColumn display
name"].InternalName].ToString();