I have the following input table:
Sales Order
Asset Serial Number
Asset Model
Licence Class
License Type
License Name
Account Name
10000
1234, 5643, 3463
test-pro
A123
software
LIC-0002, LIC-0188, LIC-0188, LIC-0013
ABC
2000
5678, 9846, 5639
test-pro
A123
software
LIC-00107, LIC-08608, LIC-009, LIC-0610
ABC
Here the screenshot
I need it transformed into form:
.
I tried it first with the Replace function & transponate it but I didn't find a way to add the other empty columns other than do it manually.
My second thought was the text-to-column function, didn't work either.
Here two solutions one using Excel formulas and the other one using Power Query. See Explanation section for more information about each approach:
Excel
It is possible with excel without using Power Query, but several manipulations are required. On cell I2 put the following formula:
=LET(counts, BYROW(F2:F3, LAMBDA(a, LEN(a) - LEN(SUBSTITUTE(a, ",", "")))), del, "|",
emptyRowsSet, MAP(A2:A3, B2:B3, C2:C3, D2:D3, E2:E3, F2:F3, G2:G3, counts,
LAMBDA(a,b,c,d,e,f,g,cnts, LET(rep, REPT(";",cnts),a&rep &del& b&rep &del& c&rep &del&
d&rep &del& e&rep &del& SUBSTITUTE(f,", ",";") &del& g&rep ))),
emptyRowsSetByCol, TEXTSPLIT(TEXTJOIN("&",,emptyRowsSet), del, "&"),
byColResult, BYCOL(emptyRowsSetByCol, LAMBDA(a, TEXTJOIN(";",,a))),
singleLine, TEXTJOIN(del,,byColResult),
TRANSPOSE(TEXTSPLIT(singleLine,";",del))
)
Here is the output:
Update
A simplified version of previous formula is the following one:
=LET(counts, BYROW(F2:F3, LAMBDA(a, LEN(a) - LEN(SUBSTITUTE(a, ",", "")))), del, "|",
reps, MAKEARRAY(ROWS(A2:G3),COLUMNS(A2:G3), LAMBDA(a,b, INDEX(counts, a,1))),
emptyRowsSetByCol, MAP(A2:G3, reps, LAMBDA(a,b, IF(COLUMN(a)=6,
SUBSTITUTE(a,", ",";"), a&REPT(";",b)))),
byColResult, BYCOL(emptyRowsSetByCol, LAMBDA(a, TEXTJOIN(";",,a))),
singleLine, TEXTJOIN(del,,byColResult),
TRANSPOSE(TEXTSPLIT(singleLine,";",del))
)
Power Query
The following M Code provides the expected result:
let
Source = Excel.CurrentWorkbook(){[Name="TB_Sales"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Sales Order", type text}}),
#"Split License Name" = Table.ExpandListColumn(Table.TransformColumns(#"Changed Type", {{"License Name",
Splitter.SplitTextByDelimiter(", ", QuoteStyle.Csv),
let itemType = (type nullable text) meta [Serialized.Text = true] in type {itemType}}}), "License Name"),
ListOfColumns = List.Difference(Table.ColumnNames(#"Split License Name"), {"License Name"}),
RemainingColumns = List.Difference(Table.ColumnNames(#"Changed Type"), ListOfColumns),
RemoveDups = (lst as list) =>
let
concatList = (left as list, right as list) => List.Transform(List.Positions(left), each left{_}&"_"& right{_}),
prefixList = Table.Column(#"Split License Name", "Sales Order"),
tmp = concatList(prefixList, lst),
output = List.Accumulate(tmp, {}, (x, y) => x & {if List.Contains(x, y) then null else y})
in
output,
replaceValues = List.Transform(ListOfColumns, each RemoveDups(Table.Column(#"Split License Name", _))),
#"Added Empty Rows" = Table.FromColumns(
replaceValues & Table.ToColumns(Table.SelectColumns(#"Split License Name", RemainingColumns)),
ListOfColumns & RemainingColumns),
#"Extracted Text After Delimiter" = Table.TransformColumns(#"Added Empty Rows", {{"Sales Order",
each Text.AfterDelimiter(_, "_"), type text}, {"Asset Serial Number", each Text.AfterDelimiter(_, "_"), type text},
{"Asset Model", each Text.AfterDelimiter(_, "_"), type text}, {"Licence Class",
each Text.AfterDelimiter(_, "_"), type text}, {"License Type", each Text.AfterDelimiter(_, "_"), type text},
{"Account Name", each Text.AfterDelimiter(_, "_"), type text}}),
#"Reordered Columns" = Table.ReorderColumns(#"Extracted Text After Delimiter",{"Sales Order", "Asset Serial Number", "Asset Model",
"Licence Class", "License Type", "License Name", "Account Name"})
in
#"Reordered Columns"
And here is the output:
And the corresponding Excel Output:
Explanation
Here we provide the explanation for each approach: Excel formula and Power Query.
Excel Formula
We need to calculate how many empty rows we need to add based on License Name column values. We achieve that via counts name from LET:
BYROW(F2:F3, LAMBDA(a, LEN(a) - LEN(SUBSTITUTE(a, ",", ""))))
The output for this case is: {3;3}, i.e 2x1 array, which represents how many empty rows we need to add for each input row.
Next we need to build the set that includes empty rows. We name it emptyRowsSet and the calculation is as follow:
MAP(A2:A3, B2:B3, C2:C3, D2:D3, E2:E3, F2:F3, G2:G3, counts,
LAMBDA(a,b,c,d,e,f,g,cnts,
LET(rep, REPT(";",cnts),a&rep &del& b&rep &del& c&rep &del&
d&rep &del& e&rep &del& SUBSTITUTE(f,", ",";") &del& g&rep)))
We use inside MAP an additional LET function to avoid repetition of rep value. Because we want to consider the content of License Name as additional rows we replace the , by ; (we are going to consider this token as a row delimiter). We use del (|) as a delimiter that will serve as a column delimiter.
Here would be the intermediate result of emptyRowsSet:
10000;;;|1234, 5643, 3463;;;|test-pro;;;|A123;;;|software;;;|LIC-0002;LIC-0188;LIC-0188;LIC-0013|ABC;;;
2000;;;|5678, 9846, 5639;;;|test-pro;;;|A123;;;|software;;;|LIC-00107;LIC-08608;LIC-009;LIC-0610|ABC;;;
As you can see additional ; where added per number of items we have in License Name column per row. In the sample data the number of empty rows to add is the same per row, but it could be different.
The rest is how to accommodate the content of emptyRowsSet in the way we want. Because we cannot invoke TEXTSPLIT and BYROW together because we get #CALC! (Nested Array error). We need to try to circumvent this.
For example the following produces an error (#CALC!):
=BYROW(A1:A2,LAMBDA(a, TEXTSPLIT(a,"|")))
where the range A1:A2 has the following: ={"a|b";"c|d"}. We don't get the desired output: ={"a","b";"c","d"}. In short the output of BYROW should be a single column so any LAMBDA function that expands the columns will not work.
In order to do circumvent that we can do the following:
Convert the input into a single string joining each row by ; for example. Now we have column delimiter (|) and row delimiter (;)
Use TEXTSPLIT to generate the array (2x2 in this case), identifying the columns and the row via both delimiters.
We can do it as follow (showing the output of each step on the right)
=TEXTSPLIT(TEXTJOIN(";",,A1:A2),"|",";") -> 1) "a|b;c|d" -> 2) ={"a","b";"c","d"}
We are using the same idea here (but using & for joining each row). The name emptyRowsSetByCol:
TEXTSPLIT(TEXTJOIN("&",,emptyRowsSet), del, "&")
Would produce the following intermediate result, now organized by columns (Table 1):
Sales Order
Asset Serial Number
Asset Model
License Class
License Type
License Name
Account Name
10000;;;
1234, 5643, 3463;;;
test-pro;;;
A123;;;
software;;;
LIC-0002;LIC-0188;LIC-0188;LIC-0013
ABC;;;
2000;;;
5678, 9846, 5639;;;
test-pro;;;
A123;;;
software;;;
LIC-00107;LIC-08608;LIC-009;LIC-0610
ABC;;;
Note: The header are just for illustrative purpose, but it is not part of the output.
Now we need to concatenate the information per column and for that we can use BYCOL function. We name the result: byColResult of the following formula:
BYCOL(emptyRowsSetByCol, LAMBDA(a, TEXTJOIN(";",,a)))
The intermediate result would be:
Sales Order
Asset Serial Number
Asset Model
License Class
License Type
License Name
Account Name
10000;;;;2000;;;
1234, 5643, 3463;;;;5678, 9846, 5639;;;
test-pro;;;;test-pro;;;
A123;;;;A123;;;
software;;;;software;;;
LIC-0002;LIC-0188;LIC-0188;LIC-0013;LIC-00107;LIC-08608;LIC-009;LIC-0610
ABC;;;;ABC;;;
1x7 array and on each column the content already delimited by ; (ready for the final split).
Now we need to apply the same idea as before i.e. convert everything to a single string and then split it again.
First we convert everything to a single string and name the result: singleLine:
TEXTJOIN(del,,byColResult)
Next we need to do the final split:
TRANSPOSE(TEXTSPLIT(singleLine,";",del))
We need to transpose the result because SPLIT processes the information row by row.
Update
I provided a simplified version of the initial approach which requires less steps, because we can obtain the result of the MAP function directly by columns.
The main idea is to treat the input range A2:G3 all at once. In order to do that we need to have all the MAP input arrays of the same shape. Because we need to take into account the number of empty rows to add (;), we need to build this second array of the same shape. The name reps, is intended to create this second array as follow:
MAKEARRAY(ROWS(A2:G3),COLUMNS(A2:G3),
LAMBDA(a,b, INDEX(counts, a,1)))
The intermediate output will be:
3|3|3|3|3|3|3
3|3|3|3|3|3|3
which represents a 2x7 array, where on each row we have the number of empty rows to add.
Now the name emptyRowsSetByCol:
MAP(A2:G3, reps,
LAMBDA(a,b, IF(COLUMN(a)=6, SUBSTITUTE(a,", ",";"),
a&REPT(";",b))))
Produces the same intermediate result as in above Table 1. We treat different the information from column 6 (License Name) replacing the , with ;. For other columns just add as many ; as empty rows we need to add for each input row. The rest of the formula is just similar to the first approach.
Power Query
#"Split License Name" is a standard Power Query (PQ) UI function: Split Column by Delimiter.
To generate empty rows we do it by removing duplicates elements on each column that requires this transformation, i.e. all columns except License Name. We do it all at once identifying the columns that require such transformation. In order to do that we define two lists:
ListOfColumns: Identifies the columns we are going to do the transformation, because we need to do it in all columns except for License Name. We do it by difference via the PQ function: List.Difference().
RemainingColumns: To build back again the table, we need to identify the columns don't require such transformation. We use same idea via List.Difference(), based on ListOfColumns list.
The user defined function RemoveDups(lst as list) does the magic of this transformation.
Because we need to remove duplicates, but having unique elements based on each initial row, we use the first column Sales Order as a prefix, so we can "clean" the column within each partition.
In order to do that we define inside of RemoveDups() function a new user defined function concatList() to add the first column as prefix.
concatList = (left as list, right as list) =>
List.Transform(List.Positions(left), each left{_}&"-"& right{_}),
we concatenate each element of the lists (row by row) using a underscore delimiter (_). Later we are going to use this delimiter to remove the first column as prefix added at this point.
To remove duplicates and replace them with null we use the following logic:
output = List.Accumulate(tmp, {}, (x, y) =>
x & {if List.Contains(x, y) then null else y})
where tmp is a modified list (lst) with the first column as prefix.
Now we invoke the List.Transform() function for all the columns that require the transformation using as transform (second input argument) the function we just defined previously:
replaceValues = List.Transform(ListOfColumns, each
RemoveDups(Table.Column(#"Split License Name", _))),
#"Added Empty Rows" represents the step of this calculation and the output will be the following table:
The step #"Extracted Text After Delimiter" is just to remove the prefix we added and for that we use standard PQ UI Transform->Extract->Text After Delimiter.
Finally we need to reorder the column to put in a way it is expected via the step: #"Reordered Columns" using PQ UI functionality.
I'm trying to use Excel's get&transform functionality (previously known as powerquery) to import an XML data source. The data source has a list of b tags, each with a variable number of d tags in a c2 child, such as the following:
<a>
<b>
<c1>foo</c1>
<c2>
<d>bar</d>
</c2>
</b>
<b>
<c1>fuz</c1>
<c2>
<d>baz</d>
<d>quz</d>
</c2>
</b>
</a>
When I import this data with the following query the data type for column c2.d is different for the two different rows representing the b items, for the first row it is a general spreadsheet cell type, for the second row it is a Table type.
let
Source = Xml.Tables(File.Contents("C:\Localdata\excel-powerquery-test2.xml")),
Table0 = Source{0}[Table],
#"Changed Type" = Table.TransformColumnTypes(Table0,{{"c1", type text}}),
#"Expanded c2" = Table.ExpandTableColumn(#"Changed Type", "c2", {"d"}, {"c2.d"})
in
#"Expanded c2"
It seems that for the first row it automatically converts the d tag into a simple spreadsheet cell as there is only one and it only contains text. However for the second row it sees there are two d tags and hence keeps it as a table. The problem now is that I can neither load the data as is as the Table in the second row is loaded into the spreadsheet as the literal string "Table" leaving me without the actual data, nor can I further expand the Table using Table.ExpandTableColumn as it (rightly) complains that bar in the first row is not a table.
I presume the automatic conversion of a single tag containing text to a simple cell rather than a table happens either in the Xml.Tables or ExpandTableColumn functions. The tooltip for Xml.Tables shows that it has an options parameter, unfortunately the documentation for Xml.Tables does not give any details on this options parameter.
How can I get this second row expanded out to two rows, one each for the two d tags contained in the second b tag having the same "fuz" string in the first column? Such an expansion works fine if the contents of the d tags are further XML tags, but apparently not if the d tags only contain text.
Let's add a step to make sure everything is at the same level:
let
Source = Xml.Tables(File.Contents("C:\Localdata\excel-powerquery-test2.xml")),
Table0 = Source{0}[Table],
Expandc2 = Table.ExpandTableColumn(Table0, "c2", {"d"}, {"d"}),
ToLists = Table.TransformColumns(Expandc2,
{"d", each if _ is table then Table.ToList(_) else {_}}),
ExpandLists = Table.ExpandListColumn(ToLists, "d")
in
ExpandLists
The ToLists step turns this:
Into a more consistent list format:
c1 d
-----------------------
foo {"bar"}
fuz {"baz", "quz"}
Then you can expand to rows without mixed data types.
I would like to create some kind of LOOKUPVALUE on text in DAX that will match a sentence with a particular keyword. For instance in the example below the second and third row have a hit because “Apple” and "Chicken" is in the string. The problem is that the text is in a string and not a standalone value.
Table 1
Table 2
Output
EDIT, improved answer: this new version also works when there are multiple keys in one string.
I think PowerQuery is the natural place to perform an operation like this.
The Output table would look like this:
A description of the applied steps:
Source: a reference to Table1
Added Column Key lists: adds a custom column with lists of the Table2[Key] value(s) that are in the [String] value. This is the logic for this Custom column:
For each row the function selects the values from the Table2[Key] column that it finds in the [String] value. It then returns a list that holds only the selected values.
Expanded Key list: expands the lists in the [Key] column
Join with Table2 on Key: Joins with Table2 on the Key Value
Expanded Table2: Expands the table values in the [ItemTables] column and keeps the [Item] column
Group and concate keys/items: Groups the Output table on String, concatenating the Keys and the Items. If you don't want to see the [Key] column, delete {"Key", each Text.Combine([Key], " | "), type text}, from this step
The script in the Advanced Editor looks like this:
let
Source = #"Table1",
#"Added Column Key lists" = Table.AddColumn(Source, "Key", (r) => List.Select(Table.Column(Table2,"Key"),each Text.Contains(r[String],_,Comparer.OrdinalIgnoreCase)),type text),
#"Expanded Key lists" = Table.ExpandListColumn(#"Added Column Key lists", "Key"),
#"Join with Table2 on Key" = Table.NestedJoin(#"Expanded Key lists", {"Key"}, Table2, {"Key"}, "ItemTables", JoinKind.LeftOuter),
#"Expanded ItemTables" = Table.ExpandTableColumn(#"Join with Table2 on Key", "ItemTables", {"Item"}, {"Item"}),
#"Group and concate keys / items" = Table.Group(#"Expanded ItemTables", {"String"},{{"Key", each Text.Combine([Key], " | "), type text},{"Item", each Text.Combine([Item], " | "), type text}})
in
#"Group and concate keys / items"
Here is a link to my .pbix file
I created the following dummy data sets.
My interpretation of what your after is to Identify if a sentence contains a key word.
This can be done via a calculated column with the following formula -
Lookup = LOOKUPVALUE(Table2[Result],Table2[LookUp], IF(SEARCH("Apple",Table1[Sentence],,0)> 0, "Apple",""))
You can combine the If and Search Functions with the Lookupvalue function.
The formula is searching for the word "Apple" and then returning its position within the text and if no result is found, displays 0.
The IF statement then takes any result greater then 0, as anything greater then 0 means a result has been found and that is its position within the string, and states "Apple". This then becomes your lookup value.
This then displays as bellow
You can then replace the Blank ("") that is currently the result if false, with another if statement to look for another key word such as "Orange" and then add add that to your lookup table to pull through the result your after.
Hope this makes sense and helps!
Try this formula (see the picture which cell is where in my assumptions):
=IFERROR(INDEX($B$7:$B$9,MATCH(1,--NOT(ISERROR(FIND($A$7:$A$9,$A12))),0)),"-")
Background
I have a dataset with 10,000+ variables as column headers, which I want to reduce to the amount needed. I know how to select a sample of columns by listing columns that contain manually specified strings, say "glu" and "pep", that the columns must contain in order to be selected. This is the M code used to select the sample columns:
let
Source = Excel.CurrentWorkbook(){[Name="data"]}[Content],
ColumnsToSelect = List.Select(Table.ColumnNames(Source), each Text.Contains(_, "glu") or Text.Contains(_, "pep")),
SelectColumns = Table.SelectColumns(Source, ColumnsToSelect)
in
SelectColumns
This Power Query produces a table that i call "Data". Since I want to select columns based on multiple strings they must contain, I have made a dynamic list of strings that I have called "Outcomes". I want my Power Query to utilize this list of strings when choosing what columns to select.
Question
Is it possible to get my Power Query to utilize this dynamic list in the List.Select() or Table.SelectColumns() function or any other function, that will make my Power Query select only the columns that contain the strings on the list?
Use with this lines:
let
Source = Excel.CurrentWorkbook(){[Name="Data"]}[Content],
Source2 = Excel.CurrentWorkbook(){[Name="Outcomes"]}[Content],
Outcomes = Source2[Outcomes],
UnpivotedColumns = Table.UnpivotOtherColumns(Source, {}, "ColumnNames", "Filters"),
FilteredRows = Table.SelectRows(UnpivotedColumns, each List.AnyTrue(List.Transform(Outcomes, (substring) => Text.Contains([Filters], substring)))),
ColumnNames = List.Sort(List.Distinct(FilteredRows[ColumnNames]),Order.Ascending),
SelectColumns = Table.SelectColumns(Source,ColumnNames)
in
SelectColumns
the magic is in this line:
FilteredRows = Table.SelectRows(UnpivotedColumns, each List.AnyTrue(List.Transform(Outcomes, (substring) => Text.Contains([Filters], substring)))),
We have cassandra column family.
each row have multiple columns. columns have name, but value is empty.
if we have 5-10 row keys, how we can find column names that appear in all of these keys.
e.g.
row1: php, programming, accounting
row2: php, bookkeeping, accounting
row3: php, accounting
must return:
result: php, accounting
note we can not easily load whole row into the memory, because it may contain 1M+ columns
solution not need to be fast.
In order to do intersection of several rows, we will need to intersect two of them first, then to intersect the result with third and so on.
Looks like in cassandra we can query the data by column names and this is relatively fast operation.
So we first get Column Slice of 10k rows. Making list of column names (in PHP Cassa - put them in array). Then select those from second row.
Code may be looking like this:
$x = $cf->get($first_key, <some column slice>);
$column_names = array();
foreach(array_keys($x) as $k)
$column_names[] = $k;
$result = $cf->get($second_key, $column_slice = null, $column_names);
// write result somewhere, and proceed with next slice
You columns names are sorted and you can create an iterator for each row (this iterator load portion of date at once, for example 10k of columns). Now put each iterator into a priority queue (by the next column name). If you take for queue the k times the iterator with the same column names, this is common names between all rows, in the other case we move to the next element and return iterators to queue.
You could use a Hadoop map/reduce job as follows:
Map output key = column name
Map output value = row key
Reducer counts row keys for each column and outputs column name & count to a CF with the following schema:
key : [column name] {
Count : [count]
}
You can then query counts from this CF in reverse order. The first record will be the max, so you can keep iterating until a value is < max. This will be your intersection.