I'm working with a CSV export of identity data containing ~22000 records. One of the fields included is titled 'ExtendedAttributes' and each cell in the column contains a quote bound string of comma separated Key:Value pairs. Each record in the file has an arbitrary number of extended attributes (up to around 50). My ultimate objective is to expand these extended attributes into their own columns in Excel (2016). I already have solutions for the expansions into columns from other data using formulae, simple VBA and most recently Power Query based approaches.
However, my previous solutions have all been based on the Key:Value pairs being simple to delimit. In this export, the ExtendedAttributes field has:
Value data that may contain unescaped/unquoted commas. e.g.
"Key1: Value1, name: surname, forename, Key2: Value2, ... "
Keys that may contain multiple comma separated values, which are also unquoted/unescaped. e.g.
"Key1: Value1, emailAlias: alias1#domain, alias2#domain, alias3#domain, Key2: Value2, ... "
My usual approach to this, where Key:Value pairs don't have these problems would be to delimit using commas to break it into the key value pairs, transpose the data into rows, then delimit using the colon to populate my new columns and their values as described here in the PowerBI community pages
This doesn't work here because delimiting using a comma breaks the values.
Is there a straightforward way to parse this into the constituent Key:Value pairs using (ideally) Power Query? Happy to also go with VBA or formula based solutions.
My instinctive approach would be to try and identify substrings containing a colon and prepend them with a unique character, which can then be used as a delimiter. (It's not impossible that the data may also include unescaped colons, but I'm happy to assume that it doesn't) But recognise that this may be a needlessly complex approach and I'm unsure how best to do it.
I'm happy to keep values with multiple comma separated items as a single unit (A problem for me to deal with later).
For the example data:
"Key1: Value1, name: surname, forename, emailAlias: alias1#domain, alias2#domain, alias3#domain, Key2: Value2, ... "
I'd like to end up with something that lets me treat the data like this, using maybe a ! as an example unique character that I could then use as a delimiter:
"Key1: Value1!name: surname, forename!emailAlias: alias1#domain, alias2#domain, alias3#domain!Key2: Value2!..."
I don't have access to the original data (vendor controlled system) and have limited data processing tools on my corporate desktop (Excel 2016, VBA, PQ).
Appreciate any help.
In Power Query, you can define a function Partition as follows:
let
Output = (str as text, sep as text) as text =>
Text.RemoveRange(
Text.Replace(
Text.Combine(
List.Transform(
Text.Split(str, " "),
each if Text.Contains(_, ":") then sep & _ else _
),
" "
), ", " & sep, sep
),
0, Text.Length(sep)
)
in
Output
Example text transformation using separator !
Starting text:
Key1: Value1, name: surname, forename, emailAlias: alias1#domain
Split the string based on spaces into a list
Key1:
Value1,
name:
surname,
forename,
emailAlias:
alias1#domain
Prepend any list items containing : with separator !
!Key1:
Value1,
!name:
surname,
forename,
!emailAlias:
alias1#domain
Combine the list back into a string
!Key1: Value1, !name: surname, forename, !emailAlias: alias1#domain
Replace , ! with !
!Key1: Value1!name: surname, forename!emailAlias: alias1#domain
Remove the first separator
Key1: Value1!name: surname, forename!emailAlias: alias1#domain
Once you have this function defined, you can call it in a column transformation that would look something like
= Table.TransformColumns(#"Prev Step", {{"ColName", each Partition(_,"!") , type text}})
You do not say anything about the file records not being counted in the "ExtendedAttribute field" category... I prepared a function able to separate that area which you put in discussion. Please, use the next code:
Function separateKeys(x As String, sep As String) As String
Dim arr1, arr2, i As Long, k As Long
arr1 = Split(x, ": ")
ReDim arr2(UBound(arr1))
For i = 0 To UBound(arr1) - 1
If arr1(i + 1) = arr1(UBound(arr1)) Then Exit For
arr2 = Split(arr1(i + 1), " ")
arr2(UBound(arr2) - 1) = Replace(arr2(UBound(arr2) - 1), ",", sep)
arr1(i + 1) = Join(arr2, " ")
Next
separateKeys = Replace(Join(arr1, ":"), sep & " ", sep)
End Function
The above function can (probably) be adapted in a way to skip calculations for rest of the file, or also transform each comma in the sep character (simple using Replace).
In order to test the above function, please use the next testing Sub:
Sub testSepKeys()
Dim x As String, sep As String
sep = "|" 'you can try something else, but improbable to appear in the processed text
x = "Key1: Value1, name: surname, forename, emailAlias: alias1#domain, alias2#domain, alias3#domain, Key2: Value2, Value3, key3: Val1, Val2"
Debug.Print separateKeys(x, sep)
End Sub
Like global way of working, I would suggest splitting the file on line separator, then process all the array elements (lines) using the above (adapted) function and finally join it on the line separator.
The newly created file should be open using Workbooks.OpenText, DataType:=xlDelimited, OtherChar:=sep.
Please, test the above function and send some feedback.
I'm inserting data from Salesforce to Azure SQL incrementally using Python. In the source I have some columns that contains characters in multiple lines (with back to new line), as shown bellow:
Name
Familly_Name
Age
My python code by default generates a '\n' for this. And from my part, I have tried to replace it by CHAR(13) and CHAR(13)+CHAR(10) and CHAR(10) but none of them worked; example (bellow) on a simple select on Azure SQL:
select 'hi' + char(13) + 'there'
Output:
hi there
What I need is:
hi
there
Best regards,
CHAR(13) is a carriage return. Most applications will treat that as a new line, but you should really be using CHAR(13) + CHAR(10), which is a carriage return followed by a line break:
SELECT 'Hello' + CHAR(13) + CHAR(10) + 'Goodbye' AS WithNewLine;
Outputs the below:
WithNewLine
-----------
Hello
Goodbye
db<>fiddle
I am using a regex expression to validate an input. I would like to match the exact strings "spe1" and "spe1.grl".
So far I have written the following code:
'\\b(?!i' + this.stringToIgnore + '\\b)\\w+';
where this.stringToIgnore = 'spe1'
this work if I type:
"spe1 " (note the space)
"spe1."
I would like to have a match as soon as I type:
"spe1" (without the need to add a space or dot at the end)
"spe1.grl" (without the need to add a space or dot at the end)
Thank you for your help
After palying a bit with an online Regex tool I have find the answer to my own question. I am posting it in case someone needs it:
'\\b(?!' + this.stringToIgnore + '\\b)\\w+';
or
''\\b(?!' + this.stringToIgnore + '|' + this.stringToIgnore.toUpperCase + '\\b)\\w+';
in case the user has the uppercase key on.
I am trying to clean text strings containing any ' or ' (which includes an ; but if i add it here you will see just ' again. Because the the ANSI is also encoded by stackoverflow. The string content contains ' and when it does there is an error.
when i insert the string to my database i get this error:
psycopg2.ProgrammingError: syntax error at or near "s"
LINE 1: ...tment and has commenced a search for mr. whitnell's
the original string looks like this:
...a search for mr. whitnell's...
To remove the ' and ' ; I use:
stripped_content = stringcontent.replace("'","")
stripped_content = stringcontent.replace("' ;","")
any advice is welcome, best regards
When you try to replace("' ;","") it literally searching for "' ;" occurrences in string. You need to convert "' ;" to its character equivalent. Try this:
s = "That's how we 'roll"
r = s.replace(chr(int('''[2:])), "")
and with this chr(int('''[2:])) you'll get ' character.
Output:
Thats how we roll
Note
If you try to run this s.replace(chr(int('''[2:])), "") without saving your result in variable then your original string would not be affected.
Is there a nice way to assign a string to a variable when it spans many lines?
The reason for this is I have some large SQL statements (which I want in the pas) but it's annoying like this
var
sql : string;
begin
sql := 'SELECT * ' +
'FROM foo ' +
'WHERE `this`=0';
That is annoying to copy and paste into terminal / another program because I have to remove the ' and ' + etc.
Is there a way to so something like...
var
sql : string;
begin
sql := ""SELECT *
FROM foo
WHERE `this`=0"";
So some way to assign a block of text/string with new lines without having to concat it.
As there is no way of expressing strings in this way in SQL, I normally use the RegEx search and replace available in the Delphi IDE to format strings in the required way.
SELECT *
FROM foo
WHERE `this`=0
This replaces any line with the line enclosed in quotes, followed by + sLineBreak +
sql :=
' SELECT *' + sLineBreak +
' FROM foo' + sLineBreak +
' WHERE `this`=0' + sLineBreak +
I then just tidy up the last line:
sql :=
' SELECT *' + sLineBreak +
' FROM foo' + sLineBreak +
' WHERE `this`=0';
Of course the same can be done with any preceding or trailing text, such as qry.SQL.Add('\0');
Your question is:
Can a Delphi string literal span multiple lines?
The answer is no.
Not that I know of (at least not out of the box). Anyway, you might want to take a look at this:
How to assign a multiline string value without quoting each line?
You could keep the SQL in a component which has a TStrings property like TSQLQuery, but my solution for longer / complex statements is to keep an 'example' copy as a source code comment, which has actual parameters to make tests easier, and keep both version in sync.
If you like the way C# does it (like I do), then don't forget to vote for this QC report:
http://qc.embarcadero.com/wc/qcmain.aspx?d=2012
It suggests to make your example look like this:
var
sql : string;
begin
sql := #'SELECT *
FROM foo
WHERE `this`=0';
If you install GExperts in Delphi, the IDE will automatically insert a '+ after pressing >enter< if you're inside a string and haven't closed it yet.
Download link: http://www.gexperts.org/download.html