I am trying to convert values under 'key' column to a single string delimited with ','
$TheTable = (get-command get-mailbox).Parameters
Command returns:
Key Value
--- -----
ErrorAction System.Management.Automation.ParameterMetadata
IncludeInactiveMailbox System.Management.Automation.ParameterMetadata
Verbose System.Management.Automation.ParameterMetadata
OutVariable System.Management.Automation.ParameterMetadata
I am trying to achieve:
$TheTable = "ErrorAction,IncludeInactiveMailbox,Verbose,OutVariable"
I am completely lost as everything I attempt (foreach loop, .ToString) returns:
System.Collections.Generic.Dictionary`2[System.String,System.Management.Automation.ParameterMetadata],
Is there any way too do that?
To get a hashtable's / dictionary's keys, use its .Keys property.
To convert a collection of strings to a single string with a separator, use the -join operator.
Therefore:
$TheTable = (get-command get-mailbox).Parameters.Keys -join ","
Related
I'm working with a CSV export of identity data containing ~22000 records. One of the fields included is titled 'ExtendedAttributes' and each cell in the column contains a quote bound string of comma separated Key:Value pairs. Each record in the file has an arbitrary number of extended attributes (up to around 50). My ultimate objective is to expand these extended attributes into their own columns in Excel (2016). I already have solutions for the expansions into columns from other data using formulae, simple VBA and most recently Power Query based approaches.
However, my previous solutions have all been based on the Key:Value pairs being simple to delimit. In this export, the ExtendedAttributes field has:
Value data that may contain unescaped/unquoted commas. e.g.
"Key1: Value1, name: surname, forename, Key2: Value2, ... "
Keys that may contain multiple comma separated values, which are also unquoted/unescaped. e.g.
"Key1: Value1, emailAlias: alias1#domain, alias2#domain, alias3#domain, Key2: Value2, ... "
My usual approach to this, where Key:Value pairs don't have these problems would be to delimit using commas to break it into the key value pairs, transpose the data into rows, then delimit using the colon to populate my new columns and their values as described here in the PowerBI community pages
This doesn't work here because delimiting using a comma breaks the values.
Is there a straightforward way to parse this into the constituent Key:Value pairs using (ideally) Power Query? Happy to also go with VBA or formula based solutions.
My instinctive approach would be to try and identify substrings containing a colon and prepend them with a unique character, which can then be used as a delimiter. (It's not impossible that the data may also include unescaped colons, but I'm happy to assume that it doesn't) But recognise that this may be a needlessly complex approach and I'm unsure how best to do it.
I'm happy to keep values with multiple comma separated items as a single unit (A problem for me to deal with later).
For the example data:
"Key1: Value1, name: surname, forename, emailAlias: alias1#domain, alias2#domain, alias3#domain, Key2: Value2, ... "
I'd like to end up with something that lets me treat the data like this, using maybe a ! as an example unique character that I could then use as a delimiter:
"Key1: Value1!name: surname, forename!emailAlias: alias1#domain, alias2#domain, alias3#domain!Key2: Value2!..."
I don't have access to the original data (vendor controlled system) and have limited data processing tools on my corporate desktop (Excel 2016, VBA, PQ).
Appreciate any help.
In Power Query, you can define a function Partition as follows:
let
Output = (str as text, sep as text) as text =>
Text.RemoveRange(
Text.Replace(
Text.Combine(
List.Transform(
Text.Split(str, " "),
each if Text.Contains(_, ":") then sep & _ else _
),
" "
), ", " & sep, sep
),
0, Text.Length(sep)
)
in
Output
Example text transformation using separator !
Starting text:
Key1: Value1, name: surname, forename, emailAlias: alias1#domain
Split the string based on spaces into a list
Key1:
Value1,
name:
surname,
forename,
emailAlias:
alias1#domain
Prepend any list items containing : with separator !
!Key1:
Value1,
!name:
surname,
forename,
!emailAlias:
alias1#domain
Combine the list back into a string
!Key1: Value1, !name: surname, forename, !emailAlias: alias1#domain
Replace , ! with !
!Key1: Value1!name: surname, forename!emailAlias: alias1#domain
Remove the first separator
Key1: Value1!name: surname, forename!emailAlias: alias1#domain
Once you have this function defined, you can call it in a column transformation that would look something like
= Table.TransformColumns(#"Prev Step", {{"ColName", each Partition(_,"!") , type text}})
You do not say anything about the file records not being counted in the "ExtendedAttribute field" category... I prepared a function able to separate that area which you put in discussion. Please, use the next code:
Function separateKeys(x As String, sep As String) As String
Dim arr1, arr2, i As Long, k As Long
arr1 = Split(x, ": ")
ReDim arr2(UBound(arr1))
For i = 0 To UBound(arr1) - 1
If arr1(i + 1) = arr1(UBound(arr1)) Then Exit For
arr2 = Split(arr1(i + 1), " ")
arr2(UBound(arr2) - 1) = Replace(arr2(UBound(arr2) - 1), ",", sep)
arr1(i + 1) = Join(arr2, " ")
Next
separateKeys = Replace(Join(arr1, ":"), sep & " ", sep)
End Function
The above function can (probably) be adapted in a way to skip calculations for rest of the file, or also transform each comma in the sep character (simple using Replace).
In order to test the above function, please use the next testing Sub:
Sub testSepKeys()
Dim x As String, sep As String
sep = "|" 'you can try something else, but improbable to appear in the processed text
x = "Key1: Value1, name: surname, forename, emailAlias: alias1#domain, alias2#domain, alias3#domain, Key2: Value2, Value3, key3: Val1, Val2"
Debug.Print separateKeys(x, sep)
End Sub
Like global way of working, I would suggest splitting the file on line separator, then process all the array elements (lines) using the above (adapted) function and finally join it on the line separator.
The newly created file should be open using Workbooks.OpenText, DataType:=xlDelimited, OtherChar:=sep.
Please, test the above function and send some feedback.
I have a powershell string which can contain multiple email address, for example below is a exmaple that contains two email ids. In that i have two scenarios.
1) Scenario One where the #gmail.com is consistent
strEmail=john.roger#gmail.com,smith.david#gmail.com
2) Secnario second: where the #mail.com could be different
strEmail2 = john.roger#gmail.com,smith.david#outlook.com
I need to get rid of anything after # including it.
So for result for
scenario (1) will be: john.roger,smith.david
Scenario (2) will be: john.roger,smith.david
SO for Scenarion(1) i can use replace with "hardcoded" value of "#gmail.com", How about second secnario.
I am looking for some solution which will work for both scenarion... like something in Regex or any other way i don't know.
Splitting and joining would return the names on one line
Following
$strEmail = "john.roger#gmail.com,smith.david#outlook.com"
($strEmail -split "," | % {($_ -split "#")[0]}) -join ","
returns
john.roger,smith.david
Breakdown
$strEmail -split "," returns an array of two elements
[0] john.roger#gmail.com
[1] smith.david#outlook.com
% {($_ -split "#")[0]} loops over the array
and splits each item into an array of two elements
[0] john.roger
[1] gmail.com
[0] smith.david
[1] outlook.com
and returns the first element [0] from each array
- join "," joins each returned item into a new string
Both of these should work.
This will print each name on a new line:
$strEmail = "john.roger#gmail.com,smith.david#outlook.com"
$strEmail = $strEmail.Split(',') | Foreach {
$_.Substring(0, $_.IndexOf('#'))
}
$strEmail
This will give you the same output as you outlined above:
$strEmail = "john.roger#gmail.com,smith.david#outlook.com"
$strEmailFinal = ""
$strEmail = $strEmail.Split(',') | Foreach {
$n = $_.Substring(0, $_.IndexOf('#'))
$strEmailFinal = $strEmailFinal + $n + ","
}
$strEmailFinal.TrimEnd(',')
Another approach... Well, if you like RegEx of course
Clear-Host
$SomeEmailAddresses = #'
1) Scenario One where the #gmail.com is consistent
strEmail=john.roger#gmail.com,smith.david#gmail.com
2) Secnario second: where the #mail.com could be different
strEmail2 = john.roger#gmail.com,smith.david#outlook.com
'#
((((Select-String -InputObject $SomeEmailAddresses `
-Pattern '\w+#\w+\.\w+|\w+\.\w+#\w+\.\w+|\w+\.\w+#\w+\.\w+\.\w+' `
-AllMatches).Matches).Value) -replace '#.*') -join ','
Results
john.roger,smith.david,john.roger,smith.david
Just comment out or delete the -join for one per line
I have a CSV file with some integer column, now it 's saved as "" (empty string).
I want to COPY them to a table as NULL value.
With JAVA code, I have try these:
String sql = "COPY " + tableName + " FROM STDIN (FORMAT csv,DELIMITER ',', HEADER true)";
String sql = "COPY " + tableName + " FROM STDIN (FORMAT csv,DELIMITER ',', NULL '' HEADER true)";
I get: PSQLException: ERROR: invalid input syntax for type numeric: ""
String sql = "COPY " + tableName + " FROM STDIN (FORMAT csv,DELIMITER ',', NULL '\"\"' HEADER true)";
I get: PSQLException: ERROR: CSV quote character must not appear in the NULL specification
Any one has done this before ?
I assume you are aware that numeric data types have no concept of "empty string" ('') . It's either a number or NULL (or 'NaN' for numeric - but not for integer et al.)
Looks like you exported from a string data type like text and had some actual empty string in there - which are now represented as "" - " being the default QUOTE character in CSV format.
NULL would be represented by nothing, not even quotes. The manual:
NULL
Specifies the string that represents a null value. The default is \N
(backslash-N) in text format, and an unquoted empty string in CSV format.
You cannot define "" to generally represent NULL since that already represents an empty string. Would be ambiguous.
To fix, I see two options:
Edit the CSV file / stream before feeding to COPY and replace "" with nothing. Might be tricky if you have actual empty string in there as well - or "" escaping literal " inside strings.
(What I would do.) Import to an auxiliary temporary table with identical structure except for the integer column converted to text. Then INSERT (or UPSERT?) to the target table from there, converting the integer value properly on the fly:
-- empty temp table with identical structure
CREATE TEMP TABLE tbl_tmp AS TABLE tbl LIMIT 0;
-- ... except for the int / text column
ALTER TABLE tbl_tmp ALTER col_int TYPE text;
COPY tbl_tmp ...;
INSERT INTO tbl -- identical number and names of columns guaranteed
SELECT col1, col2, NULLIF(col_int, '')::int -- list all columns in order here
FROM tbl_tmp;
Temporary tables are dropped at the end of the session automatically. If you run this multiple times in the same session, either just truncate the existing temp table or drop it after each transaction.
Related:
How to update selected rows with values from a CSV file in Postgres?
Rails Migrations: tried to change the type of column from string to integer
postgresql thread safety for temporary tables
Since Postgres 9.4 you now have the ability to use FORCE_NULL. This causes the empty string to be converted into a NULL. Very handy, especially with CSV files (actually this is only allowed when using CSV format).
The syntax is as follow:
COPY table FROM '/path/to/file.csv'
WITH (FORMAT CSV, DELIMITER ';', FORCE_NULL (columnname));
Further details are explained in the documentation: https://www.postgresql.org/docs/current/sql-copy.html
If we want to replace all blank and empty rows with null then you just have to add emptyasnull blanksasnull in copy command
syntax :
copy Table_name (columns_list)
from 's3://{bucket}/{s3_bucket_directory_name + manifest_filename}'
iam_role '{REDSHIFT_COPY_COMMAND_ROLE}' emptyasnull blanksasnull
manifest DELIMITER ',' IGNOREHEADER 1 compupdate off csv gzip;
Note: It will apply for all the records which contains empty/blank values
I got data like this:
3LLO24MACT01 24MOB_6012010051700000020100510105010 123456
It contains different values for different columns when I import it.
Every column is fixed width:
Col#1 is the ID and just 1 long. Meaning it is "3" here.
Col#2 is 3 in length and here "LLO".
Col#3 is 9 in length and "24MACT01 " (notice that the missing ones gets filled up by blanks).
This goes on for 15 columns or so...
Is there a method to quickly cut it into different elements based on sequence length? I couldn't find any.
This can be done with RegEx matching, and creating an array of custom objects. Something like this:
$AllRecords = Get-Content C:\Path\To\File.txt | Where{$_ -match "^(.)(.{3})(.{9})"} | ForEach{
[PSCustomObject]#{
'Col1' = $Matches[1]
'Col2' = $Matches[2]
'Col3' = $Matches[3]
}
}
That will take each line, match by how many characters are specified, and then create an object based off those matches. It collects all objects in an array and could be exported to CSV or whatever. The 'Col1', 'Col2' etc are just generic column headers I suggested due to a lack of better information, and could be anything you wanted.
Edit: Thank you iCodez for showing me, perhaps inadvertantly, that you can specify a language for your code samples!
[Regex]::Matches will do this rather easily. All you need to do is specify a Regex pattern that has . followed by the number of characters you want in curly braces. For example, to match a column of three characters, you would write .{3}. You then do this for all 15 columns.
To demonstrate, I will use a string that contains the first three columns of your example data (since I know their sizes):
PS > $data = '3LLO24MACT01 '
PS > $pattern = '(.{1})(.{3})(.{9})'
PS > ([Regex]::Matches($data, $pattern).Groups).Value
3LLO24MACT01
3
LLO
24MACT01
PS >
Note that the first value outputted will be the text matched be all of the capture groups. If you do not need this, you can remove it with slicing:
$columns = ([Regex]::Matches($data, $pattern).Groups).Value
$columns = $columns[1..$columns.Length]
New-PSObjectFromMatches is a helper function for creating PS Objects from regex matches.
The -Debug option can help with the process of writing the regex.
I have a string that I want to split up in 2 pieces. The first piece is before the comma (,) and the second piece is all stuff after the comma (including the commas).
I already managed to retrieve the first piece before the comma in the variable $Header, but I don't know how to retrieve the pieces after the first comma in one big string.
$string = "Header text,Text 1,Text 2,Text 3,Text 4,"
$header = $string.Split(',')[0] # $Header = "Header text"
$content = "Text 1,Text 2,Text 3,Text 4,"
# There might be more text then visible here, like say Text 5, Text 6, ..
PowerShell's -split operator supports specifying the maximum number of sub-strings to return, i.e. how many sub-strings to return. After the pattern to split on, give the number of strings you want back:
$header,$content = "Header text,Text 1,Text 2,Text 3,Text 4," -split ',',2
Try something like :
$Content=$String.Split([string[]]"$Header,", [StringSplitOptions]"None")[1]
As you split according to a String, you are using a different signature of the function split.
The basic use needs only 1 argument, a separator character (more info about it can be found here, for instance). However, to use strings, the signature is the following :
System.String[] Split(String[] separator, StringSplitOptions options)
This is why you have to cast your string as an array of string. We use the None option in this case, but you can find the other options available in the split documentation.
Finally, as the value of $Heasder, is at the beggining of your $String, you need to catch the 2nd member of the resulting array.
method of Aaron is the best, but i propose my solution
$array="Header text,Text 1,Text 2,Text 3,Text 4," -split ','
$array[0],($array[1..($array.Length -1)] -join ",")
This alternate solution makes use of PowerShell's ability to distribute arrays to multiple variables with a single assignment. Note, however, that the -split operator splits on every comma and PowerShell's built-in conversion from Array back to String results in the elements being concatenated back together. So it's not as efficient as String.Split, but in your example, it's negligible.
$OFS = ','
$Content = 'Header text,Text 1,Text 2,Text 3,Text 4,'
[String]$Header,[String]$Rest = $Content -split $OFS
$OFS = ' '
Write-Host "Header = $Header"
Write-Host "Rest = $Rest"
Finally, $OFS is a special variable in PowerShell that determines which character will be used when joining the array elements back into a single string. By default, it's a space. But it can be changed to anything.