This is simple enough:
PS C:\Users\saunders\Desktop\data>
PS C:\Users\saunders\Desktop\data> ls .\test.csv
Directory: C:\Users\saunders\Desktop\data
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 2023-01-31 1:38 PM 640 test.csv
PS C:\Users\saunders\Desktop\data>
PS C:\Users\saunders\Desktop\data> cat .\test.csv
UserPrincipalName,"DisplayName","Title","UserType","IsLicensed"
LeeG#lazydev.com,"Lee Gu","jr engineer","Member","True"
MeganB#lazydev.com,"Megan Bowen","recruiter","Member","True"
GradyA#lazydev.com,"Grady Archie","sr engineer","Member","True"
MiriamG#lazydev.com,"Miriam Graham","Director","Member","True"
openmailbox#lazydev.com,"openmailbox",,"Member","False"
JohannaL#lazydev.com,"Johanna Lorenz","Senior Engineer","Member","True"
JoniS#lazydev.com,"Joni Sherman","recruiter","Member","False"
AlexW#lazydev.com,"Alex Wilber","Marketing Assistant","Member","True"
IsaiahL#lazydev.com,"Isaiah Langer","Sales Rep","Member","True"
PS C:\Users\saunders\Desktop\data>
PS C:\Users\saunders\Desktop\data> $test = Import-CSV .\test.csv
PS C:\Users\saunders\Desktop\data>
PS C:\Users\saunders\Desktop\data> $test[3]
UserPrincipalName : MiriamG#lazydev.com
DisplayName : Miriam Graham
Title : Director
UserType : Member
IsLicensed : True
PS C:\Users\saunders\Desktop\data>
But how would CSV formatted data be obtained from a formatted text file?
PS C:\Users\saunders\Desktop\data>
PS C:\Users\saunders\Desktop\data> $records = Get-Content .\records.txt
PS C:\Users\saunders\Desktop\data>
PS C:\Users\saunders\Desktop\data> $records
UserPrincipalName : LeeG#lazydev.com
DisplayName : Lee Gu
Title : jr engineer
UserType : Member
IsLicensed : True
UserPrincipalName : MeganB#lazydev.com
DisplayName : Megan Bowen
Title : recruiter
UserType : Member
IsLicensed : True
UserPrincipalName : GradyA#lazydev.com
DisplayName : Grady Archie
Title : sr engineer
UserType : Member
IsLicensed : True
UserPrincipalName : MiriamG#lazydev.com
DisplayName : Miriam Graham
Title : Director
UserType : Member
IsLicensed : True
UserPrincipalName : openmailbox#lazydev.com
DisplayName : openmailbox
Title :
UserType : Member
IsLicensed : False
UserPrincipalName : JohannaL#lazydev.com
DisplayName : Johanna Lorenz
Title : Senior Engineer
UserType : Member
IsLicensed : True
UserPrincipalName : JoniS#lazydev.com
DisplayName : Joni Sherman
Title : recruiter
UserType : Member
IsLicensed : False
UserPrincipalName : AlexW#lazydev.com
DisplayName : Alex Wilber
Title : Marketing Assistant
UserType : Member
IsLicensed : True
UserPrincipalName : IsaiahL#lazydev.com
DisplayName : Isaiah Langer
Title : Sales Rep
UserType : Member
IsLicensed : True
PS C:\Users\saunders\Desktop\data>
So that the data for each record is transposed and then written to a row in a CSV file. No doubt there's a term for this inverse operation. It doesn't have to be CSV per se, it's just that the above sample originates as CSV.
Please do correct any terminological errors.
as it stands, no the $records object cannot itself be directly exported back to CSV with:
PS C:\Users\saunders\Desktop\data>
PS C:\Users\saunders\Desktop\data> $records = Get-Content .\records.txt
PS C:\Users\saunders\Desktop\data>
PS C:\Users\saunders\Desktop\data> Export-Csv $records
Export-Csv : Cannot convert 'System.Object[]' to the type 'System.String' required by parameter 'Path'. Specified method is not supported.
At line:1 char:12
+ Export-Csv $records
+ ~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Export-Csv], ParameterBindingException
+ FullyQualifiedErrorId : CannotConvertArgument,Microsoft.PowerShell.Commands.ExportCsvCommand
PS C:\Users\saunders\Desktop\data>
as it will first have to parsed somehow.
The text format your showing is a for-display format, as produced by PowerShell's Format-List cmdlet.
As such, it isn't meant for programmatic processing, so there's no standard cmdlet that can parse that format.
For programmatic processing, a structured text format should be used, such as JSON or CSV, or - for the most structural flexibility and best (albeit limited) type fidelity - PowerShell's XML-based CLIXML format, as produced by Export-Clixml and understood by Import-Clixml.
If you're stuck with the given format, you'll have to do your own parsing; e.g.:
# Outputs [pscustomobject] instances that can be exported to CSV, for instance.
((Get-Content -Raw records.txt) -replace ':', '=') -split '\r?\n\r?\n' |
ConvertFrom-StringData |
ForEach-Object { [pscustomobject] $_ }
The above is relatively simple, but:
assumes that none of the property values contain : (this could be worked around) and that all values are single-line; also, the values are subject to interpretation of \ chars. as the start of escape sequence (see cmdlet link in next bullet point).
has one unavoidable drawback: because ConvertFrom-StringData outputs inherently unordered hashtables, the property order isn't preserved.
Potentially fixing this problem in a future PowerShell version (post-7.3.2, current as of this writing) is the subject of GitHub issue #19070
If maintaining the property order is important, and you know the property names, you can pipe to a Select-Object call with the property names listed in the desired order as its -Property argument.
Otherwise, you'll have to parse the text-file lines yourself.
Note: I suggest avoiding the obsolescent, Windows-only ConvertFrom-String cmdlet, despite its allure:
It provides separator-based parsing as well as heuristics-based parsing based on templates containing example values.
The separator-based parsing applies automatic type conversions that you cannot control, and the template language is poorly documented, with the exact behavior being hard to predict (as is unavoidable in a heuristics-based solution).
With Windows PowerShell specifically, you have the option of using ConvertFrom-String's template-based parsing capability:
# define a template using the data from 2 consecutive records from the input data
$recordTemplate = #'
UserPrincipalName : {UserPrincipalName*:LeeG#lazydev.com}
DisplayName : {DisplayName:Lee Gu}
Title : {Title:jr engineer}
UserType : {UserType:Member}
IsLicensed : {IsLicensed:True}
UserPrincipalName : {UserPrincipalName*:MeganB#lazydev.com}
'#
# ConvertFrom-String will use the example(s) in the template to infer the format of the remaining records
Get-Content .\records.txt | ConvertFrom-String -TemplateContent $recordTemplate
While not fully realized, something like
wsl >
wsl > pwd
/mnt/c/WINDOWS/system32/awk
wsl >
wsl > awk '{print $1}' input.txt | datamash transpose
A B C D
wsl >
wsl > awk '{print $2}' input.txt | datamash transpose
2014 2013 2014 2014
wsl >
wsl > awk '{print $3}' input.txt | datamash transpose
1002 990 2030 599
wsl >
wsl > cat input.txt
A 2014 1002
B 2013 990
C 2014 2030
D 2014 599
wsl >
where the matrix is transposed one line at a time per record.
The output above, when concatted, would be:
A B C D
2014 2013 2014 2014
1002 990 2030 599
Which, at first glance at least, would be easily imported as CSV. Not a PowerShell solution, however.
Related
I have the output of an Invoke-SSHCommand command for a host that gives me the following info in the form of a string:
PS C:\Users\user> $t.Output
system> Machine Type-Model Serial Number UUID
-------------- --------- ----
8871AC1 XXXXXXX ABCDEFGHIJKLMNOPQRSTUV
system>
Does anybody have any advice or hints as to how to take the above output and manipulate it to a CSV or something that would give me the functionality of the following:
PS user> $t.Output.UUID
ABCDEFGHIJKLMNOPQRSTUV
PS user> $t.Output.Serial
XXXXXXXX
At the moment, the most I can do with it is below:
PS C:\Users\user> $t.Output.Get(0)
system> Machine Type-Model Serial Number UUID
Any help is appreciated.
Assuming $t.Output is not an object, but rather an unstructured text, hence you can't use ConvertTo-Csv, you first have to create a structured object from your unstructured text. This is what ConvertFrom-String is for.
Example
$tObject = $t.Output | ConvertFrom-String -PropertyNames 'Machine Type-Model', 'Serial Number', 'UUID' | Select-Object -Skip 2
$tObject | ConvertTo-Csv -NoTypeInformation
"Machine Type-Model","Serial Number","UUID"
"8871AC1","XXXXXXX","ABCDEFGHIJKLMNOPQRSTUV"
I looked almost everywhere (there, there, there, there and there) with no luck.
What I have here is a bunch of log files in a directory, where I need to look for a specific ID (myID) and sort the output by date. Here is an example :
in file1.log :
2015-09-26 15:39:50,788 - DEBUG - blabla : {'id' : myID}
in file2.log:
2015-09-26 15:39:51,788 - ERROR - foo : {'id' : myID}
in file3.log:
2015-09-26 15:39:48,788 - ERROR - bar : {'id' : myID}
Exepected output :
2015-09-26 15:39:48,788 - ERROR - bar : {'id' : myID}
2015-09-26 15:39:50,788 - DEBUG - blabla : {'id' : myID}
2015-09-26 15:39:51,788 - ERROR - foo : {'id' : myID}
What I am doing now (and it works pretty well), is :
grep -hri --color=always "myID" | sort -n
The only problem is that with the -h option of grep, the file names are hidden. I'd like to keep the file names AND keep the sorting.
I tried :
grep -ri --color=always "myID" | sort -n -t ":" -k1,1 -k2,2
But it doesn't work. Basically, the grep command outputs the name of the file followed by ":", I'd like to sort the results from this character.
Thanks a lot
Try this:
grep --color=always "myID" file*.log | sort -t : -k2,2 -k3,3n -k4,4n
Output:
file3.log:2015-09-26 15:39:48,788 - ERROR - bar : {'id' : myID}
file1.log:2015-09-26 15:39:50,788 - DEBUG - blabla : {'id' : myID}
file2.log:2015-09-26 15:39:51,788 - ERROR - foo : {'id' : myID}
Another solution, a little bit longer but I think it should work:
grep -l "myID" file* > /tmp/file_names && grep -hri "myID" file* | sort -n > /tmp/grep_result && paste /tmp/file_names /tmp/grep_result | column -s $'\t' -t
What it does basically is, first store files names by:
grep -l "myID" file* > /tmp/file_names
Store grep sorted results:
grep -hri "myID" file* | sort -n > /tmp/grep_result
Paste the results column-wise (using a tab separator):
paste /tmp/file_names /tmp/grep_result | column -s $'\t' -t
The column ordering for sort is 1-based, so k1 will be your filename part. That means that in your attempt, you are sorting by filename, then by date and hour of your log line. Also, the -n means that you are using numeric ordering, which won't be playing nicely with yyyy-mm-dd hh:mm:ss format (it will read yyyy-mm-dd hh as only the first number, i.e. the year).
You can use:
sort -t ":" -k2
Note that I specified column 2 as the start, and left the end blank. The end defaults to the end-of-line.
If you want to sort specific columns, you need to explicitly set the start and end, for example: -k2,2. You can use this to sort out-of-sequence columns, for example -k4,4 -k2,2 will sort by column 4 and use column 2 for tie-breaking.
You could also use -k2,4, which would stop sorting at the colon just before your log details (i.e. it would use 2015-09-26 15:39:48,788 - ERROR - bar)
Finally, perhaps you want to have your log files in a consistent order if the time is the same:
sort -t ":" -k2,4 -k1,1
Try rust-based tool Super Speedy Syslog Searcher
(assuming you have rust installed)
cargo install super_speedy_syslog_searcher
then
s4 file1.log file2.log file3.log | grep "myID"
The only problem is that with the -h option of grep, the file names are hidden. I'd like to keep the file names AND keep the sorting.
You could try
$ s4 --color=never -nw file1.log file2.log file3.log | grep "myID"
file1.log:2015-09-26 15:39:48,788 - ERROR - bar : {'id' : myID}
file2.log:2015-09-26 15:39:50,788 - DEBUG - blabla : {'id' : myID}
file3.log:2015-09-26 15:39:51,788 - ERROR - foo : {'id' : myID}
I have to create a report based on the nagios log file. I am going to write a shell script for this.
The log file is as follows :
[1420520400] CURRENT SERVICE STATE: abc.com;service;CRITICAL;HARD;3;OK : OK : Last on 10-01-2015, Users = 2, Employees = 0
[1420520400] CURRENT SERVICE STATE: def.com;service;CRITICAL;HARD;3;WARNING : Last on 10-01-2015, Users = 2, Employees = 0
[1420520400] CURRENT SERVICE STATE: ghi.com;service;CRITICAL;HARD;3;CRITICAL : Last on 2014-11-19, Users = 2, Employees = 0
From this file, I want to generate the report as follows :
Name : abc.om
Date : 10-01-2015
Users : 2
Employees : 0
Name : def.om
Date : 10-01-2015
Users : 2
Employees : 0
Name : ghi.om
Date : 2014-11-19
Users : 2
Employees : 0
It would be great if anyone help me to achieve this.
This command will give you the above output, from the log file just change the file name from input.log to the actual file name.
$ cat input.log |cut -d';' -f1,6|sed -e 's/\<CURRENT SERVICE STATE\>/NAME=/g'|sed -e 's/\<OK\>//g'|sed -e 's/\<Last on\>/Date =/g'|tr -d ':'|sed 's/WARNING//g'|sed 's/CRITICAL//g'|cut -c 14-|tr -s ' '|tr ',;' '\n'
Output:
Here, I used '=' but you can change the output exactly same as above if you use, following command,
$ cut -d';' -f1,6 input.log|sed -e 's/\<CURRENT SERVICE STATE\>/NAME=/g'|sed -e 's/\<OK\>//g'|sed -e 's/\<Last on\>/Date =/g'|tr -d ':'|sed 's/WARNING//g'|sed 's/CRITICAL//g'|cut -c 14-|tr -s ' '|tr ',;' '\n' |tr '=' ':'
I'm using Windows 2012 powershell to scrape values from a 3rd party executable. My goal is to re-arrange and simplify the output of the tool to display only a subset of the content and allow me to collect the data on many devices at once. I have most of the script working, but I'm stuck on a simple task that's so easy in Linux bash.
The 3rd party program is pulling status of a computer device to standard out. I can successfully set the standard out content to a variable. For example:
PS C:\> $status = mycmd.exe device1 --status
PS C:\> $status
The $status variable would return a multi-line list of values as follows:
Device Number: 1
PCIe slot: 3
Firmware Version: 5.1.4
Temperature: 45C
State: Online
In this case, I would like to create a new variable for the firmware version. In Linux I would use something like this (although there are many options available):
Firmware=$(mycmd device1 --status | grep "Firmware" | cut -c 19-24)
In Powershell I can use the Select-String command to can find the "firmware" pattern and set it to a varible as follows:
$Firmware = $Status | select-string -Pattern "Firmware Version"
This gives me the entire Firmware version line. I can't figure out how to substring just the version number from the line as Powershell seems to only want to manipulate the item I'm searching for and not the content next to the pattern. Since this isn't a built in command with an object name, manipulating text seems much more difficult.
I would like the $Firmware variable to equal "5.1.4" or it could be placed into another variable if necessary.
Firmware = ($Status | Select-String -Pattern '\d{1}\.\d{1,2}\.\d{1,2}' -AllMatches | % { $_.Matches } | % { $_.Value }
$Firmware = ($Status | Select-String -Pattern '(?<=Firmware Version:\s+)[\d.]+').Matches.Value
The regular expression here is looking for 1 or more combinations of digits \d and literal dots which are preceded by the Firmware Version: line.
Note that Select-String returns an object, so we use .Matches.Value to get the actual match (which in this case will only be the number).
Using -replace with a multi-line regex:
$Var =
#'
Device Number: 1
PCIe slot: 3
Firmware Version: 5.1.4
Temperature: 45C
State: Online
'#
$Firmware = $var -replace '(?ms).+^Firmware Version:\s+([0-9.]+).+','$1'
$Firmware
5.1.4
This works...I set used a [system.version] to parse the version number (and remove spaces).
$FirmwareLine = $Status | select-string -Pattern "Firmware Version"| select -expand line
[system.version]$firmware=$firmwareLine -split ":" | select -first 1 -skip 1
If you just need the string, you can remove the "cast"
$firmware=$firmwareLine -split ":" | select -first 1 -skip 1
Another option is to replace the colons with '=', converting the output to key=value pairs and then turn that into a hash table (and then to a PS Object if you want) using ConvertFrom-StringData:
$Var =
#'
Device Number: 1
PCIe slot: 3
Firmware Version: 5.1.4
Temperature: 45C
State: Online
'#
$DeviceStatus = New-object PSObject -Property $(ConvertFrom-StringData $var.Replace(':','='))
$DeviceStatus.'Firmware Version'
5.1.4
I'm currently attempting to pull the version number out of an assemblyinfo.cs file and then append JUST the version number, 1.2.3.4, to another text file.
I've attempted to use the .split() method, but that only works if the item is already a string(from what I've read) and I pull the entirety of the string from the text file by using select-string
Here's what I'm working with
$a = cat "c:\path_to_file\AssemblyInfo.cs" | select-string AssemblyVersion
Which will produce: [assembly: AssemblyVersion("5.7.4.1")]
And then if I try to do $a.split(' " ') I will get
Method invocation failed because [System.Object[]] doesn't contain a method named 'split'.
At line:1 char:21
+ $versionstring.split <<<< (' " ')
+ CategoryInfo : InvalidOperation: (split:String) [], RuntimeException
+ FullyQualifiedErrorId : MethodNotFound
Which as I said, is due to the fact that it's considered as an Object, not a string.
The result that I want is like this though:
PS C:\> $a = '[assembly: AssemblyVersion("5.7.4.1")]'
$b = $a.split(' " ')
$b[2]
5.7.4.1
I'm assuming the issue is regarding the fact that I'm using CAT to display the contents and then pull the string, I'm not sure.
Select-String returns a MatchInfo object. The string value of the line is in the .line property of that object
$a.line.split(' " ')
should give you the result you're looking for.
Use
$a | Get-Member
to see all the available properties that are returned.