Convert name and Value to column and rows during Export to CSV in Powershell - excel

I am trying to convert name and values returned from my API call into column and rows during export but so far I have no luck.
$searchResponse = Invoke-RestMethod -Method POST -Uri $searchUri -ContentType application/json -Header $requestHeader -Body $searchResultsRequest
$lists = $searchResponse.businessO.fields | Select name, value
The $list returns:
name value
---- ----
Name YY
Address ABC street
City Seattle
Zip 01256
Name XYZ
Address XYZ street
City XYZ
Zip 45456
Name Mike
Address 1256 Street
City New York
Zip 78965
I want to output this result as following in CSV:
Name Address City Zip
YY ABC street Seattle 01256
.
.
.
I tried looping through list and applied condition to check name and populate value based on it, but i end up getting either duplicates or all my output data are out of sysnc meaning Mike gets Address of YY and so on....
If($lists.count -ge 0){
ForEach($list in $lists){
if($list.name -eq "Name") {$name= $list.value }
}
I would really appreciate any help on this one. Thank You.
$searchResponse results
$searchResponse |ConvertTo-Json
{
"businessObjects": [
{
"busObId": "xxxxxxxxxxxxxx",
"busObPublicId": "abc345",
"busObRecId": "xxxxxxxxxxxxxxxxxxx",
"fields": " ",
"links": "",
"errorCode": null,
"errorMessage": null,
"hasError": false
},
{
"busObId": "xxxxxxxxxxxxx",
"busObPublicId": "rty567",
"busObRecId": "xxxxxxxxxxxxxxxx",
"fields": " ",
"links": "",
"errorCode": null,
"errorMessage": null,
"hasError": false
}
],
"hasPrompts": false,
"links": [
],
"prompts": [
],
"searchResultsFields": [
],
"totalRows": 2500
}
Fields has the name and value which I want to output.
$searchResponse.businessObjects.fields |ConvertTo-Json
[
{
"dirty": false,
"displayName": "Name",
"fieldId": "xxxxxxxxxxx",
"html": null,
"name": "Name",
"value": "Mike"
},
{
"dirty": false,
"displayName": "Address",
"fieldId": "456451212113132",
"html": null,
"name": "Address",
"value": "Seattle"
},
and so on.

This answer as a teaching exercise with the expectation, one will walk through the code in the VSCode or ISE debugger and learn more about the objects, members, and more.
There are a few ways to solve this. In this answer, I try to break down the steps in a way newer PowerShell user may learn more about the language.
# mock your data
$list1 = #(
[pscustomobject]#{Name='YY'},
[pscustomobject]#{Address='ABC street'},
[pscustomobject]#{City='Seattle'},
[pscustomobject]#{Zip='01256'},
[pscustomobject]#{Name='XYZ'},
[pscustomobject]#{Address='XYZ street'},
[pscustomobject]#{City='XYZ'},
[pscustomobject]#{Zip='45456'},
[pscustomobject]#{Name='Mike'},
[pscustomobject]#{Address='1256 Street'},
[pscustomobject]#{City='New York'},
[pscustomobject]#{Zip='78965'}
)
# mock your data
$list2 = #(
#{Name='YY'},
#{Address='ABC street'},
#{City='Seattle'},
#{Zip='01256'},
#{Name='XYZ'},
#{Address='XYZ street'},
#{City='XYZ'},
#{Zip='45456'},
#{Name='Mike'},
#{Address='1256 Street'},
#{City='New York'},
#{Zip='78965'}
)
# debuggng...
#$list1
#$list1.GetType()
#$list1[0].GetType()
#$list2
#$list2.GetType()
#$list2[0].GetType()
# seems your data looks like list1
$data = #()
$entry = [ordered]#{}
# transform the data into separate objects based on the field you want
foreach ($item in $list)
{
if ($item.Name) {$entry.Name = $item.Name}
elseif ($item.Address) {$entry.Address = $item.Address}
elseif ($item.City) {$entry.City = $item.City}
elseif ($item.Zip) {$entry.Zip = $item.Zip; $data += $entry; $entry = [ordered]#{}}
}
# data transformed now into an array of hashtables
# there are a few says to save as csv, one is to just roll your own rather
# than GetEnumerator(), etc.
$out = #()
$out += $data[0].Keys -join ', '
$out += $data | % {
#$_['Name']+','+$_['Address']+','+$_['City']+','+$_['Zip']
# or
#$_.Name+','+$_.Address+','+$_.City+','+$_.Zip
# or
$_.Name,$_.Address,$_.City,$_.Zip -join ', '
}
$out
# save $out to a file, uncomment to use after editing path
#$out | Out-File -FilePath 'c:\mycsvfile.csv'

Related

Is there a way to draw arrows from/to table rows in PlantUML?

If you had an ER-Diagram in PlantUML, stylized this way using tables, is there a way to connect the table rows with arrows?
#startuml
entity User {
<#transparent,#transparent>| <color:gold><&key> | <b>id |<r> bigint |
| | name |<r> varchar |
}
entity Email {
<#transparent,#transparent>| <color:gold><&key> | <b>id |<r> bigint |
| <color:purple><&link-intact> | <i>user_id |<r> bigint |
| | address |<r> varchar |
}
User::id -> Email::user_id
#enduml
I know you can get exactly what I want using fields, but the way it looks is a bit boring and simple.
Also, there is a bit of micromanagement by putting the tabs characters.
#startuml
entity User {
#id \t\t bigint
name \t varchar
}
entity Email {
#id \t\t bigint
+user_id \t bigint
address \t varchar
}
User::id -> Email::user_id
#enduml
There is a workaround that will not involve the tab micromanagement in the member/fields approach.
You can design a procedure, to take care of the tabs characters for you. The only thing left is to replace the member visibility icons (something I do not know how to do).
This approach requires the table data to be store as a JSON (one per each table).
For example:
table.iuml
#startuml
!global $tab_size = 4 ' The tab size in plantuml is 4 chars wide
' Returns the size in tabs, rounded down, that the $data has
!function $tab_size($data)
!return %strlen($data)/$tab_size
!endfunction
' Get the maximum number of tabs from the $data array
!function $get_max_tabs($data)
!local $tabs = 0
!foreach $field in $data
!local $field_tabs = $tab_size($field.name)
!if ($tabs < $field_tabs)
!local $tabs = $field_tabs
!endif
!endfor
!return $tabs
!endfunction
' Get the field name followed by the "perfect" number of tabs
!function $get_name($name, $max_tabs)
!local $tabs = "\t"
!local $length = $tab_size($name)
!while $max_tabs > $length
!local $tabs = $tabs + "\t"
!local $length = $length + 1
!endwhile
!return $name + $tabs
!endfunction
' Create a entity member for the table field
!procedure field($data, $max_tabs)
!local $field = $get_name($data.name, $max_tabs) + $data.type
!if %json_key_exists($data, "index")
!if $data.index == "primary"
# $field
!elseif $data.index == "unique"
~ $field
!elseif $data.index == "simple"
+ $field
!endif
!else
$field
!endif
!endprocedure
' Process the JSON data to create a plantuml entity
!procedure table($data)
!local $tabs = $get_max_tabs($data.fields)
entity $data.name {
!foreach $field in $data.fields
field($field, $tabs)
!endfor
}
!endprocedure
#enduml
database.puml
#startuml
!include table.iuml
!$user = %load_json("user.json")
table($user)
!$email = %load_json("email.json")
table($email)
user::id -> email::user_id
#enduml
user.json
{
"name": "user",
"fields": [
{
"name": "id",
"type": "bigint",
"index": "primary"
},
{
"name": "name",
"type": "varchar"
}
]
}
email.json
{
"name": "email",
"fields": [
{
"name": "id",
"type": "bigint",
"index": "primary"
},
{
"name": "user_id",
"type": "bigint"
},
{
"name": "address",
"type": "varchar"
}
]
}
Obtaining the following diagram

Azure Devops : Replace variable in list with unknown order

We are running a build pipeline in azure DevOps. After the build we deploy the stuff and substitute variables in JSON files.
I have a problem substituting a variable in a list of equal objects. My son looks like this:
"Friends": [
{ "Name": "John",
"Phone": "12345678"
},
{ "Name": "Frank",
"Phone": "12235578"
},
{ "Name": "Bill",
"Phone": "13790678"
}
]
I can substitute Franks phone using the path: Friends.1.Phone
What should I do, if the order of "Friends" is not determined and I can not use the index?
I searched for the azure plugin "variable substitution condition" but I couldn't find any useful task.
You can use a powershell task and use Convertfrom-Json command
$Json = #'
{
"Friends": [
{ "Name": "John",
"Phone": "12345678"
},
{ "Name": "Frank",
"Phone": "12235578"
},
{ "Name": "Bill",
"Phone": "13790678"
}
]
}
'#
$result = $Json | ConvertFrom-Json
$result.Friends[0].Phone='12345'
$result | ConvertTo-Json -Depth 4
This will change 'John' phone number.
Answer 2:
$result = $Json | ConvertFrom-Json
$result.Friends | % {if($_.name -eq 'john'){$_.phone=98765}}
$result | ConvertTo-Json -depth 10

Accessing unknown levels of PSCustomObject nested object

I'm getting a response from an API with unknown nested levels of properties, this is an example:
affects_rating : True
assets : {#{asset=xxxxxxxxxxxxx; identifier=; category=low; importance=0.0; is_ip=True}}
details : #{check_pass=; diligence_annotations=; geo_ip_location=NL; grade=GOOD; remediations=System.Object[]; vulnerabilities=System.Object[]; dest_port=443; rollup_end_date=2021-06-06;
rollup_start_date=2020-03-18}
evidence_key : xxxxxxxxx:xxxx
first_seen : 2020-03-18
last_seen : 2021-06-06
related_findings : {}
risk_category : Diligence
risk_vector : open_ports
risk_vector_label : Open Ports
rolledup_observation_id : xxxx-xxxx==
severity : 1.0
severity_category : minor
tags : {}
asset_overrides : {}
duration :
comments :
remaining_decay : 59
temporary_id : xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
affects_rating : True
assets : {#{asset=xxxx.xxxx.com; identifier=; category=low; importance=0.0002340946; is_ip=False}, #{asset=xxxx.xxxx.com; identifier=; category=critical; importance=0.45131093; is_ip=False},
So far I've tried to access each value with a table, but sometimes the records contains an object, which outputs System.Object[] to the CSV file.
foreach ($item in $findings.results) {
$tabledata = [ordered]#{
temporary_id = $item.temporary_id
affects_rating = $item.affects_rating
asset = $item.assets.asset
asset_identifier = $item.assets.identifier
asset_category = $item.assets.category
asset_importance = $item.assets.importance
asset_is_ip = $item.assets.is_ip
modal_data = $item.details.diligence_annotations.modal_data
modal_tags = $item.details.diligence_annotations.modal_tags
server = $item.details.diligence_annotations.server
}
}
The type of the variable $findings is a PSCustomObject
PS C:\Users\bryanar> $findings.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True False PSCustomObject System.Object
Any recommendations?
Let's say I have an object $result full of properties, lists, or other objects:
$result = [pscustomobject]#{Name='Foo';Level='1'
Object= [pscustomobject]#{Name='Bar';Level='2'
List= #(
[pscustomobject]#{User='John Smith';Dept='Accounting'},
[pscustomobject]#{User='Bob Smith';Dept='Accounting'}
)}}
$result | fl
Name : Foo
Level : 1
Object : #{Name=Bar; Level=2; List=System.Object[]}
If you just want to see the whole object for troubleshooting/exploring purposes, I find the easiest way is to convert it to json or xml:
$result | ConvertTo-Json -Depth 10
{
"Name": "Foo",
"Level": "1",
"Object": {
"Name": "Bar",
"Level": "2",
"List": [
{
"User": "John Smith",
"Dept": "Accounting"
},
{
"User": "Bob Smith",
"Dept": "Accounting"
}
]
}
}
If you want to save an object like this, use Export-CLIXML instead of CSV. It's very verbose, but great for when you need to re-use an object since it keeps the type information.

Interacting with AzureDevops using python

I am trying to achieve following things while interacting with AzureDevops using python:
To get the stories under Boards section
To get the branches under Repos section
I am able to establish connection to AzureDevops URL using requests API, but not sure how to read the data using it.
Is there any way by which I can do the same or if there is some other API using which I can do this?
Thanks in advance!!
Agree with Krzysztof Madej,
We could use write a WIQL query to list all User Story, you could refer this link for more details.
from vsts.vss_connection import VssConnection
from msrest.authentication import BasicAuthentication
import json
from vsts.work_item_tracking.v4_1.models.wiql import Wiql
def emit(msg, *args):
print(msg % args)
def print_work_item(work_item):
emit(
"{0} {1}: {2}".format(
work_item.fields["System.WorkItemType"],
work_item.id,
work_item.fields["System.Title"],
)
)
personal_access_token = 'YourPATToken'
organization_url = 'https://dev.azure.com/YourorgName'
# Create a connection to the org
credentials = BasicAuthentication('', personal_access_token)
connection = VssConnection(base_url=organization_url, creds=credentials)
wiql = Wiql(
query="""select [System.Id] From WorkItems """
)
wit_client = connection.get_client('vsts.work_item_tracking.v4_1.work_item_tracking_client.WorkItemTrackingClient')
wiql_results = wit_client.query_by_wiql(wiql).work_items
if wiql_results:
# WIQL query gives a WorkItemReference with ID only
# => we get the corresponding WorkItem from id
work_items = (
wit_client.get_work_item(int(res.id)) for res in wiql_results
)
for work_item in work_items:
print_work_item(work_item)
List all branches, we could use Refs - List and add variable filter to list all branches, If we do not add it, it will also list pull request.
Sample:
List all repo and get repo ID.
GET https://dev.azure.com/{Org name}/{Project name}/_apis/git/repositories?api-version=4.1
List repo branches via repo ID
GET https://dev.azure.com/{Org name}/{Project name}/_apis/git/repositories/{Repo ID}/refs?filter=heads&api-version=4.1
Result:
Update1
Power shell script
#List all branches name
$connectionToken="PAT"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($connectionToken)"))
$BranchDetailURL = "https://dev.azure.com/{Org name}/{Project name}/_apis/git/repositories/{Repo ID}/refs?filter=heads&?api-version=6.0"
$BranchInfo = Invoke-RestMethod -Uri $BranchDetailURL -Headers #{authorization = "Basic $base64AuthInfo"} -Method Get
Write-Host $BranchInfo.value.name
#List all User Story ID
$WorkItemWiqlQuery = "https://dev.azure.com/v-viliu/test_Agile/test/_apis/wit/wiql?api-version=5.1"
$query = "Select [System.Id], [System.Title], [System.State] From WorkItems Where [System.WorkItemType] = 'User Story'"
$body = #{query=$query} | ConvertTo-Json
$WorkItemDetailInfo = Invoke-RestMethod -Uri $WorkItemWiqlQuery -Method Post -ContentType "application/json" -Headers #{Authorization=("Basic $base64AuthInfo")} -Body $body
Write-Host $WorkItemDetailInfo.workItems.id
Result:
To get stories please check this answer but keep in mind to set proper filter [System.WorkItemType] = 'Task'".
To get branches you need to use Refs - List endpoint
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/refs?api-version=6.0
Sample response
{
"value": [
{
"name": "refs/heads/feature/calcApp",
"objectId": "ffe9cba521f00d7f60e322845072238635edb451",
"creator": {
"displayName": "Normal Paulk",
"url": "https://vssps.dev.azure.com/fabrikam/_apis/Identities/ac5aaba6-a66a-4e1d-b508-b060ec624fa9",
"_links": {
"avatar": {
"href": "https://dev.azure.com/fabrikam/_apis/GraphProfile/MemberAvatars/aad.YmFjMGYyZDctNDA3ZC03OGRhLTlhMjUtNmJhZjUwMWFjY2U5"
}
},
"id": "ac5aaba6-a66a-4e1d-b508-b060ec624fa9",
"uniqueName": "dev#mailserver.com",
"imageUrl": "https://dev.azure.com/fabrikam/_api/_common/identityImage?id=ac5aaba6-a66a-4e1d-b508-b060ec624fa9",
"descriptor": "aad.YmFjMGYyZDctNDA3ZC03OGRhLTlhMjUtNmJhZjUwMWFjY2U5"
},
"url": "https://dev.azure.com/fabrikam/7484f783-66a3-4f27-b7cd-6b08b0b077ed/_apis/git/repositories/d3d1760b-311c-4175-a726-20dfc6a7f885/refs?filter=heads%2Ffeature%2FcalcApp"
},
{
"name": "refs/heads/feature/replacer",
"objectId": "917131a709996c5cfe188c3b57e9a6ad90e8b85c",
"creator": {
"displayName": "Normal Paulk",
"url": "https://vssps.dev.azure.com/fabrikam/_apis/Identities/ac5aaba6-a66a-4e1d-b508-b060ec624fa9",
"_links": {
"avatar": {
"href": "https://dev.azure.com/fabrikam/_apis/GraphProfile/MemberAvatars/aad.YmFjMGYyZDctNDA3ZC03OGRhLTlhMjUtNmJhZjUwMWFjY2U5"
}
},
"id": "ac5aaba6-a66a-4e1d-b508-b060ec624fa9",
"uniqueName": "dev#mailserver.com",
"imageUrl": "https://dev.azure.com/fabrikam/_api/_common/identityImage?id=ac5aaba6-a66a-4e1d-b508-b060ec624fa9",
"descriptor": "aad.YmFjMGYyZDctNDA3ZC03OGRhLTlhMjUtNmJhZjUwMWFjY2U5"
},
"url": "https://dev.azure.com/fabrikam/7484f783-66a3-4f27-b7cd-6b08b0b077ed/_apis/git/repositories/d3d1760b-311c-4175-a726-20dfc6a7f885/refs?filter=heads%2Ffeature%2Freplacer"
},

Azure table storage REST Query doesn't show new entries

I'm building a query for a Azure Storage Account(tables in this case). I try to receive the last log entries of the last day.
The filter looks like this in the storage query builder:
{
"filter": [
{
"id": "{ID}",
"andOr": "And",
"field": "TIMESTAMP",
"type": "DateTime",
"operator": ">=",
"value": "datetime'2019-11-18T10:08:41.296Z'",
"timeValue": "Last 24 hours",
"timestampType": "time",
"customTimestamp": {
"lastNumber": 7,
"lastTimeUnit": "Days"
},
"isLocal": true
}
],
"select": null,
"group": {
"id": "{ID}",
"children": [
"{CHILD}"
]
},
"top": null,
"filterType": "Query Builder",
"fileVersion": 0.2
}
This query returnes all the entries from the last day.
I tried to use this filter in the Azure REST API call. in a powershell function like this:
function GetTableEntityAll($TableName) {
$version = "2017-04-17"
$resource = "$tableName()"
$DST = [DateTime]::Today.AddDays(-1)
$table_url = "https://$storageAccount.table.core.windows.net/$resource"+ '?$filter=Timestamp gt datetime' + "'"+ $DST.ToString("s") + "'"
write-host $table_url
$GMTTime = (Get-Date).ToUniversalTime().toString('R')
$stringToSign = "$GMTTime`n/$storageAccount/$resource"
$hmacsha = New-Object System.Security.Cryptography.HMACSHA256
$hmacsha.key = [Convert]::FromBase64String($accesskey)
$signature = $hmacsha.ComputeHash([Text.Encoding]::UTF8.GetBytes($stringToSign))
$signature = [Convert]::ToBase64String($signature)
$headers = #{
'x-ms-date' = $GMTTime
Authorization = "SharedKeyLite " + $storageAccount + ":" + $signature
"x-ms-version" = $version
Accept = "application/json;odata=fullmetadata"
}
$item = Invoke-RestMethod -Method GET -Uri $table_url -Headers $headers -ContentType application/json
return $item.value
}
The URL I generate with this is:
https://{Accountname}.table.core.windows.net/{Tablename}()?$filter=Timestamp gt datetime'2019-11-18T00:00:00'
This doesn't return any records. I also tried to encode the url but that doesns't work either. If I change the datetime to 2016 I receive the first few records.
Azure documentation: https://learn.microsoft.com/en-us/rest/api/storageservices/querying-tables-and-entities
Any ideas?

Resources