Accessing unknown levels of PSCustomObject nested object - excel

I'm getting a response from an API with unknown nested levels of properties, this is an example:
affects_rating : True
assets : {#{asset=xxxxxxxxxxxxx; identifier=; category=low; importance=0.0; is_ip=True}}
details : #{check_pass=; diligence_annotations=; geo_ip_location=NL; grade=GOOD; remediations=System.Object[]; vulnerabilities=System.Object[]; dest_port=443; rollup_end_date=2021-06-06;
rollup_start_date=2020-03-18}
evidence_key : xxxxxxxxx:xxxx
first_seen : 2020-03-18
last_seen : 2021-06-06
related_findings : {}
risk_category : Diligence
risk_vector : open_ports
risk_vector_label : Open Ports
rolledup_observation_id : xxxx-xxxx==
severity : 1.0
severity_category : minor
tags : {}
asset_overrides : {}
duration :
comments :
remaining_decay : 59
temporary_id : xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
affects_rating : True
assets : {#{asset=xxxx.xxxx.com; identifier=; category=low; importance=0.0002340946; is_ip=False}, #{asset=xxxx.xxxx.com; identifier=; category=critical; importance=0.45131093; is_ip=False},
So far I've tried to access each value with a table, but sometimes the records contains an object, which outputs System.Object[] to the CSV file.
foreach ($item in $findings.results) {
$tabledata = [ordered]#{
temporary_id = $item.temporary_id
affects_rating = $item.affects_rating
asset = $item.assets.asset
asset_identifier = $item.assets.identifier
asset_category = $item.assets.category
asset_importance = $item.assets.importance
asset_is_ip = $item.assets.is_ip
modal_data = $item.details.diligence_annotations.modal_data
modal_tags = $item.details.diligence_annotations.modal_tags
server = $item.details.diligence_annotations.server
}
}
The type of the variable $findings is a PSCustomObject
PS C:\Users\bryanar> $findings.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True False PSCustomObject System.Object
Any recommendations?

Let's say I have an object $result full of properties, lists, or other objects:
$result = [pscustomobject]#{Name='Foo';Level='1'
Object= [pscustomobject]#{Name='Bar';Level='2'
List= #(
[pscustomobject]#{User='John Smith';Dept='Accounting'},
[pscustomobject]#{User='Bob Smith';Dept='Accounting'}
)}}
$result | fl
Name : Foo
Level : 1
Object : #{Name=Bar; Level=2; List=System.Object[]}
If you just want to see the whole object for troubleshooting/exploring purposes, I find the easiest way is to convert it to json or xml:
$result | ConvertTo-Json -Depth 10
{
"Name": "Foo",
"Level": "1",
"Object": {
"Name": "Bar",
"Level": "2",
"List": [
{
"User": "John Smith",
"Dept": "Accounting"
},
{
"User": "Bob Smith",
"Dept": "Accounting"
}
]
}
}
If you want to save an object like this, use Export-CLIXML instead of CSV. It's very verbose, but great for when you need to re-use an object since it keeps the type information.

Related

json file in terraform

I have a JSON file with the following and trying to list the shapes ex: "t3-nano, t3-micro, t3-small, t2.medium, t3-2xlarge, r6g-medium".
json file = info.json
{
"t3-nano" : {
"service_name" : "t3",
"existing" : 100
},
"t3-micro" : {
"service_name" : "t3",
"existing" : 1
},
"t3-small" : {
"service_name" : "t3",
"existing" : 2
},
"t2.medium" : {
"service_name" : "t2",
"existing" : 0
},
"t3-2xlarge" : {
"service_name" : "t3-2",
"existing" : 5
},
"r6g-medium" : {
"service_name" : "r6g.medium",
"existing" : 10
}
}
I tried the following
locals {
service_name = flatten([for i in local.info : i[*].service_name])
shapes = flatten([for i in local.info : i[*].index])
}
and it got failed
Error: Unsupported attribute
This object does not have an attribute named "index".
I was expecting to print shapes = [t3-nano, t3-micro, t3-small, t2.medium, t3-2xlarge, r6g-medium]. Can someone help if there is a way to just list the shapes?
The flatten function and for expression are both unnecessary here. The function keys already has the functionality and return value that you want to achieve:
shapes = keys(local.info)
and that will assign the requested value.

print from json with a given condition in terraform

I'm into the terraform world recently and learning based on the requirements. I've a question on printing the values with a given condition
json file:
{
"team1" : [
{
"engg_name" : "Alex",
"guid" : 1001,
"scope" : "QA"
},
{
"engg_name" : "Trex",
"guid" : 1002,
"scope" : "QA"
},
{
"engg_name" : "Jessica",
"guid" : 1003,
"scope" : "QA"
},
{
"engg_name" : "Tom",
"guid" : 1004,
"scope" : "DEV"
}
],
"team2" : [
{
"engg_name" : "Roger",
"guid" : 2001,
"scope" : "DEV"
},
{
"engg_name" : "Jhonny",
"guid" : 2002,
"scope" : "DEV"
}
]
}
What I'm trying:
print the engg whose scope is DEV from the json file
locals {
teams = jsondecode(file("${path.module}/teams_info.json"))
engg_with_scope_dev = flatten([for i in local.teams : i.teams if keys(local.teams).scope == "DEV"])
}
Error:
engg_with_scope_dev = flatten([for i in local.teams : i.teams if keys(local.teams).scope == "DEV"])
|----------------
| local.teams is object with 2 attributes
This value does not have any attributes.
Can someone suggest me what's the right way to just print based on the condition?
output must be as following:
engg_with_scope_dev = ["Tom", "Roger", "Jhonny"]
You need an embedded for loop for this:
locals {
teams = jsondecode(file("${path.module}/teams_info.json"))
engg_with_scope_dev = flatten([for team in local.teams : [
for engineer in team : engineer.engg_name if engineer.scope == "DEV"
]])
}
Other solution would be to use a concatenation of the lists with ellipsis operator:
locals {
teams = jsondecode(file("${path.module}/teams_info.json"))
engg_with_scope_dev = ([
for engineer in concat(values(local.teams)...) : engineer.engg_name if engineer.scope == "DEV"
])
}
But also, a simple flatten with values would work as well:
locals {
teams = jsondecode(file("${path.module}/teams_info.json"))
engg_with_scope_dev = ([
for engineer in flatten(values(local.teams)) : engineer.engg_name if engineer.scope == "DEV"
])
}

Unnesting map values as individual columns in Athena / presto

My question is somewhat similar to this ( Athena/Presto - UNNEST MAP to columns ). But in my case, I know what columns I need before hand.
My use case is this
I have a json blob which contains the following structures
{
"reqId" : "1234",
"clientId" : "client",
"response" : [
{
"name" : "Susan",
"projects" : [
{
"name" : "project1",
"completed" : true
},
{
"name" : "project2",
"completed" : false
}
]
},
{
"name" : "Adams",
"projects" : [
{
"name" : "project1",
"completed" : true
},
{
"name" : "project2",
"completed" : false
}
]
}
]
}
I need to create a view which will return output something like this
name | project | Completed |
----------+-------------+------------+
Susan | project1 | true |
Susan | project2 | false |
Adams | project1 | true |
Adams | project2 | false |
I tried the following and other approaches. This one was the closest I can get
WITH dataset AS (
SELECT 'Susan' as name, transform(filter(CAST(json_extract('{
"projects": [{"name":"project1", "completed":false}, {"name":"project3", "completed":false},
{"name":"project2", "completed":true}]}', '$.projects') AS ARRAY<MAP<VARCHAR, VARCHAR>>), p -> (p['name'] != 'project1')), p -> ROW(map_values(p))) AS projects
)
SELECT * from dataset
CROSS JOIN UNNEST(projects)
This is the output I am getting
name projects _col2
1 Susan [{field0=[project3, false]}, {field0=[project2, true]}] {field0=[project3, false]}
2 Susan [{field0=[project3, false]}, {field0=[project2, true]}] {field0=[project2, true]}
I basically want to unnest the key-value pairs of my map as separate columns. How do I do this in presto / Athena?
Your JSON example seems to be invalid, it misses a , after "name" : "Susan" and "name" : "Adams". Besides that, you can achieve your expected output by this query, you need to UNNEST two times and also requires some casting:
with dataset as
(
select json_parse('{"reqId" : "1234","clientId" : "client","response" : [{"name" : "Susan","projects" : [{"name" : "project1","completed" : true},{"name" : "project2","completed" : false}]},{"name" : "Adams","projects" : [{"name" : "project1","completed" : true},{"name" : "project2","completed" : false}]}]}') as json_col
)
,unnest_response as
(
select *
from dataset
cross join UNNEST(cast(json_extract(json_col, '$.response') as array<JSON>)) as t (response)
)
select
json_extract_scalar(response, '$.name') name,
json_extract_scalar(project, '$.name') project_name,
json_extract_scalar(project, '$.completed') project_completed
from unnest_response
cross join UNNEST(cast(json_extract(response, '$.projects') as array<JSON>)) as t (project);

adding new documents not being show in ElasticSearch index

I am new to ElasticsSearch and was messing around with it today. I have a node running on my localhost and was creating/updating my cat index. As I was adding more documents into my cat indexes, I noticed that when I do a GET request to see all of the documents in Postman, the new cats I make are not being added. I started noticing the issue after I added my tenth cat. All code is below.
ElasticSearch Version: 6.4.0
Python Version: 3.7.4
my_cat_mapping = {
"mappings": {
"_doc": {
"properties": {
"breed": { "type": "text" },
"info" : {
"cat" : {"type" : "text"},
"name" : {"type" : "text"},
"age" : {"type" : "integer"},
"amount" : {"type" : "integer"}
},
"created" : {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
}
}
}
}
}
cat_body = {
"breed" : "Persian Cat",
"info":{
"cat":"Black Cat",
"name": " willy",
"age": 5,
"amount": 1
}
}
def document_add(index_name, doc_type, body, doc_id = None):
"""Funtion to add a document by providing index_name,
document type, document contents as doc and document id."""
resp = es.index(index=index_name, doc_type=doc_type, body=body, id=doc_id)
print(resp)
document_add("cat", "cat_v1", cat_body, 100 )
Since the document id is passed as 100 it just updates the same cat document. I'm assuming its not changed on every run !?
You have to change the document id doc_id with every time to add new cat instead of updating existing ones.
...
cat_id = 100
cat_body = {
"breed" : "Persian Cat",
"info":{
"cat":"Black Cat",
"name": " willy",
"age": 5,
"amount": 1
}
}
...
document_add("cat", "cat_v1", cat_body, cat_id )
With this you can change both cat_id and cat_body to get new cats.

Convert name and Value to column and rows during Export to CSV in Powershell

I am trying to convert name and values returned from my API call into column and rows during export but so far I have no luck.
$searchResponse = Invoke-RestMethod -Method POST -Uri $searchUri -ContentType application/json -Header $requestHeader -Body $searchResultsRequest
$lists = $searchResponse.businessO.fields | Select name, value
The $list returns:
name value
---- ----
Name YY
Address ABC street
City Seattle
Zip 01256
Name XYZ
Address XYZ street
City XYZ
Zip 45456
Name Mike
Address 1256 Street
City New York
Zip 78965
I want to output this result as following in CSV:
Name Address City Zip
YY ABC street Seattle 01256
.
.
.
I tried looping through list and applied condition to check name and populate value based on it, but i end up getting either duplicates or all my output data are out of sysnc meaning Mike gets Address of YY and so on....
If($lists.count -ge 0){
ForEach($list in $lists){
if($list.name -eq "Name") {$name= $list.value }
}
I would really appreciate any help on this one. Thank You.
$searchResponse results
$searchResponse |ConvertTo-Json
{
"businessObjects": [
{
"busObId": "xxxxxxxxxxxxxx",
"busObPublicId": "abc345",
"busObRecId": "xxxxxxxxxxxxxxxxxxx",
"fields": " ",
"links": "",
"errorCode": null,
"errorMessage": null,
"hasError": false
},
{
"busObId": "xxxxxxxxxxxxx",
"busObPublicId": "rty567",
"busObRecId": "xxxxxxxxxxxxxxxx",
"fields": " ",
"links": "",
"errorCode": null,
"errorMessage": null,
"hasError": false
}
],
"hasPrompts": false,
"links": [
],
"prompts": [
],
"searchResultsFields": [
],
"totalRows": 2500
}
Fields has the name and value which I want to output.
$searchResponse.businessObjects.fields |ConvertTo-Json
[
{
"dirty": false,
"displayName": "Name",
"fieldId": "xxxxxxxxxxx",
"html": null,
"name": "Name",
"value": "Mike"
},
{
"dirty": false,
"displayName": "Address",
"fieldId": "456451212113132",
"html": null,
"name": "Address",
"value": "Seattle"
},
and so on.
This answer as a teaching exercise with the expectation, one will walk through the code in the VSCode or ISE debugger and learn more about the objects, members, and more.
There are a few ways to solve this. In this answer, I try to break down the steps in a way newer PowerShell user may learn more about the language.
# mock your data
$list1 = #(
[pscustomobject]#{Name='YY'},
[pscustomobject]#{Address='ABC street'},
[pscustomobject]#{City='Seattle'},
[pscustomobject]#{Zip='01256'},
[pscustomobject]#{Name='XYZ'},
[pscustomobject]#{Address='XYZ street'},
[pscustomobject]#{City='XYZ'},
[pscustomobject]#{Zip='45456'},
[pscustomobject]#{Name='Mike'},
[pscustomobject]#{Address='1256 Street'},
[pscustomobject]#{City='New York'},
[pscustomobject]#{Zip='78965'}
)
# mock your data
$list2 = #(
#{Name='YY'},
#{Address='ABC street'},
#{City='Seattle'},
#{Zip='01256'},
#{Name='XYZ'},
#{Address='XYZ street'},
#{City='XYZ'},
#{Zip='45456'},
#{Name='Mike'},
#{Address='1256 Street'},
#{City='New York'},
#{Zip='78965'}
)
# debuggng...
#$list1
#$list1.GetType()
#$list1[0].GetType()
#$list2
#$list2.GetType()
#$list2[0].GetType()
# seems your data looks like list1
$data = #()
$entry = [ordered]#{}
# transform the data into separate objects based on the field you want
foreach ($item in $list)
{
if ($item.Name) {$entry.Name = $item.Name}
elseif ($item.Address) {$entry.Address = $item.Address}
elseif ($item.City) {$entry.City = $item.City}
elseif ($item.Zip) {$entry.Zip = $item.Zip; $data += $entry; $entry = [ordered]#{}}
}
# data transformed now into an array of hashtables
# there are a few says to save as csv, one is to just roll your own rather
# than GetEnumerator(), etc.
$out = #()
$out += $data[0].Keys -join ', '
$out += $data | % {
#$_['Name']+','+$_['Address']+','+$_['City']+','+$_['Zip']
# or
#$_.Name+','+$_.Address+','+$_.City+','+$_.Zip
# or
$_.Name,$_.Address,$_.City,$_.Zip -join ', '
}
$out
# save $out to a file, uncomment to use after editing path
#$out | Out-File -FilePath 'c:\mycsvfile.csv'

Resources