I have a fairly straightforward use case, but I can't seem to wrap my head around the shift specification that would make this transpose possible. It's primarily just flattening the tree hierarchy into simple output arrays.
How would a turn this input JSON:
{
"123": [
{
"VALUE_ONE": "Y",
"VALUE_TWO": "12"
},
{
"VALUE_ONE": "N",
"VALUE_TWO": "2"
}
],
"456": [
{
"VALUE_ONE": "Y",
"VALUE_TWO": "35"
}
]
}
Into this output:
[
{
"value_one_new_name": "Y",
"value_two_new_name": "12",
"key": "123"
},
{
"value_one_new_name": "N",
"value_two_new_name": "2",
"key": "123"
},
{
"value_one_new_name": "Y",
"value_two_new_name": "35",
"key": "456"
}
]
NOTE that I don't know what the key ("456", "123" .. etc) would be for each object, so the jolt spec needs to be generic enough to convert any keys, only known field names are "VALUE_ONE" and "VALUE_TWO".
This steps will do the trick:
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"VALUE_ONE": "&2.[&1].value_one_new_name",
"VALUE_TWO": "&2.[&1].value_two_new_name",
"$1": "&2.[&1].key"
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": "[]"
}
}
}
]
Related
Using the following code I was able to map a list of dictionaries by a key
import json
values_list = [{"id" : 1, "user":"Rick", "title":"More JQ"}, {"id" : 2, "user":"Steve", "title":"Beyond"}, {"id" : 1, "user":"Rick", "title":"Winning"}]
result = {}
for data in values_list:
id = data['id']
user = data['user']
title = data['title']
if id not in result:
result[id] = {
'id' : id,
'user' : user,
'books' : {'titles' : []}
}
result[id]['books']['titles'].append(title)
print(json.dumps((list(result.values())), indent=4))
Knowing how clean is Jolt Spec and trying to separate the schema outside of the code.
Is there a way to use Jolt Spec to achieve the same result.
The Result
[
{
"id": 1,
"user": "Rick",
"books": {
"titles": [
"More JQ",
"Winning"
]
}
},
{
"id": 2,
"user": "Steve",
"books": {
"titles": [
"Beyond"
]
}
}
]
You can use three levels of consecutive specs
[
{
"operation": "shift",
"spec": {
"*": {
"*": "#(1,id).&",
"title": "#(1,id).books.&s[]"
}
}
},
{
"operation": "shift",
"spec": {
"*": ""
}
},
{
"operation": "cardinality",
"spec": {
"*": {
"id": "ONE",
"user": "ONE"
}
}
}
]
in the first spec, the common id values are combined by "#(1,id)." expression
in the second spec, the integer keys(1,2) of the outermost objects are removed
in the last spec,only the first of the repeating elements are picked
I have the following Json array:
[ {
"name" : [ "roger", "roger" ],
"state" : [ "primary", "quality" ],
"value" : [ 1, 2 ]
}, {
"name" : [ "david", "david" ],
"state" : [ "primary", "quality" ],
"value" : [ 4, 5 ]
} ]
and I want to have the following Json object result using Jolt
{
"name" : [ "roger", "roger" , "david", "david" ],
"state" : [ "primary", "quality" ,"primary", "quality" ],
"value" : [ 1, 2 , 4, 5]
}
please someone can help me?
You can apply shift transformation twice such as
[
{
"operation": "shift",
"spec": {
"*": {
"*": "&.&1"
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"0": {
"*": "&2[]"
},
"1": {
"*": "&2[]"
}
}
}
}
]
where determine the keys(&) and respective indexes(&1->0 and 1) by prepending ampersand of keys such as "&.&1" in the first step, then dissipate each respective values through use of "*": "&2[]" in which &2 represents going two levels up in order to traverse two curly braces in order to reach the root key to target the each values of the arrays.
I am very new to MongoDB and I need to do a somewhat complex Update operation on my collection.
I have this kind of collection:
[
{
"Id": 1,
"extension": [
{
"keyName": "Name",
"value": "Bob"
},
{
"keyAge": "Age",
"value": 20
}
]
},
{
"Id": 2,
"extension": [
{
"keyName": "Name",
"value": "Sam"
},
{
"key": "Name",
"value": "Sam"
}
]
},
{
"Id": 3,
"extension": [
{
"keyName": "Age",
"value": 25
},
{
"key": "Age",
"value": 25
}
]
},
{
"Id": 4
}
]
I would like to update any items in the extension array of all documents
so that when an item is found with a key property, to rename it keyAge.
Here is the expected result:
[
{
"Id": 1,
"extension": [
{
"keyName": "Name",
"value": "Bob"
},
{
"keyAge": "Age",
"value": 20
}
]
},
{
"Id": 2,
"extension": [
{
"keyName": "Name",
"value": "Sam"
},
{
"keyAge": "Name",
"value": "Sam"
}
]
},
{
"Id": 3,
"extension": [
{
"keyName": "Age",
"value": 25
},
{
"keyAge": "Age",
"value": 25
}
]
},
{
"Id": 4
}
]
I tried to use $rename in a similar way to this question:
MongoDB rename database field within array
but I get the same error $rename source may not be dynamic array
I think this solution might also apply to me, I tried using it but it's not updating anything on my side, so I guess I cannot understand how to apply that answer to me...
https://stackoverflow.com/a/49193743/215553
Thanks for the help!
I tried to use $rename in a similar way to this question: MongoDB rename database field within array but I get the same error $rename source may not be dynamic array
There is a note in $rename:
$rename does not work if these fields are in array elements.
You can try update with aggregation pipeline starting from MongoDB 4.2,
check condition id key field is exists
$map to iterate loop of extension array
$map to iterate loop of array that is converted from extension object to array in key-value format
$cond check condition if k is key then return keyAge otherwise return current
$arrayToObject back to convert key-value array return from above map to object original format
db.collection.update(
{ "extension.key": { $exists: true } },
[{
$set: {
extension: {
$map: {
input: "$extension",
in: {
$arrayToObject: {
$map: {
input: { $objectToArray: "$$this" },
in: {
k: {
$cond: [
{ $eq: ["$$this.k", "key"] }, // check "key" field name
"keyAge", // update new name "keyAge"
"$$this.k"
]
},
v: "$$this.v"
}
}
}
}
}
}
}
}],
{ multi: true }
)
Playground
Question : How to Rename or add new key for existing array object key?
Answer : Inside projection of mongodb query we have map property which will resolve this.
Solution Example :
{
parents: {
$map: {
input: "$parents",
as: "parent",
in: {
caaUserId: "$$parent._id",
email: "$$parent.email",
countryCode: "$$parent.countryCode",
mobile: "$$parent.mobile",
reportType : "single"
}
}
}
}
In this example if we want to rename $parent._id as caaUserId in parents array for each Element.
Then we can use map and define caaUserId like $$parent._id. This whole code will work in mongoose projection of Query.
It should return following :
{
"parents" : [
{
"caaUserId" : "62d17fa164057000149e283f",
"email" : "john.doe#hotmail.com",
"countryCode" : 91,
"mobile": 9876543210,
"reportType":"single",
},
{
"caaUserId" : "6195d50f15ae2b001293c486",
"email" : "akka.ash#hotmail.com",
"countryCode" : 91,
"mobile": 9876543211,
"reportType":"multi",
},
]
}
This is something that works in your case. Might not be the most readable though.
import json
data = json.loads(strdata)
for entry in data:
if 'extension' in entry:
for x in entry['extension']:
for k, v in x.items():
if k == 'key':
x['keyAge'] = x.pop(k)
Hi I am new to JOLT transformation and I am trying to transform some thing like below.
Main goal here is to have a list of objects without making the constant indexing in jolt.
Transformation of different objects to a common list .
Any Help is appreciate .
Data provides here is an example of what I expected.
Input :
{
"CIT": [
{
"name": "name_CIT_1",
"desc": "desc_CIT_1"
},
{
"name": "name_CIT_2",
"desc": "desc_CIT_2"
},
{
"name": "name_CIT_3",
"desc": "desc_CIT_3"
}
],
"BIT": {
"name": "name_BIT",
"desc": "desc_BIT"
},
"NIT": {
"name": "name_NIT",
"desc": "desc_NIT"
},
"KIT": {
"name": "name_KIT",
"desc": "desc_KIT"
}
}
Jolt:
[
{
"operation": "modify-default-beta",
"spec": {
"*": {}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {}
},
{
"operation": "shift",
"spec": {
"CIT": {
"*": {
"name": "CollegeList[0].name",
"desc": "CollegeList[0].desc"
}
},
"BIT": {
"name": "CollegeList[1].name",
"desc": "CollegeList[1].desc"
},
"NIT": {
"name": "CollegeList[2].name",
"desc": "CollegeList[2].desc"
},
"KIT": {
"name": "CollegeList[3].name",
"desc": "CollegeList[3].desc"
}
}
}
]
Output:
{
"CollegeList" : [ {
"name" : [ "name_CIT_1", "name_CIT_2", "name_CIT_3" ],
"desc" : [ "desc_CIT_1", "desc_CIT_2", "desc_CIT_3" ]
}, {
"name" : "name_BIT",
"desc" : "desc_BIT"
}, {
"name" : "name_NIT",
"desc" : "desc_NIT"
}, {
"name" : "name_KIT",
"desc" : "desc_KIT"
} ]
}
Expected Output:
{
"CollegeList": [
{
"name": "name_CIT_1",
"desc": "desc_CIT_1"
},
{
"name": "name_CIT_2",
"desc": "desc_CIT_2"
},
{
"name": "name_CIT_3",
"desc": "desc_CIT_3"
},
{
"name": "name_BIT",
"desc": "desc_BIT"
},
{
"name": "name_NIT",
"desc": "desc_NIT"
},
{
"name": "name_KIT",
"desc": "desc_KIT"
}
]
}
You can use two levels of shift transformations. Indeed, the desired array is obtained within the first level except for the key of the array which is root as default. Then only renaming of the array's key occurs within the second level such as
[
{
"operation": "shift",
"spec": {
"*": "&1"
}
},
{
"operation": "shift",
"spec": {
"#(0,&)": "CollegeList"
}
}
]
Another approach for the same :
[
{
"operation": "shift",
"spec": {
"CIT": {
"*": "CollegeList[]"
},
"*": "CollegeList[]"
}
}
]
I want to return all values in the array that match the criteria.
but when I query, It returns only one result.
This is the value of my DB.
[{
"channel": "a",
"video": [
{
"name": 1
"status": ''
},
{
"name": 2
"status": 'err'
},
{
"name": 3
"status": 'err'
}
]
},
{
"channel": "b",
"video": [
{
"name": 4
"status": 'err'
},
{
"name": 5
"status": 'err'
},
{
"name": 6
"status": ''
}
]
}]
I want a get result like this
[
{
"channel": "a",
"video": [
{
"name": 2
"status": 'err'
},
{
"name": 3
"status": 'err'
}
]
},
{
"channel": "b",
"video": [
{
"name": 4
"status": 'err'
},
{
"name": 5
"status": 'err'
}
]
}
]
but when I using my code
var errData = await DB.find({
'video.status' : { $in : 'err' }
},
{
'channel': true,
'video' : {
$elemMatch : {
'status': { $in : 'err' }
}
}
} )
it returns like this
[
{
"channel": "a",
"video": [
{
"name": 2
"status": 'err'
}
]
},
{
"channel": "b",
"video": [
{
"name": 4
"status": 'err'
}
]
},
]
how can I fix it in mongoose(don't use aggregate)?
If it is impossible without using an aggregate, how can I make it by using an aggregate?
would you please help me?
As docs explain:
The $elemMatch operator limits the contents of an field from the query results to contain only the first element matching the $elemMatch condition
That's why you get only one element for each document.
Also, it seems (look here) that is not possible without aggregation.
Using aggregation you can use $filter in this way:
db.collection.aggregate([
{
"$set": {
"video": {
"$filter": {
"input": "$video",
"as": "v",
"cond": {"$eq": ["$$v.status","err"]}
}
}
}
}
])
Example here