I am trying to get the first element in each nested list in a list of lists in JMESpath. Specifically, I am trying variations on:
aws redshift-data get-statement-result \
--id 0de36215-f1db-421f-8704-3242e1a03f21 \
--query 'Records[][0]'
Unfortunately this results in the parent list being firstly flattened, and thus I get just the first element of the first list.
The input data (simplified) is:
{
"Records": [
[{"stringValue": "rdsdb"},{"etc": 1}],
[{"stringValue": "testuser"},{"etc": 100}]
],
"ColumnMetadata": [{"etc": true}],
"TotalNumRows": 2
}
Given that input data, the output I would expect from that:
[
{"stringValue": "rdsdb"},
{"stringValue": "testuser"}
]
How can I get the first element in each nested list in a list of lists?
From what I gather from your comment, you over-simplified your example and you actually after a whole object, and not a single property.
With that requirement, you can achieve it using [*] instead of [], as the first one does not flatten projections, compared to the second one.
So, given the JSON input:
{
"Records": [
[{"stringValue": "rdsdb", "some-property": "foo"},{"etc": 1}],
[{"stringValue": "testuser"},{"etc": 100}]
],
"ColumnMetadata": [{"etc": true}],
"TotalNumRows": 2
}
The query:
Records[*][0]
Yields the expected:
[
{
"stringValue": "rdsdb",
"some-property": "foo"
},
{
"stringValue": "testuser"
}
]
Otherwise, you can filter out all the object in the array of array Records using a filter projection that would look if the key stringValue is not null with [?stringValue]
But because [] is creating a projection, as you raised, you have to stop that projection first, with the pipe operator |, as explained in the linked documentation chapter.
So given:
Records[]|[?stringValue]
This yields the expected:
[
{
"stringValue": "rdsdb"
},
{
"stringValue": "testuser"
}
]
Related
I have an issue where I want to pass a list of vpc_ids to aws_route53_zone while getting the id from a couple of module calls and iterating it from the state file.
The out put format I am using is:
output "development_vpc_id" {
value = [for vpc in values(module.layout)[*] : vpc.id if vpc.environment == "development"]
description = "VPC id for development env"
}
where I get the output like:
"development_vpc_id": {
"value": [
"xxxx"
],
"type": [
"tuple",
[
"string"
]
]
},
instead I want to achieve below:
"developmemt_vpc_id": {
"value": "xxx",
"type": "string"
},
Can someone please help me with the same.
There isn't any automatic way to "convert" a sequence of strings into a single string, because you need to decide how you want to represent the multiple separate strings once you've reduced it into only a single string.
One solution would be to apply JSON encoding so that your output value is a string containing JSON array syntax:
output "development_vpc_id" {
value = jsonencode([
for vpc in values(module.layout)[*] : vpc.id
if vpc.environment == "development"
])
}
Another possibility is to concatenate all of the strings together with a particular character as a marker to separate each one, such as a comma:
output "development_vpc_id" {
value = join(",", [
for vpc in values(module.layout)[*] : vpc.id
if vpc.environment == "development"
])
}
If you expect that this list will always contain exactly one item -- that is, if each of your objects has a unique environment value -- then you could also tell Terraform about that assumption using the one function:
output "development_vpc_id" {
value = one([
for vpc in values(module.layout)[*] : vpc.id
if vpc.environment == "development"
])
}
In this case, Terraform will either return the one element of this sequence or will raise an error saying there are too many items in the sequence. The one function therefore acts as an assertion to help you detect if there's a bug which causes there to be more than one item in this list, rather than just silently discarding some of the items.
nlapi allows to use joins to refer to values in another record or a sublist, when searching for a record. E.g. if there is a record FOO
FOO: {
type: 'rectype',
field1: '111',
sublists:
bar: [
{value: 1,}, {value: 2,}
]
}
I can search for this record with a query, provided there is a join, let's say, bar to the bar sublits:
nlapiSearchRecord('rectype', null,
[
['field', 'equalto', 'bar',],
'and',
['bar.value', 'equalto', '1',],
],
[ new nlobjSearchColumn('anotherfield', null, null), ]
);
This works. And the question is: is there a way to find a record (the one from the example) by specifying more sublist values, like:
nlapiSearchRecord('rectype', null,
[
['field', 'equalto', 'bar',],
'and',
['bar.value', 'equalto', '1',],
'and',
['bar.value', 'equalto', '2',],
],
[ new nlobjSearchColumn('anotherfield', null, null), ]
);
I've tried numerous approaches with different expressions, but it finds nothing with such a query.
A more concrete example is to find a bomrevision, which has specific components and bomquantities (the commented code is just for demonstration purposes of some approaches with the filter expression I've tried):
nlapiSearchRecord('bomrevision', null, [
['isinactive', 'equalto', 'F',],
'and',
['component.item', 'is', '4942',],
'and',
['component.item', 'is', '4936',],
//[
// 'and',
// ['component.bomquantity', 'equalto', '38',],
//],
//'and',
//[
// ['component.item', 'anyof', '4936',],
// 'and',
// ['component.bomquantity', 'equalto', '38',],
//],
],
[ new nlobjSearchColumn('name', null, null), ]
);
Thank you
I understand your question to be "How can I find items where the component list includes multiple specified items?"
The challenge here is that when you join to the component list, NetSuite returns a result for each line in the list. This means any line that contains 'Item 1' will not contain 'Item 2' and vice versa, so no result meets the criteria.
You can work around this using the following approach:
Group the results by 'main' or 'parent' record ('FOO' or 'bomrevision' in your examples).
Use NS_CONCAT to give a field in each result that contains values from all the sublist results. (Note that NS_CONCAT is undocumented, so use at your own risk).
Use summary criteria formulae to inspect the concatenated field for each value of interest.
Example (tested with assembly items and their member items as I don't have access to an account with BOM revisions):
[
["type","anyof","Assembly"],
"AND",
["min(formulatext: NS_CONCAT({memberitem.name}) )","contains","ITEM 1"],
"AND",
["min(formulatext: NS_CONCAT({memberitem.name}))","contains","ITEM 2"]
]
Or, for your 'bomrevision' (untested, adapted from above):
[
["min(formulatext: NS_CONCAT({component.item}) )","contains","4942"],
"AND",
["min(formulatext: NS_CONCAT({component.item}))","contains","4936"]
]
Be aware that under some circumstances NetSuite will default to returning the text of a list field rather than the value. If this is the case (and the values shown in your example are internal IDs as I'm assuming), you may need to replace component.item with component.item.id, or change the search values to the text name instead.
There may be a more straightforward way of doing this so I'd be glad to hear from anyone with a better solution.
Edit in response to request in comments:
Here is the full code of a working example in my account:
var assemblyitemSearch = nlapiSearchRecord("assemblyitem",null,
[
["type","anyof","Assembly"],
"AND",
["min(formulatext: NS_CONCAT({memberitem.name}) )","contains","ITEM 1"],
"AND",
["min(formulatext: NS_CONCAT({memberitem.name}))","contains","ITEM 2"]
],
[
new nlobjSearchColumn("itemid",null,"GROUP").setSort(false),
new nlobjSearchColumn("displayname",null,"GROUP"),
new nlobjSearchColumn("salesdescription",null,"GROUP"),
new nlobjSearchColumn("formulatext",null,"MIN").setFormula("NS_CONCAT({memberitem})")
]
);
This is an Item search filtering by Member Items, as the closest thing in my account to the BOM Revisions in the example. The only change I have made is to replace the item names with "ITEM 1" and "ITEM 2". You would need to replace these with relevant items from your system.
I can go to a Sales Order page and open developer tools on my browser and add the above code to create a search, then append the following additional code to retrieve results and print out to the console:
var data = [];
assemblyitemSearch.forEach(r => {
var vals = {};
r.rawValues.forEach(v => {
vals[v.name] = v.value
});
data.push(vals);
});
console.table(data);
This returns a table with results exactly as expected.
Edit 2: Adding example using Sales Orders and Items as possibly more consistent between different accounts.
var salesorderSearch = nlapiSearchRecord("salesorder",null,
[
["type","anyof","SalesOrd"],
"AND",
["trandate","within","thismonth"],
"AND",
["min(formulatext: NS_CONCAT({item}))","contains","ITEM 1"],
"AND",
["min(formulatext: NS_CONCAT({item}))","contains","ITEM 2"]
],
[
new nlobjSearchColumn("trandate",null,"GROUP"),
new nlobjSearchColumn("tranid",null,"GROUP"),
new nlobjSearchColumn("formulatext",null,"MIN").setFormula("NS_CONCAT({item})")
]
);
I am new to elasticsearch i want to index a JSON file and perform search queries from elasticsearch
How can I index this json and perform queries to get value if i pass parameter as "field3.innerfield" : "someval"
I have tried indexing this file with helpers.bulk and search but it returns all the fields instead of a selected field.
Below is the JSON sample
[
{
"id": "someid",
"metadata": {
"docType": "value",
"otherfield": " ",
morefields
.
.
},
"field1":"value1",
"field2":"value2,
"field3": [
{
"innerfield": "someval",
"innerfield1":[
"kind of a paragraph"
]
}
],
"field4": [
{
"innerfield": "someval",
"innerfield1": "kind of a paragraph"
}
],
},
{ again the format repeats with different id but same fields
},
{
}
]
Your question lacks clarity however what I understood is that you want to fetch values from its key for a nested json. You can do that in the following way as shown below.
Parse it multiple times and make the required changes as per your need.
import json
data = data.apply(lambda x: json.loads(json.loads(x).get("metadata","{}")).get("doctype") if x else None)
I'm trying to update a field (totalPrice) in my document(s) based on a value in a nested, nested array (addOns > get matching array from x number of arrays > get int in matching array[always at pos 2]).
Here's an example of a document with 2 arrays nested in addOns:
{
"_id": ObjectID('.....'),
"addOns": [
["appleId", "Apples", 2],
["bananaID", "Bananas", 1]
],
"totalPrice": 5.7
}
Let's say the price of bananas increased by 20cents, so I need to look for all documents that have bananas in its addOns, and then increase the totalPrice of these documents depending on the number of bananas bought. I plan on using a updateMany() query using an aggregation pipeline that should roughly look like the example below. The ??? should be the number of bananas bought, but I'm not sure how to go about retrieving that value. So far I've thought of using $unwind, $filter, and $arrayElemAt, but not sure how to use them together. Would appreciate any help!
db.collection('orders').updateMany(
{ $elemMatch: { $elemMatch: { $in: ['bananaID'] } } },
[
{ $set: {'totalPrice': { $add: ['$totalPrice', { $multiply: [{$toDecimal: '0.2'}, ???] } ] } } }
]
I'm not exactly sure whats my mongo version is, but I do know that I can use the aggregation pipeline because I have other updateMany() calls that also use the pipeline without any issues, so it should be (version >= 4.2).
**Edit: Thought of this for the ???, the filter step seems to work somewhat as the documents are getting updated, but the value is null instead of the updated price. Not sure why
Filter the '$addOns' array to return the matching nested array.
For the condition, use $eq to check if the element at [0] ('$$this.0') matches the string 'bananaID', and if it does return this nested array.
Get the array returned from step 1 using $arrayElemAt and position [0].
Use $arrayElemAt again on the array from step 3 with positon [2] as it is the index of the quantity element
{ $arrayElemAt: [{ $arrayElemAt: [ { $filter: { input: "$addOns", as:'this', cond: { $eq: ['$$this.0', 'bananaID'] } } } , 0 ] }, 2 ] }
Managed to solve it myself - though only use this if you don't mind the risk of updateMany() as stated by Joe in the question's comments. It's very similar to what I originally shared in **Edit, except you cant use $$this.0 to access elements in the array.
Insert this into where the ??? is and it'll work, below this code block is the explanation:
{ $arrayElemAt: [{ $arrayElemAt: [ { $filter: { input: "$addOns", as:'this', cond: { $eq: [{$arrayElemAt:['$$this', 0]}, 'bananaID'] } } } , 0 ] }, 2 ] }
Our array looks like this: [["appleId", "Apples", 2], ["bananaID", "Bananas", 1]]
Use $filter to return a subset of our array that only contains the elements that matches our condition - in this case we want the array for bananas. $$this represents each element in input, and we check whether the first element of $$this matches 'bananaID'.
{ $filter: { input: "$addOns", as:'this', cond: { $eq: [{$arrayElemAt:['$$this', 0]}, 'bananaID'] } } }
// how the result from $filter should look like
// [["bananaID", "Bananas", 1]]
Because the nested banana array will always be the first element, we use $arrayElemAt on the result from step 2 to retrieve the element at position 0.
// How it looks like after step 3: ["bananaID", "Bananas", 1]
The last step is to use $arrayElemAt again, except this time we want to retrieve the element at position 2 (quantity of bananas bought)
This is how the final updateMany() query looks like after steps 1-4 are evaluated.
db.collection('orders').updateMany(
{ $elemMatch: { $elemMatch: { $in: ['bananaID'] } } },
[
{ $set: {'totalPrice': { $add: ['$totalPrice', { $multiply: [{$toDecimal: '0.2'}, 1] } ] } } }
], *callback function code*)
// notice how the ??? is now replaced by the quantity of bananas which is 1
Is there a direct way to project a new field if a value matches one in a huge sub array. I know i can use the $elemMatch or $ in the $match condition, but that would not allow me to get the rest of the non matching values (users in my case).
Basically i want to list all type 1 items and show all the users while highlighting the subscribed user. The reason i want to do this through mongodb is to avoid iterating over multiple thousand users for every item. Infact that is the part 2 of my question, can i limit the number of user's array that would be returned, i just need around 10 array values to be returned not thousands.
The collection structure is
{
name: "Coke",
type: 2,
users:[{user: 13, type:1},{ user:2: type:2}]
},
{
name: "Adidas",
type: 1,
users:[{user:31, type:3},{user: 51, type:1}]
},
{
name: "Nike",
type: 1,
users:[{user:21, type:3},{user: 31, type:1}]
}
Total documents are 200,000+ and growing...
Every document has 10,000~50,000 users..
expected return
{
isUser: true,
name: "Adidas",
type: 1,
users:[{user:31, type:3},{user: 51, type:1}]
},
{
isUser: false,
name: "Nike",
type: 1,
users:[{user:21, type:3},{user: 31, type:1}]
}
and i've been trying this
.aggregate([
{$match:{type:1}},
{$project:
{
isUser:{$elemMatch:["users.user",51]},
users: 1,
type:1,
name: 1
}
}
])
this fails, i get an error "Maximum Stack size exceeded". Ive tried alot of combinations and none seem to work. I really want to avoid running multiple calls to mongodb. Can this be done in a single call?
I've been told to use unwind, but i am bit worried that it might lead to memory issues.
If i was using mysql, a simple subquery would have done the job... i hope i am overlooking a similar simple solution in mongodb.
Process the conditions for the array elements and match the result by using a combination of the $anyElementTrue which evaluates an array as a set and returns true if any of the elements are true and false otherwise, the $ifNull operator will act as a safety net that evaluates the following $map expression and returns the value of the expression if the expression evaluates to a non-null value. The $map in the $ifNull operator is meant to apply the conditional statement expression to each item in the users array and returns an array with the applied results. The resulting array will then be used evaluated by the $anyElementTrue and this will ultimately calculate and return the isUser field for each document:
db.collection.aggregate([
{ "$match": { "type": 1} },
{
"$project": {
"name": 1, "type": 1,
"isUser": {
"$anyElementTrue": [
{
'$ifNull': [
{
"$map": {
"input": "$users",
"as": "el",
"in": { "$eq": [ "$$el.user",51] }
}
},
[false]
]
}
]
}
}
}
])