Flattening conditional nested lists to a single flat list - node.js

Lodash Flatten nested lists.
I have a function in my Typescript script, where I convert some objects.
Unfortunately, some of the objects are nested, so the return type will sometimes be a nested list.
const values: (IRobotics | IRobotics[][])[] //this is an interface type i described
I used the LoDash FlatMap function to try to flatten it out, but it still gives one nested level for each of the objects.
When the response comes, it only removes the outer list.
I tried using the _.flatMap() function initially,
const parsedValues = _.flatMap(values) as IRobotics[];
however The values within the nested lists, is still nested one level too deep (see image for example).
The types are correct, but i just want a single list of objects and not a nested list (which happens with the examples where it is IRobotics[][])
Anybody has a nice solution to flatten a conditional two-layer array from here?
Entire function for reference. The values attribute is where the initial values are returned
static createFromeData(item: EveryMatrix) {
const values = Object.entries(item).map(entry => {
return this.createFromEntry(entry);
});
const nestedValues = values.filter(entry => Array.isArray(entry));
const parsedValues = _.flatMap(values) as IRobotics[];
const IRobotics = new IRoboticsEveryMatrix(parsedValues);
return IRobotics;
}

Use Array.flatMap() instead of Array.map():
const values = Object.entries(item).flatMap(entry =>
this.createFromEntry(entry)
);
And if you're correct environment doesn't support it, use _.flatMap():
const values = _.flatMap(Object.entries(item), entry =>
this.createFromEntry(entry)
);
If you want to flatten an array of arrays (because you need to nested list for nestedValues), you can use Array.flat() or _.flatten():
const parsedValues = values.flat as IRobotics[];
Or
const parsedValues = _.flatten(values) as IRobotics[];

Related

How to combine and sort key-value pair in Terraform

since the last update of the Logicmonitor provider in Terraform we're struggling with a sorting isse.
In LogicMonitor the properties of a device are a name-value pair, and they are presented alfabetically by name. Also in API requests the result is alphabetical. So far nothing fancy.
But... We build our Cloud devices using a module. Calling the module we provide some LogicMonitor properties specially for this device, and a lot more are provided in the module itself.
In the module this looks like this:
`
custom_properties = concat([
{
name = "host_fqdn"
value = "${var.name}.${var.dns_domain}"
},
{
name = "ocid"
value = oci_core_instance.server.id
},
{
name = "private_ip"
value = oci_core_instance.server.private_ip
},
{
name = "snmp.version"
value = "v2c"
}
],
var.logicmonitor_properties)
`
The first 4 properties are from the module and combined with anyting what is in var.logicmonitor_properties. On the creation of the device in LogicMonitor all properties are set in the order the are and no problem.
The issue arises when there is any update on a terraform file in this environment. Due to the fact the properties are presented in alphabetical order, Terraform is showing a lot of changes if finds (but which are in fact just a mixed due to sorting).
The big question is: How can I sort the complete list of properties bases on the "name".
Tried to work with maps, sort and several other functions and examples, but got nothing working on key-value pairs. Merging single key's works fine in a map, but how to deal with name/value pairs/
I think you were on the right track with maps and sorting. Terraform maps do not preserve any explicit ordering themselves, and so whenever Terraform needs to iterate over the elements of a map in some explicit sequence it always do so by sorting the keys lexically (by Unicode codepoints) first.
Therefore one answer is to project this into a map and then project it back into a list of objects again. The projection back into list of objects will implicitly sort the map elements by their keys, which I think will get the effect you wanted.
variable "logicmonitor_properties" {
type = list(object({
name = string
value = string
}))
}
locals {
base_properties = tomap({
host_fqdn = "${var.name}.${var.dns_domain}"
ocid = oci_core_instance.server.id
private_ip = oci_core_instance.server.private_ip
"snmp.version" = "v2c"
})
extra_properties = tomap({
for prop in var.logicmonitor_properties : prop.name => prop.value
})
final_properties = merge(local.base_properties, local.extra_properties)
# This final step will implicitly sort the final_properties
# map elements by their keys.
final_properties_list = tolist([
for k, v in local.final_properties : {
name = k
value = v
}
])
}
With all of the above, local.final_properties_list should be similar to the custom_properties structure you showed in your question except that the elements of the list will be sorted by their names.
This solution assumes that the property names will be unique across both base_properties and extra_properties. If there are any colliding keys between both of those maps then the merge function will prefer the value from extra_properties, overriding the element of the same key from base_properties.
First, use the sort() function to sort the keys in alphabetical order:
sorted_keys = sort(keys(var.my_map))
Next, use the map() function to create a new map with the sorted keys and corresponding values:
sorted_map = map(sorted_keys, key => var.my_map[key])
Finally, you can use the jsonencode() function to print the sorted map in JSON format:
jsonencode(sorted_map)```

Transforming large array of objects to csv using json2csv

I need to transform a large array of JSON (that can have over 100k positions) into a CSV.
This array is created directly in the application, it's not the result of an uploaded file.
Looking at the documentation, I've thought on using parser but it says that:
For that reason is rarely a good reason to use it until your data is very small or your application doesn't do anything else.
Because the data is not small and my app will do other things than creating the csv, I don't think it'll be the best approach but I may be misunderstanding the documentation.
Is it possible to use the others options (async parser or transform) with an already created data (and not a stream of data)?
FYI: It's a nest application but I'm using this node.js lib.
Update: I've tryied to insert with an array with over 300k positions, and it went smoothly.
Why do you need any external modules?
Converting JSON into a javascript array of javascript objects is a piece of cake with the native JSON.parse() function.
let jsontxt=await fs.readFile('mythings.json','uft8');
let mythings = JSON.parse(jsontxt);
if (!Array.isArray(mythings)) throw "Oooops, stranger things happen!"
And, then, converting a javascript array into a CSV is very straightforward.
The most obvious and absurd case is just mapping every element of the array into a string that is the JSON representation of the object element. You end up with a useless CSV with a single column containing every element of your original array. And then joining the resulting strings array into a single string, separated by newlines \n. It's good for nothing but, heck, it's a CSV!
let csvtxt = mythings.map(JSON.stringify).join("\n");
await fs.writeFile("mythings.csv",csvtxt,"utf8");
Now, you can feel that you are almost there. Replace the useless mapping function into your own
let csvtxt = mythings.map(mapElementToColumns).join("\n");
and choose a good mapping between the fields of the objects of your array, and the columns of your csv.
function mapElementToColumns(element) {
return `${JSON.stringify(element.id)},${JSON.stringify(element.name)},${JSON.stringify(element.value)}`;
}
or, in a more thorough way
function mapElementToColumns(fieldNames) {
return function (element) {
let fields = fieldnames.map(n => element[n] ? JSON.stringify(element[n]) : '""');
return fields.join(',');
}
}
that you may invoke in your map
mythings.map(mapElementToColumns(["id","name","element"])).join("\n");
Finally, you might decide to use an automated for "all fields in all objects" approach; which requires that all the objects in the original array maintain a similar fields schema.
You extract all the fields of the first object of the array, and use them as the header row of the csv and as the template for extracting the rest of the elements.
let fieldnames = Object.keys(mythings[0]);
and then use this field names array as parameter of your map function
let csvtxt= mythings.map(mapElementToColumns(fieldnames)).join("\n");
and, also, prepending them as the CSV header
csvtxt.unshift(fieldnames.join(','))
Putting all the pieces together...
function mapElementToColumns(fieldNames) {
return function (element) {
let fields = fieldnames.map(n => element[n] ? JSON.stringify(element[n]) : '""');
return fields.join(',');
}
}
let jsontxt=await fs.readFile('mythings.json','uft8');
let mythings = JSON.parse(jsontxt);
if (!Array.isArray(mythings)) throw "Oooops, stranger things happen!";
let fieldnames = Object.keys(mythings[0]);
let csvtxt= mythings.map(mapElementToColumns(fieldnames)).join("\n");
csvtxt.unshift(fieldnames.join(','));
await fs.writeFile("mythings.csv",csvtxt,"utf8");
And that's it. Pretty neat, uh?

NODE.JS: iterating over an array of objects, creating a new key if it does not exist

I am iterating over a collection of data, in my case, an array of objects. Here is a sample of 2 data points from it:
{
violation_id: '211315',
inspection_id: '268804',
violation_category: 'Garbage and Refuse',
violation_date: '2012-03-22 0:00',
violation_date_closed: '',
violation_type: 'Refuse Accumulation' },
{
violation_id: '214351',
inspection_id: '273183',
violation_category: 'Building Conditions',
violation_date: '2012-05-01 0:00',
violation_date_closed: '2012-04-17 0:00',
violation_type: 'Mold or Mildew' }
I need to create a new array of objects from this, one for each "violation_category" property. If Violation category already exists in the new array I am creating, i simply add the information to that existing category object (instead of having two "building conditions" objects for example, I would just add to an existing one).
However, I am having trouble assigning to the existing object if the current one exists (it's easy to check if it does not, but not the other way around). This is what am attempting to do currently:
if (violationCategory.uniqueCategoryName) {
violationCategory.uniqueCategoryName.violations = results[i].violation_id;
violationCategory.uniqueCategoryName.date = results[i].violation_date;
violationCategory.uniqueCategoryName.closed =
results[i].violation_date_closed;
} else {
category.violations = results[i].violation_id;
category.date = results[i].violation_date;
category.closed = results[i].violation_date_closed;
violationCategory.push(category);
}
In first condition, if this category (key) exists, I simply add to it, and in the second condition, this is where I am struggling. Any help appreciated. Thanks guys.
Just add an empty object to the key if there no object there :
violationCategory.uniqueCategoryName = violationCategory.uniqueCategoryName || {};
And only then, add the data you want to the object.
violationCategory.uniqueCategoryName.violations = results[i].violation_id;
violationCategory.uniqueCategoryName.date = results[i].violation_date;
violationCategory.uniqueCategoryName.closed =
results[i].violation_date_closed;
No condition needed.
Good luck!
Assuming that you have an input variable which is an array of objects, where the objects are looking like the objects of the question, you can generate your output like this:
var output = {};
for (var item of input) {
if (!output[item.violation_category]) output[item.violation_category] = [];
output[item.violation_category].push(item);
}
Of course you might customize it like you want.

How to get object x or y in nested document

I know this has been asked before but I can't seem to find the answer, I want to get data/object in the nested array.
show image problem
schedule = await Schedule.findById({_id:'5b496ec3444152122c8d839e'})
console.log(schedule.datalayout.section.data.x)
If you want to get the specified field in the image, you need to determine field index in the array as down below:
console.log(schedule.datalayout.section[0].data[0].x)
And also, If you want to get all of the x fields in the data array, you need to write something like this:
for(let singleData of schedule.datalayout.section[0].data){
console.log(singleData.x)
}
// for using 'of' keyword, your function must be a async function.

Efficient way to search object from list of object in c#

I have a list which contains more than 75 thousand object. To search item from list currently I am using following code.
from nd in this.m_ListNodes
where
nd.Label == SearchValue.ToString()
select
nd;
Is this code is efficient?
How often do you need to search the same list? If you're only searching once, you might as well do a straight linear search - although you can make your current code slightly more efficient by calling SearchValue.ToString() once before the query.
If you're going to perform this search on the same list multiple times, you should either build a Lookup or a Dictionary:
var lookup = m_ListNodes.ToLookup(nd => nd.Label);
or
var dictionary = m_ListNodes.ToDictionary(nd => nd.Label);
Use a dictionary if there's exactly one entry per label; use a lookup if there may be multiple matches.
To use these, for a lookup:
var results = lookup[SearchValue.ToString()];
// results will now contain all the matching results
or for a dictionary:
WhateverType result;
if (dictionary.TryGetValue(SearchValue.ToString(), out result))
{
// Result found, stored in the result variable
}
else
{
// No such item
}
No. It would be better if you used a Dictionary or a HashSet with the label as the key. In your case a Dictionary is the better choice:
var dictionary = new Dictionary<string, IList<Item>>();
// somehow fill dictionary
IList<Item> result;
if(!dictionary.TryGetValue(SearchValue.ToString(), out result)
{
// if you need an empty list
// instead of null, if the SearchValue isn't in the dictionary
result = new List<Item>();
}
// result contains all items that have the key SearchValue

Resources