How can I group items close to each other without knowing the distribution? - node.js

Question

I'm preparing some data about sold apartment prices for regression analysis. One category is what street the houses are on, but some streets have very different areas, so I want to make a category with the combination of construction year and street name.
Broadway 1910
Broadway 2001
Forexample my challenge is that sometimes the construction spans over several two years. The data is from Sweden, known for huge centralized housing projects. I would like to group these houses together into a period somehow. This is my current code. I know it's not very efficient, but it will only run once on a not huge dataset.
(async () =>{
let client;
try {
client = await MongoClient;
let collection = client.db("booliscraper").collection("sold");
let docs = await collection.find();
await docs.forEach((sale) => {
sale.street = sale.location.address.streetAddress.split(/[0-9]/)[0] + sale.location.namedAreas[0]
sale.streetYear = sale.street+" "+sale.constructionYear
log(sale);
collection.replaceOne({_id: ObjectId(sale._id)}, doc)
});
client.close();
} catch(err) {
log(err)
}
})()

Answer 1

As you correctly said, your current code is inefficient when it comes to dealing with huge datasets so instead of making several calls to the server to do replaceOne within your forEach loop, you can create an aggregate query that computes the category fields you want with the $group pipeline and push the documents that fall into those categories into an array that you will later use to do a bulk update.
For the bulk update you can use bulkWrite method on the collection that will have multiple updateMany operations.
The following operation shows the intuition above in practice:
(async () => {
try {
let client = await MongoClient;
let collection = client.db("booliscraper").collection("sold");
let pipeline = [
{ '$group': {
'_id': {
'street': {
'$concat': [
{
'$arrayElemAt': [
{ '$split': [
'$location.address.streetAddress',
/[0-9]/
] },
0
]
},
{ '$arrayElemAt': [ '$location.namedAreas', 0 ] },
]
},
'streetYear': { '$concat': ['$street', ' ', '$constructionYear'] }
},
'ids': { '$push': '$_id' }
} }
]
let docs = await collection.aggregate(pipeline);
let ops = docs.map(({ _id, ids }) => ({
'updateMany': {
'filter': { '_id': { '$in': ids } },
'update': { '$set': {
'street': _id.street, 'streetYear': _id.streetYear
} }
}
}));
let result = await collection.bulkWrite(ops);
log(result)
client.close()
} catch(err) {
log(err)
}
})()