PouchDB-find automatically building selectors from variable input - couchdb

I am looking for a way to adapt query fields in the db-find selector. I need this because filters access different fields which may or may not be active all the time.
I have tried two options, but none of them work:
In the first option I tried to build the query as a string and let it be evaluated:
public getFilteredItems(filters: ItemFilter[]) {
let selectors: string[] = [];
filters.forEach((filter: ItemFilter) => {
selectors.push(
"\n{\n'" +
filter.field +
"': { $in: \n ['" +
filter.values.join("','") +
"']} \n} \n "
);
});
return this.getDB('items').find({
selector: {
$and: eval('selectors.join()'),
// $and: selectors.join(),
// $and: selectors,
},
});
}
2: In the second try I have tried to give a variable as query field, but the syntax does not allow it.
public getFilteredItems(filters: ItemFilter[]) {
return this.getDB('items').find({
selector: {
filters[0].field: { $in: filter.values }
},
});
}
Is there any other way?
EDIT:
filters look like following:
filters: [
{
name: 'Filtergroup Template',
field: 'template',
values: ['case'],
always_active: true,
},
...
]
Error messages:
For option 1, $and: eval('selectors.join()'), and $and: selectors.join(),, I get the following error:
index.es.js?26cc:74 Uncaught (in promise) TypeError: selectors.forEach is not a function
at mergeAndedSelectors (index.es.js?26cc:74)
at massageSelector (index.es.js?26cc:242)
at find$1 (index-browser.es.js?5d16:1194)
at eval (index-browser.es.js?5d16:119)
at eval (index-browser.es.js?5d16:112)
at PouchDB.eval (index-browser.es.js?5d16:1380)
at eval (index-browser.es.js?a2df:138)
at new Promise (<anonymous>)
at PouchDB.eval (index-browser.es.js?a2df:125)
at PouchDB.eval [as find] (index.js?dd8f:14)
If I use $and: selectors,, I will only get a note with {docs: Array(0), warning: 'no matching index found, create an index to optimize query time'}

Given the comments above, simply create a collection for an explicit $and
const filters = [{
name: 'Filter A',
field: 'lead guitar',
values: ['jerry']
},
{
name: 'Filter B',
field: 'rhythm guitar',
values: ['bobby']
}
];
const clauses = {};
filters.forEach(filter => clauses[filter.field] = {
$in: filter.values
});
const query = {
selector: {
$and: [clauses]
}
};
document.getElementById("content").innerText = JSON.stringify(query, undefined, 2);
<pre id="content"></pre>
DO note however that $and is implicit which is covered here in the CouchDB docs.

Related

How to show list of field values of a collection of objects

I am making an API that shows a collection of ads with MongoDB and Node.js
I need to display the list of collection tags in a JSON string.
Example: 'home', 'mobile', 'motor'
This is the API initializer code:
const readline = require('readline');
const Product = require('./models/Product');
async function main() {
const advance = await question('Are you sure to continue with the deletion of the database? (yes or no) ')
if (!advance) {
process.exit();
}
const connection = require('./lib/connectMongoose')
await initProducts();
connection.close();
}
async function initProducts() {
const deleted = await Product.deleteMany();
console.log(`Remove ${deleted.deletedCount} products.`);
const inserted = await Product.insertMany([
{name: 'Table', sale: true, price: 150, photo: 'Table.png', tags: ['home']},
{name: 'Iphone 13 pro', sale: false, price: 950, photo: 'Iphone 13 pro.png', tags: ['mobile']},
{name: 'Car Mini Cooper', sale: true, price: 1000, photo: 'Car Mini Cooper.png', tags: ['motor']}
]);
console.log(`Create ${inserted.length} products.`)
}
main().catch(err => console.log('Hubo un error', err))
function question(text) {
return new Promise((resolve, reject) => {
const interface = readline.createInterface({
input: process.stdin,
output: process.stdout
});
interface.question(text, answer => {
interface.close();
if (answer.toLowerCase() === 'yes') {
resolve(true);
return;
}
resolve(false);
})
})
}
I need to find a MongoDB method that allows me to show when the API route calls the list that shows in JSON format all the tags that the collection includes
If I've understood correctly, one option is $unwind the tags array to get all tags as strings and be able to $group adding to a set to avoid duplicates.
db.collection.aggregate([
{
"$unwind": "$tags"
},
{
"$group": {
"_id": null,
"tags": {
"$addToSet": "$tags"
}
}
}
])
I think this works but $unwind and $group the entire collection is not always a good idea. It may be a slow process.
Example here

Using updateOne method to update an object field inside array - throws error "Cannot create field 'url' in element"

I have MongoDB database (with Mongoose) containing a collection of Products (among others), which looks like this:
[
{
name: 'Product A',
url: 'product-a',
category: 'accesory',
price: 12,
shortDescription: ['example description'],
technicalSpecs: [{ speed: 10, weight: 20 }],
images: [],
reviews: [],
relatedProducts: [
{
url: 'product-b',
name: 'Product B',
// to be added in Update query
//id: id_of_related_product
}
]
} /* other Product objects */
]
As every MongoDB document is provided with _id property by default, but within the relatedProducts array i only have url and name properties, i want to add the id property (associated with corresponding Product) for each object in the relatedProducts array, so i will be able to conveniently query and process those related products.
I came up with an idea to query all Products to get only those, which have non-empty relatedProducts array. Then i loop them and i search for Product model, which has specific url and name properties - this let's me get it's true (added by MongoDB) _id. At the end i want to add this _id to matching object inside relatedProducts array.
My code:
async function assignIDsToRelatedProducts(/* Model constructor */ Product) {
const productsWithRelatedOnes = await Product.find(
{ relatedProducts: { $ne: [] }}, ['relatedProducts', 'name', 'url']
);
for (const productItem of productsWithRelatedOnes) {
for (const relatedProduct of productItem.relatedProducts) {
const product = await Product.findOne(
{ url: relatedProduct.url, name: relatedProduct.name },
'_id'
);
// throws error
await productItem.updateOne(
{ 'relatedProducts.url': relatedProduct.url },
{ $set: { 'relatedProducts.$.id': product._id } }
);
}
}
}
However it throws the following error:
MongoError: Cannot create field 'url' in element {relatedProducts: [ /* array's objects here */ ]}
I don't know why MongoDB tries to create field 'url', as i use it to project/query url field (not create it) in updateOne method. How to fix this?
And - as i am newbie to MongoDB - is there a simpler way of achieving my goal? I feel that those two nested for..of loops are unnecessary, or even preceding creation of productsWithRelatedOnes variable is.
Is it possible to do with Mongoose Virtuals? I have tried it, but i couldn't match virtual property within the same Product Model - attach it to each object in relatedProducts array - after calling .execPopulate i received either an empty array or undefined (i am aware i should post at-the-time code of using Virtual, but for now i switched to above solution).
Although i didn't find solution or even reason of my problem, i solved it with a slightly other approach:
async function assignIDsToRelatedProducts(Product) {
const productsHavingRelatedProducts = Product.find({ relatedProducts: { $ne: [] }});
for await (const withRelated of productsHavingRelatedProducts) {
for (const relatedProductToUpdate of withRelated.relatedProducts) {
const relatedProduct = await Product
.findOne(
{ url: relatedProductToUpdate.url, name: relatedProductToUpdate.name },
['url', '_id']
);
await Product.updateMany(
{ 'relatedProducts.url': relatedProduct.url },
{ $set: { 'relatedProducts.$.id': relatedProduct._id } }
);
}
}
const amountOfAllProducts = await Product.find({}).countDocuments();
const amountOfRelatedProductsWithID = await Product
.find({ 'relatedProducts.id': { $exists: true } }).countDocuments();
console.log('All done?', amountOfAllProducts === amountOfRelatedProductsWithID);
}
Yet, i still suppose it can be done more concisely, without the initial looping. Hopefully somebody will suggest better solution. :)

Mongoose: updateMany() is not working as expected

I'm using mongoose to handle my DB queries. I'm trying to update a set of records entirely using this method. Mode code looks like this:
// prepare database query
const filter = { type: 'company' };
const update = req.body.payload; // payload contains the array of objects (i.e. updated records)
const options = { new: true, runValidators: true }
// find and update the taxonomy record
await Taxonomy.updateMany(filter, update, options);
But whenever I run this query I'm getting following error in the console:
Error [MongooseError]: Invalid update pipeline operator: "_id"
I suppose there is something wrong in my update payload. The req.body.payload looks like this:
[
{
_id: '5ef3d08c745428001d92f896',
type: 'company',
name: 'Company Size',
__v: 0
},
{
_id: '5ef3cdc5745428001d92f893',
type: 'company',
name: 'Company Industry',
__v: 0
}
]
Can you please tell me what actually is wrong here?
This is not the right usage of updateMany() - it is aimed to update many documents with a single change.
To update many documents use bulkwrite() (docs) :
async function myUpdateMany(Model, objectsArray) {
try {
let ops = []
for (let obj of (objectsArray || [])) {
ops.push({
updateOne: {
filter: { platformId: obj.platformId },
update: obj,
upsert: false, // set "true" if you want to add a new document if it doesn't exist
}
})
}
Model.bulkWrite(ops, { ordered: false });
} catch (err) {
throw Error("myUpdateMany error: " + err)
}
}
Regarding runValidators, according to this, it seems to work by default.

NodeJS MongoDB Mongoose export nested subdocuments and arrays to XLSX columns

I have query results from MongoDB as an array of documents with nested subdocuments and arrays of subdocuments.
[
{
RecordID: 9000,
RecordType: 'Item',
Location: {
_id: 5d0699326e310a6fde926a08,
LocationName: 'Example Location A'
}
Items: [
{
Title: 'Example Title A',
Format: {
_id: 5d0699326e310a6fde926a01,
FormatName: 'Example Format A'
}
},
{
Title: 'Example Title B',
Format: {
_id: 5d0699326e310a6fde926a01,
FormatName: 'Example Format B'
}
}
],
},
{
RecordID: 9001,
RecordType: 'Item',
Location: {
_id: 5d0699326e310a6fde926a08,
LocationName: 'Example Location C'
},
Items: [
{
Title: 'Example Title C',
Format: {
_id: 5d0699326e310a6fde926a01,
FormatName: 'Example Format C'
}
}
],
}
]
Problem
I need to export the results to XLSX in column order. The XLSX library is working to export the top-level properties (such as RecordID and RecordType) only. I also need to export the nested objects and arrays of objects. Given a list of property names e.g. RecordID, RecordType, Location.LocationName, Items.Title, Items.Format.FormatName the properties must be exported to XLSX columns in the specified order.
Desired result
Here is the desired 'flattened' structure (or something similar) that
I think should be able to convert to XLSX columns.
[
{
'RecordID': 9000,
'RecordType': 'Item',
'Location.LocationName': 'Example Location A',
'Items.Title': 'Example Title A, Example Title B',
'Items.Format.FormatName': 'Example Format A, Example Format B',
},
{
'RecordID': 9001,
'RecordType': 'Item',
'Location.LocationName': 'Example Location C',
'Items.Title': 'Example Title C',
'Items.Format.FormatName': 'Example Format C',
}
]
I am using the XLSX library to convert the query results to XLSX which works for top-level properties only.
const worksheet: XLSX.WorkSheet = XLSX.utils.json_to_sheet(results.data);
const workbook: XLSX.WorkBook = { Sheets: { 'data': worksheet }, SheetNames: ['data'] };
const excelBuffer: any = XLSX.write(workbook, { bookType: 'xlsx', type: 'array' });
const data: Blob = new Blob([excelBuffer], { type: EXCEL_TYPE });
FileSaver.saveAs(data, new Date().getTime());
POSSIBLE OPTIONS
I am guessing I need to 'flatten' the structure either using aggregation in the query or by performing post-processing when the query is returned.
Option 1: Build the logic in the MongoDB query to flatten the results.
$replaceRoot might work since it is able to "promote an existing embedded document to the top level". Although I am not sure if this will solve the problem exactly, I do not want to modify the documents in place, I just need to flatten the results for exporting.
Here is the MongoDB query I am using to produce the results:
records.find({ '$and': [ { RecordID: { '$gt': 9000 } } ]},
{ skip: 0, limit: 10, projection: { RecordID: 1, RecordType: 1, 'Items.Title': 1, 'Items.Location': 1 }});
Option 2: Iterate and flatten the results on the Node server
This is likely not the most performant option, but might be the easiest if I can't find a way to do so within the MongoDB query.
UPDATE:
I may be able to use MongoDB aggregate $project to 'flatten' the results. For example, this aggregate query effectively 'flattens' the results by 'renaming' the properties. I just need to figure out how to implement the query conditions within the aggregate operation.
db.records.aggregate({
$project: {
RecordID: 1,
RecordType: 1,
Title: '$Items.Title',
Format: '$Items.Format'
}
})
UPDATE 2:
I have abandoned the $project solution because I would need to change the entire API to support aggregation. Also, I would need to find a solution for populate because aggregate does not support it, rather, it uses $lookup which is possible but time consuming because I would need to write the queries dynamically. I am going back to look into how to flatten the object by creating a function to iterate the array of objects recursively.
Below is a solution for transforming the Mongo data on the server via a function flattenObject which recursively flattens nested objects and returns a 'dot-type' key for nested paths.
Note that the snippet below contains a function that renders and editable table to preview, however, the important part you want (download the file), should be triggered when you run the snippet and click the 'Download' button.
const flattenObject = (obj, prefix = '') =>
Object.keys(obj).reduce((acc, k) => {
const pre = prefix.length ? prefix + '.' : '';
if (typeof obj[k] === 'object') Object.assign(acc, flattenObject(obj[k], pre + k));
else acc[pre + k] = obj[k];
return acc;
}, {});
var data = [{
RecordID: 9000,
RecordType: "Item",
Location: {
_id: "5d0699326e310a6fde926a08",
LocationName: "Example Location A"
},
Items: [{
Title: "Example Title A",
Format: {
_id: "5d0699326e310a6fde926a01",
FormatName: "Example Format A"
}
},
{
Title: "Example Title B",
Format: {
_id: "5d0699326e310a6fde926a01",
FormatName: "Example Format B"
}
}
]
},
{
RecordID: 9001,
RecordType: "Item",
Location: {
_id: "5d0699326e310a6fde926a08",
LocationName: "Example Location C"
},
Items: [{
Title: "Example Title C",
Format: {
_id: "5d0699326e310a6fde926a01",
FormatName: "Example Format C"
}
}]
}
];
const EXCEL_MIME_TYPE = `application/vnd.ms-excel`;
const flattened = data.map(e => flattenObject(e));
const ws_default_header = XLSX.utils.json_to_sheet(flattened);
const ws_custom_header = XLSX.utils.json_to_sheet(flattened, {
header: ['Items.Title', 'RecordID', 'RecordType', 'Location.LocationName', 'Items.Format.FormatName']
});
const def_workbook = XLSX.WorkBook = {
Sheets: {
'data': ws_default_header
},
SheetNames: ['data']
}
const custom_workbook = XLSX.WorkBook = {
Sheets: {
'data': ws_custom_header
},
SheetNames: ['data']
}
const def_excelBuffer = XLSX.write(def_workbook, {
bookType: 'xlsx',
type: 'array'
});
const custom_excelBuffer = XLSX.write(custom_workbook, {
bookType: 'xlsx',
type: 'array'
});
const def_blob = new Blob([def_excelBuffer], {
type: EXCEL_MIME_TYPE
});
const custom_blob = new Blob([custom_excelBuffer], {
type: EXCEL_MIME_TYPE
});
const def_button = document.getElementById('dl-def')
/* trigger browser to download file */
def_button.onclick = e => {
e.preventDefault()
saveAs(def_blob, `${new Date().getTime()}.xlsx`);
}
const custom_button = document.getElementById('dl-cus')
/* trigger browser to download file */
custom_button.onclick = e => {
e.preventDefault()
saveAs(custom_blob, `${new Date().getTime()}.xlsx`);
}
/*
render editable table to preview (for SO convenience)
*/
const html_string_default = XLSX.utils.sheet_to_html(ws_default_header, {
id: "data-table",
editable: true
});
const html_string_custom = XLSX.utils.sheet_to_html(ws_custom_header, {
id: "data-table",
editable: true
});
document.getElementById("container").innerHTML = html_string_default;
document.getElementById("container-2").innerHTML = html_string_custom;
<script src="https://cdnjs.cloudflare.com/ajax/libs/xlsx/0.14.3/xlsx.full.min.js"></script>
<head>
<title>Excel file generation from JSON</title>
<meta charset="utf-8" />
<style>
.xport,
.btn {
display: inline;
text-align: center;
}
a {
text-decoration: none
}
#data-table,
#data-table th,
#data-table td {
border: 1px solid black
}
</style>
</head>
<script>
function render(type, fn, dl) {
var elt = document.getElementById('data-table');
var wb = XLSX.utils.table_to_book(elt, {
sheet: "Sheet JS"
});
return dl ?
XLSX.write(wb, {
bookType: type,
bookSST: true,
type: 'array'
}) :
XLSX.writeFile(wb, fn || ('SheetJSTableExport.' + (type || 'xlsx')));
}
</script>
<div>Default Header</div>
<div id="container"></div>
<br/>
<div>Custom Header</div>
<div id="container-2"></div>
<br/>
<table id="xport"></table>
<button type="button" id="dl-def">Download Default Header Config</button>
<button type="button" id="dl-cus">Download Custom Header Config</button>
<script src="https://cdnjs.cloudflare.com/ajax/libs/FileSaver.js/1.3.8/FileSaver.min.js"></script>
I wrote a function to iterate all object in the results array and create new flattened objects recursively. The flattenObject function shown here is similar to the previous answer and I took additional inspiration from this related answer.
The '_id' properties are specifically excluded from being added to the flattened object, since ObjectIds are still being returned as bson types even though I have the lean() option set.
I still need to figure out how to sort the objects such that they are in the order given e.g. RecordID, RecordType, Items.Title. I believe that might be easiest to achieve by creating a separate function to iterate the flattened results, although not necessarily the most performant. Let me know if anyone has any suggestions on how to achieve the object sorting by a given order or has any improvements to the solution.
const apiCtrl = {};
/**
* Async array iterator
*/
apiCtrl.asyncForEach = async (array, callback) => {
for (let index = 0; index < array.length; index++) {
await callback(array[index], index, array)
}
}
// Check if a value is an object
const isObject = (val) => {
return typeof val == 'object' && val instanceof Object && !(val instanceof Array);
}
// Check if a value is a date object
const isDateObject = (val) => {
return Object.prototype.toString.call(val) === '[object Date]';
}
/**
* Iterate object properties recursively and flatten all values to top level properties
* #param {object} obj Object to flatten
* #param {string} prefix A string to hold the property name
* #param {string} res A temp object to store the current iteration
* Return a new object with all properties on the top level only
*
*/
const flattenObject = (obj, prefix = '', res = {}) =>
Object.entries(obj).reduce((acc, [key, val]) => {
const k = `${prefix}${key}`
// Skip _ids since they are returned as bson values
if (k.indexOf('_id') === -1) {
// Check if value is an object
if (isObject(val) && !isDateObject(val)) {
flattenObject(val, `${k}.`, acc)
// Check if value is an array
} else if (Array.isArray(val)) {
// Iterate each array value and call function recursively
val.map(element => {
flattenObject(element, `${k}.`, acc);
});
// If value is not an object or an array
} else if (val !== null & val !== 'undefined') {
// Check if property has a value already
if (res[k]) {
// Check for duplicate values
if (typeof res[k] === 'string' && res[k].indexOf(val) === -1) {
// Append value with a separator character at the beginning
res[k] += '; ' + val;
}
} else {
// Set value
res[k] = val;
}
}
}
return acc;
}, res);
/**
* Convert DB query results to an array of flattened objects
* Required to build a format that is exportable to csv, xlsx, etc.
* #param {array} results Results of DB query
* Return a new array of objects with all properties on the top level only
*/
apiCtrl.buildExportColumns = async (results) => {
const data = results.data;
let exportColumns = [];
if (data && data.length > 0) {
try {
// Iterate all records in results data array
await apiCtrl.asyncForEach(data, async (record) => {
// Convert the multi-level object to a flattened object
const flattenedObject = flattenObject(record);
// Push flattened object to array
exportColumns.push(flattenedObject);
});
} catch (e) {
console.error(e);
}
}
return exportColumns;
}

Mongoose and mapReduce - map function not called

I'm attempting to use the mapReduce function of Mongodb via Mongoose, but the map function I'm passing in is never called. Here is the data currently contained in the "Post" model collection:
[ { data: 'Tag test data',
name: 'Tag Test',
_id: 5130dff2560105c235000002,
__v: 0,
comments: [],
tags: [ 'tag1', 'tag2', 'tag3' ] },
{ data: 'Testing tags. Again.',
name: 'Another test post',
_id: 5131213b611fe1f443000002,
__v: 0,
comments: [],
tags: [ 'tags', 'test', 'again' ] } ]
Here is the code:
var Schema = mongoose.Schema;
var PostSchema = new Schema ({
name : String
, data : String
, tags : [String]
});
mongoose.model('Post', PostSchema);
var o = {};
o.map = function() {
if (!this.tags) {
//console.log('No tags found for Post ' + this.name);
return;
}
for (index in this.tags) {
emit(this.tags[index], 1);
}
}
o.reduce = function(previous, current) {
var count = 0;
for (index in current) {
count += current[index];
}
return count;
}
o.out = { replace : 'tags'}
o.verbose = true;
var Post = mongoose.model('Post');
Post.mapReduce(o, function(error, model, stats) {
console.log('model: ' + model);
console.log('stats: ' + stats);
});
The "model" and "stats" objects are always undefined, and the log statements in the map function are never called. If I do something like this with the Post model outside of the mapReduce function, I get the data at the top of the post as expected:
Post.find().exec(function(err, posts) {
console.log(posts);
});
Any suggestions? I'm sure something is just slightly off...
You can't call console.log from within the map and reduce functions as it's not supported by Mongo's JavaScript engine.
To debug your map/reduce/finalize functions you can use the MongoDB print statement. The output will be added to your Mongo log file.

Resources