Here's what I want to achieve in mongodb but as a javascript example.
var array = [];
//Initiating an array
for(var i = 0; i < 30; i++) {
array[i] = 0;
}
//Changing a value at an index
array[14] = 1;
//Getting a value at an index
console.log(array[4]);
//Swapping Two Variables.
var temp = array[14];
array[14] = array[12];
array[12] = temp;
So far I can set a schema
var schema = new mongoose.Schema({
array: {Number: Number, Value: Number}
});
I can Initiate it later on
schema.statics.name = function(cb) {
var new = new Array({
{Number: 1, Value: 0},
{Number: 2, Value: 0},
...
{Number: 30, Value: 0}
});
new.save();
I can loop through them all
c.user.array.forEach(function (element) {
console.log(element.Number);
console.log(element.Value);
})
But I get stuck with finding a single value or setting/switching them.
It seems overly complicated for something so simple in a programming language; ive been trying for the last few hours and there so many {} and $'s that its making my head hurt.
From what I can find I should be using .find() but then there no examples of how the stuff in the schema is laid out.
Schema:
array: [{value: Number}]
Can be initialised as many times as you want:
backpack: [{value: 0}, {value:0}, {value:0}]
and can be accessed like
.array[1].value
more info on sub docs and children
Thanks to Tomalak
Related
I need to upload a large amount of data to mongodb.
not a file, but rather a very large amount of key value pairs.
example:
let payload = [];
for ( let i =0 ;i<1000000;i++){
payload.push({
"first name": "juan",
"hair color": ""+i,
"gender" :"male"
})
}
var body = {
"channelId":"63dd281360e269e2a9399939",
"recordCount":payload.length,
"minBidUSD": 5,
"payload": payload
}
the script above creates a huge payload. that payload is then put on the body of a POST request to our system.
I need to be able to store very large amounts of data in that payload.
context of functionality: I am working on a website where people can sell data leads. ex: I am looking for males in florida that are 30 years old. the possible data structure for that lead payload would be:
{gender:"male",state:"florida",age:30}
this is the question that have::
[1] is there a better way to store this data?
--note: the attributes or this payload change, so i cant create a model against it.
[2] if the best way to store this data is using gridfs-file-storage, how do I do it?
Additional notes about the question::
here is the model for the collection that holds the payload
const mongoose = require("mongoose");
const channelDataSchema = mongoose.Schema({
channelId: { type: String, required: true },
payload: { type: [Object], required:true },
});
module.exports = mongoose.model("ChannelData", channelDataSchema);
Gridfs is for binary data and is chunked up to avoid the 16mb MongoDB document size limitation. I don't recommend gridfs for basic data.
For rapid data ingestion, I recommend using bulkwrite. Here is a mongoshell example. This example will loop creating a bunch of fake data elements made up using the random() function. It is for illustration only. If you need to insert data from a source, such as a file, or another database system you can use bulkwrite using a custom programming language, such as Node, and using the MongoDB supported database driver - which includes bulkwrite features.
MongoShell Example:
use mydatabase;
function getRandomInteger(min, max) {
// NOT INCLUSIVE OF MAX. IF NEED TO INCLUDE MAX, BUMP IT UP BY ONE
return Math.floor(Math.random() * (max - min) + min);
}
function getRandomAlphaNumeric(length) {
var result = [];
var characters = 'abcdefghijklmnopqrstuvwxwyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789';
var charactersLength = characters.length;
for ( var i = 0; i < length; i++ ) {
result.push(characters.charAt(Math.floor(Math.random() * charactersLength)));
}
return result.join('');
}
function generateDocument() {
return {
stringFixedLength01: getRandomAlphaNumeric(1),
stringFixedLength02: getRandomAlphaNumeric(2),
stringFixedLength03: getRandomAlphaNumeric(3),
stringFixedLength04: getRandomAlphaNumeric(4),
stringFixedLength05: getRandomAlphaNumeric(5),
stringVariableLength: getRandomAlphaNumeric(getRandomInteger(5, 50)),
integer1: NumberInt(getRandomInteger(0, 2000000)),
long1: NumberLong(getRandomInteger(0, 100000000)),
date1: new Date(),
guid1: new UUID()
};
}
for (var j = 0; j < 1000000; j++) {
var batch=[];
for (var i = 0; i < 50000; i++) {
batch.push( { insertOne: { document: generateDocument() } } );
if(i % 10000 == 0) {
print("outerloop: " + j + ", innerloop: " + i);
}
}
print("BulkWrite command submitted...");
db.mycollection.bulkWrite(batch, {ordered: false});
}
I'm working on a project where I need to declare customsItem formatted in a particular way.
The format given is:
var customsItem = {
"description":"T-Shirt",
"quantity":20,
"net_weight":"1",
"mass_unit":"lb",
"value_amount":"200",
"value_currency":"USD",
"origin_country":"US",
};
In my project however, I have multiple descriptions, so I need to make customsItem an array containing both.
I have array itemInCart =
[
{
itemDescription: 't-shirt',
qty: 1,
pre_orderQty: 1,
price: 30,
weight: 8
},
{
itemDescription: 'pants',
qty: 0,
pre_orderQty: 1,
price: 40,
weight: 5
}
]
I need to get these items in the correct format and within an array called customsItem. I thought to do this using a for loop with push(). Currently, I'm not getting anything when I try to console.log(customsItem), so I'm not sure if this is the best way to achieve the results that I am trying to get. I would really appreciate any help or advice on how to correctly get the results that I need. Thank you!
const customsItem = [];
for (var item of itemInCart) {
const items = {
"description":item.itemDescription,
"quantity":item.qty + item.pre_orderQty,
"net_weight":item.weight,
"mass_unit":"oz",
"value_amount":item.price,
"value_currency":"USD",
"origin_country":"US",
}
customItem.push(
items
)
}
You are not pushing into the correct array:
const customsItem = [];
for (var item of itemInCart) {
const items = {
"description":item.itemDescription,
"quantity":item.qty + item.pre_orderQty,
"net_weight":item.weight,
"mass_unit":"oz",
"value_amount":item.price,
"value_currency":"USD",
"origin_country":"US",
}
customItem.push( <---- needs to be customsItem.push
items
)
}
I am stuck with mongo query. I have a mongo collection structure which i can not modify at this time as it is very large data.
I need to carry out some results from the collection , so tried all ways round to get it.
Here is my collection json schema:-
{
"date": "2017-01-01T00:00:00.000Z",
"bob":"P",
"jacob":"P",
"wilson":"A",
"dev":"SL"
},
{
"date": "2017-01-02T00:00:00.000Z",
"bob":"P",
"jacob":"A",
"wilson":"A",
"dev":"SL"
},
{
"date": "2017-01-03T00:00:00.000Z",
"bob":"P",
"jacob":"P",
"wilson":"A",
"dev":"SL"
},
{
"date": "2017-01-04T00:00:00.000Z",
"shashikant":"P",
"jacob":"P",
"wilson":"SL",
"dev":"SL"
}
....
As output I am looking for below kind of structure:-
from 1st jan 2017 to 30th jan 2017
bob P 17
bob A 2
wilson P 10
dev SL. 1
.....
I am using loopback for my backend but still i can use normal mongodb query to get the output.
Please help
MongoDB allows $unwind only for the arrays. But you could use a simple mapReduce to achieve what you want:
//Define the time frame here
var from = new Date('2015-01-01T00:00:00.000Z');
var to = new Date('2025-01-01T00:00:00.000Z');
db.getCollection('test').mapReduce(function () {
var keys = Object.keys(this);
//If there is no date found on a record, simply skip
if (!this.date) {
return;
}
var date = new Date(this.date);
//Skip the records that do not fit into [form; to] interval
if (!(date >= from && date <= to)) {
return;
}
for (var i = 0; i < keys.length; i++) {
var key = keys[i];
//Emit each combination of key and value
if (key !== 'date' && key !== '_id') {
emit({key: key, value: this[key]}, {count: 1});
}
}
},
function (key, values) {
var reduced = {count: 0};
for (var i = 0; i < values.length; i++) {
var value = values[i];
//Simply counting the combinations here
reduced.count += value.count;
}
return reduced;
},
{
//Passing the dates to mongo
//so 'from' and 'to' will be avail in map, reduce and finalize
scope: {from: from, to: to},
finalize: function (key, reducedValue) {
//Changing the data structure just for a convenience
return {
propertyName: key.key,
propertyValue: key.value,
from: from,
to: to,
count: reducedValue.count
};
},
out: {inline: 1}
}
);
I tested this in Mongo console, but map-reduces are also supported by mongo native Node.js and for mongoose as well.
i have the following simplified Scheme:
var restsSchema = new Schema({
name: String
menu: [mongoose.Schema.Types.Mixed]
});
My document can look like:
{
name: "Sandwiches & More",
menu: [
{id:1,name:"Tona Sandwich",price: 10, soldCounter:0},
{id:2,name:"Salami Sandwich",price: 10, soldCounter:0},
{id:3,name:"Cheese Sandwich",price: 10, soldCounter:0}
]
}
The collection rests is indexed with:
db.rests.createIndex( { "menu.id": 1} , { unique: true })
Lets say i have this array of ids [1,3] and based on that i need to increment the soldCounter by 1 of menu items with ids=1 or 3.
What will be the must efficient way of doing so?
thanks for the helpers!
EDIT:
I have used the following solution:
db.model('rests').update({ _id: restid,'menu.id': {$in: ids}}, {$inc: {'menu.$.soldCounter': 1}}, {multi: true},function(err) {
if(err)
console.log("Error while updating sold counters: " + err.message);
});
where ids is an array of integers with ids of menu items.
restid is the id of the specific document we want to edit in the collection.
For some reason only the first id in the ids array is being updated.
There is a way of doing multiple updates, here it is:
Just make sure you have the indexes in the array you want to update.
var update = { $inc: {} };
for (var i = 0; i < indexes.length; ++i) {
update.$inc[`menu.${indexes[i]}.soldCounter`] = 1;
}
Rests.update({ _id: restid }, update, function(error) {
// ...
});
it seems not possible to update multiple subdocuments at once (see this answer). So a find & save seems to be the only solution.
Rest.findById(restId).then(function(rest){
var menus = rest.menu.filter(function(x){
return menuIds.indexOf(x.id) != -1;
});
for (var menu of menus){
menu.soldCounter++;
}
rest.save();
});
In the end it's only one find and one save requests.
I have following models:
Question Model
var OptionSchema = new Schema({
correct : {type:Boolean, default:false}
value : String
});
var QuestionSchema = new Schema({
value : String
, choices : [OptionSchema]
, quiz : {type:ObjectId, ref:'quizzes'}
, createdOn : {type:Date, default:Date.now}
...
});
var Question = mongoose.model('questions', QuestionSchema);
Quiz Model
var QuizSchema = new Schema({
name : String
, questions : [{type:ObjectId, ref:'questions'}]
,company : {type:ObjectId, ref:'companies'}
...
});
var Quiz = mongoose.model('quizzes', QuizSchema);
Company Model
var CompanySchema = new Schema({
name :String
...
});
I want to shuffle choices of each question per each query, and I am doing It as follows :
shuffle = function(v){
//+ Jonas Raoni Soares Silva
//# http://jsfromhell.com/array/shuffle [rev. #1]
for(var j, x, i = v.length; i; j = parseInt(Math.random() * i), x = v[--i], v[i] = v[j], v[j] = x);
return v;
};
app.get('/api/companies/:companyId/quizzes', function(req, res){
var Query = Quiz.find({company:req.params.companyId});
Query.populate('questions');
Query.exec(function(err, docs){
docs.forEach(function(doc) {
doc.questions.forEach(function(question) {
question.choices = shuffle(question.choices);
})
});
res.json(docs);
});
});
My Question is :
Could I randomize the choices array without looping through all documents as now I am doing?
shuffle = function(v){
//+ Jonas Raoni Soares Silva
//# http://jsfromhell.com/array/shuffle [rev. #1]
for(var j, x, i = v.length; i; j = parseInt(Math.random() * i), x = v[--i], v[i] = v[j], v[j] = x);
return v;
};
app.get('/api/companies/:companyId/quizzes', function(req, res){
var Query = Quiz.find({company:req.params.companyId});
Query.populate('questions');
Query.exec(function(err, docs){
var raw = docs.toObject();
//shuffle choices
raw.questions.map(el => shuffle(el.choices))
//if you need to shuffle the questions too
shuffle(raw.questions);
//if you need to limit the output questions, especially when ouput questions needs to be a subset of a pool of questions
raw.questions.splice(limit);
res.json(raw); // output quiz with shuffled questions and answers
});
});
The essence of the question comes down to "Can I randomly shuffle results and have MongoDB do the work for me?". Well yes you can, but the important thing to remember here is that "populate" is not longer going to be your friend in helping you do so and you will need to perform the work that is doing yourself.
The short part of this is we are going to "hand-off" your client side "shuffle" to mapReduce in order to process the shuffling of the "choices" on the server. Just for kicks, I'm adding in a technique to shuffle your "questions" as well:
var Query = Quiz.findOne({ company: "5382a58bb7ea27c9301aa9df" });
Query.populate('company', 'name -_id');
Query.exec(function(err,quiz) {
var shuffle = function(v) {
for(var j, x, i = v.length; i; j = parseInt(Math.random() * i), x = v[--i], v[i] = v[j], v[j] = x);
};
if (err)
throw err;
var raw = quiz.toObject();
shuffle( raw.questions );
Question.mapReduce(
{
map: function() {
shuffle( this.choices );
var found = -1;
for ( var n=0; n<inputs.length; n++ ) {
if ( this._id.toString() == inputs[n].toString() ) {
found = n;
break;
}
}
emit( found, this );
},
reduce: function() {},
scope: { inputs: raw.questions, shuffle: shuffle },
query: { "_id": { "$in": raw.questions } }
},
function(err,results) {
if (err)
throw err;
raw.questions = results.map(function(x) {
return x.value;
});
console.log( JSON.stringify( raw, undefined, 4 ) );
}
);
});
So the essential part of this is rather than allowing "populate" to pull all the related question information into your schema object, you are doing a manual replacement using mapReduce.
Note that the "schema document" must be converted to a plain object which is done by the .toObject() call in there in order to allow us to replace "questions" with something that would not match the schema type.
We give mapReduce a query to select the required questions from the model by simply passing in the "questions" array as an argument to match on _id. Really nothing directly different to what "populate" does for you behind the scenes, it's just that we are going to handle the "merge" manually.
The "shuffle" function is now executed on the server, which since it was declared as a var we can easily pass in via the "scope", and the "options" array will be shuffled before it is emitted, and eventually returned.
The other optional as I said was that we are also "shuffling" the questions, which is merely done by calling "shuffle" on just the _id values of the "questions" array and then passing this into the "scope". Noting that this is also passed to the query via $in but that alone does not guarantee the return order.
The trick employed here is that mapReduce at the "map" stage, must "emit" all keys in their ascending order to later stages. So by comparing the current _id value to where it's position is as an index value of the "inputs" array from scope then there is a positional order that can be emitted as the "key" value here to respect the order of the shuffle done already.
The "merging" then is quite simple as we just replace the "questions" array with the values returned from the mapReduce. There is a little help here from the .map() Array function here to clean up the results from the way mapReduce returns things.
Aside from the fact that your "options" are now actually shuffled on the server rather than through a loop, this should give you ideas of how to "custom populate" for other functions such as "slicing" and "paging" the array of referenced "questions" if that is something else you might want to look at.