Flask/Pymongo Where/How do I apply validation? - python-3.x

This does not seem to apply validation to the collection. No exceptions are thrown and documents can have attributes of the wrong type. Perhaps I am doing it in the wrong section of the codebase? Right now it is in __init__.py
__init__.py
db = database(client, settings.mongo_db_name)
from api.models import Company
validation_level = 'strict'
if 'companies' not in db.collection_names():
db.create_collection('companies', validator=Company.validator, validationLevel=validation_level)
else:
db.command({
'collMod': 'companies',
'validator': Company.validator,
'validationLevel': validation_level,
})
Company Model:
from api import db
class Company(Model):
collection = db.companies
validator = {
'$jsonSchema': {
'bsonType': 'object',
'required': ['name', 'description'],
'properties': {
'logo': {
'bsonType': 'string',
},
'name': {
'bsonType': 'string',
'description': 'name of company is required',
'minLength': 4,
},
'description': {
'bsonType': 'string',
'description': 'description of company is required',
'minLength': 4,
},
'website': {
'bsonType': 'string',
},
'request_delete': {
'bsonType': 'bool',
},
'deleted': {
'bsonType': 'bool',
},
}
}
}
As discussed here, I also tried this without success:
db.command(OrderedDict([
('collMod', 'companies'),
('validator', Company.validator),
('validationLevel', validation_level),
]))
If this were successful, would I see validation rules when running the following command?
pprint(db.command('collstats', 'companies'))
Update
I added OrderedDict to both command arguments and the validator. This works... when I run specific tests. It does not work with
python -m unittest discover
I'm using Python 3.6.8, PyMongo 3.8.0, and MongoDB 3.6.3

Usually, it's better to validate a document BEFORE inserting it to the database. In this case, you'll omit errors in schema

Related

Gremlin NodeJS query returns different results than Groovy

I have a query that I originally wrote in the console:
g.V().hasLabel('group')
.has('type', 'PowerUsers')
.local(__.union(
__.project('group').by(__.valueMap().by(__.unfold())),
__.inE().outV().project('user').by(__.valueMap().by(__.unfold())))
.fold()).unfold().toList()
I get something like:
==>{group={owner=A, group_id=21651399-91fd-4da4-8608-1bd30447e773, name=Group 8, type=PowerUsers}}
==>{user={name=John, user_id=91f5e306-77f1-4aa1-b9d0-23136f57142d}}
==>{user={name=Jane, user_id=7f133d0d-47f3-479d-b6e7-5191bea52459}}
==>{group={owner=A, group_id=ef8c81f7-7066-49b2-9a03-bad731676a8c, name=Group B, type=PowerUsers}}
==>{user={name=Max, user_id=acf6abb8-08b3-4fc6-a4cb-f34ff523d628}}
==>{group={owner=A, group_id=07dff798-d6db-4765-8d74-0c7be66bec05, name=Group C, type=PowerUsers}}
==>{user={name=John, user_id=91f5e306-77f1-4aa1-b9d0-23136f57142d}}
==>{user={name=Max, user_id=acf6abb8-08b3-4fc6-a4cb-f34ff523d628}}
When I run that query with NodeJS, I was expecting to get a similar result, but I don't. I get something like this:
[ { group:
{ owner: 'A',
group_id: '21651399-91fd-4da4-8608-1bd30447e773',
name: 'Group 8',
type: 'PowerUsers' } },
{ user:
{ name: 'John',
user_id: '91f5e306-77f1-4aa1-b9d0-23136f57142d'} },
{ user:
{ name: 'John',
user_id: '91f5e306-77f1-4aa1-b9d0-23136f57142d'} },
{ user:
{ name: 'Jane',
user_id: '7f133d0d-47f3-479d-b6e7-5191bea52459'} },
{ user:
{ name: 'Jane',
user_id: '7f133d0d-47f3-479d-b6e7-5191bea52459'} },
{ group:
{ owner: 'A',
group_id: 'ef8c81f7-7066-49b2-9a03-bad731676a8c',
name: 'Group B',
type: 'PowerUsers' } },
{ user:
{ name: 'Max',
user_id: 'acf6abb8-08b3-4fc6-a4cb-f34ff523d628' } },
...
Because I have the same users in different groups, I can't use dedup(), and if the results where the same in NodeJS as Groovy, that'd be perfect. Unfortunately, they are not, and I don't understand why the results in NodeJS are all messed up, considering that the query is exactly the same
I feel like you could just use a nested project() to keep your group with your users:
g.V().hasLabel('group')
.has('type', 'PowerUsers')
.project('group', 'users')
.by(__.valueMap().by(__.unfold()))
.by(__.in().project('user').by(__.valueMap().by(__.unfold()).fold()))
In this way the you don't have to worry about ordering anything. I think it might be more Gremlin-esque though to use the "group" as a key in a Map with the values being a list of users:
g.V().hasLabel('group')
.has('type', 'PowerUsers')
.group()
.by(__.valueMap().by(__.unfold()))
.by(__.in().project('user').by(__.valueMap().by(__.unfold()).fold()))

sequelize: find in json array

In sequelize i have a model:
{
modelId: {
type: DataTypes.UUID ,
allowNull: false,
primaryKey: true,
defaultValue: DataTypes.UUIDV4
}
name: DataTypes.STRING(1024),
type: DataTypes.STRING,
obj: DataTypes.JSON
}
and in DB, my obj is an array like this:
[{
id: '123456',
text: 'test',
name 'xpto'
},{
id: '32554',
text: 'test2',
name 'xpte'
},{
id: '36201',
text: 'test3',
name 'xpta'
}]
i tried these:
btp.findAll({
where: {
obj:{
[Op.contains]:[{id: req.body.id}]
}
},
attributes: ['modelId','name','type','obj']
})
but does not work, return this error:
{"name": "SequelizeDatabaseError",
"parent": {
"name": "error",
"length": 128,
"severity": "ERROR",
"code": "42704",
"file": "parse_coerce.c",
"line": "1832",
"routine": "enforce_generic_type_consistency",
"sql":"....."}
so, i need to find in database all entries have in obj, id: '123456'
my question is the same than this:
https://github.com/sequelize/sequelize/issues/7349
but thats does not working for me, i need to return all entries that contains...
i'm using "sequelize": "4.28.6", and "pg-hstore": "^2.3.2",
can any one help?
I'm not familiar with the specific error you're getting, but one potential issue is that you're using a JSON column instead of JSONB. In Postgres, JSON columns just store the raw JSON text, and so don't support the containment operator (#>) which is needed for Sequelize's "contains".
All you need to do to fix this is change the column definition in the model to:
obj: DataTypes.JSONB
The only other issue I can think of would be req.body.id having an invalid value. I'd suggest verifying that it's actually getting a valid ID string.
Beyond these 2 potential issues, the query you wrote should work.

Elasticsearch english language analyzer with nodejs

First time trying to implement elastic search using aws hosted service with nodejs. Referred to the official docs and came up with this:
//1. create index
client.indices.create({ index: 'products' });
//2. mapping
client.indices.putMapping({
index: 'products',
type: 'products',
body: {
properties: {
'product_name': {
'type': 'string',
"analyzer": "english"
},
'product_description': {
'type': 'string',
"analyzer": "english"
}
}
}
});
//3. import documents..
product_name: Testing
//4. search
client.search({
index: 'products',
type: 'products',
q: keywords,
analyzer: 'english',
size: 10
});
Now, if I search for 'Testing' or 'Token Testing', this returns 1 result. But if I test passing 'Test' or 'Tist' as a keyword, seems the analyzer isn't picking up and I get no results.
Updated: Added analyzer: 'english' according to #Val answer, but still no results for 'Test' or 'Tist'.
That's because when using the q parameter, a query_string query is made and the default analyzer of your input string is the standard analyzer.
You have two choices here:
A. Include the field name in your query string:
client.search({
index: 'products',
type: 'products',
q: 'product_name:' + keywords, <--- modify this
size: 10
});
B. Specify english as the query-time analyzer to produce the same tokens as has been previously indexed:
client.search({
index: 'products',
type: 'products',
q: keywords,
analyzer: 'english', <--- or add this
size: 10
});

SailsJs nothing happen after calling save method

Sorry, this is my first nodeJs app, I'm using SailsJs and this is my code
update: function (req, res) {
Cargo.findOne({id: req.param('id')}).exec(function (err,result) {
result.currency = 2;
if(err) {
console.log(err);
}
result.save(function(err,model){
console.log(err);
console.log(model);
console.log('in');
});
console.log("testing");
})
},
I keep getting "testing" in my console log and not getting any log in save method.
Am I doing it wrong?
Found the issue causing it not going into .save method
My model code are as follow
module.exports = {
tableName: "cargo",
attributes: {
'remark': {
type: 'text'
},
'shipper': {
type: 'integer',
integer: true,
model: 'Company',
columnName:"shipperID",
},
'consignee': {
type: 'integer',
integer: true,
columnName:"consigneeID",
model: 'Company'
},
'packing': {
"type": 'integer',
integer: true,
columnName:"packingID",
equals: function (value) {
return value;
}
},
'commodity': {
'type': "string",
maxLength: 512
},
'status': {
'type': "string",
maxLength: 25
},
'dateIn': {
'type': "datetime"
},
'currency': {
'type': 'integer'
},
'amt': {
'type': 'float'
},
'marking': {
'type': 'string',
maxLength: 25,
required: true
},
'createdAt': {
columnName: 'createdDate'
},
'updatedAt': {
columnName: 'lastModifiedDate'
},
'packages': {
collection: 'CargoPackage',
via: 'cargo'
}
}
};
It's due to "packing" attribute having this "equal" that cause the save method to goes into infinite loop or something. I thought this is a validator for making sure the value is what I want. Well I think its built for other purposes.
I guess for others with similar issues, it will be best to have the model you are working on be as basic as possible to make sure it's not that thing holding up the process.

Mongo/Mongoose Invalid atomic update value error

I am trying to write to write an update to a Mongo document using the Mongoose findOneAndUpdate function. Essentially, I have a document that has an array of another Schema in it, and when I attempt to append more of those schema type, I get the following error:
[Error: Invalid atomic update value for $__. Expected an object, received object]
I'm having a hard time figuring out what this error even means, much less what its source is.
The data I'm attempting to update is as follows:
{ section_id: 51e427ac550dabbb0900000d,
version_id: 7,
last_editor_id: 51ca0c4b5b0669307000000e,
changelog: 'Added modules via merge function.',
committed: true,
_id: 51e45c559b85903d0f00000a,
__v: 0,
modules:
[ { orderId: 0,
type: 'test',
tags: [],
data: [],
images: [],
content: ["Some Content Here"] },
{ orderId: 1,
type: 'test',
tags: [],
data: [],
images: [],
content: ["Some Content Here"] },
{ orderId: 2,
type: 'test',
tags: [],
data: [],
images: [],
content: ["Some Content Here"] },
{ orderId: 3,
type: 'test',
tags: [],
data: [],
images: [],
content: ["Some Content Here"] },
{ orderId: 4,
type: 'test',
tags: [],
data: [],
images: [],
content: ["Some Content Here"] },
{ orderId: 5,
type: 'test',
tags: [],
data: [],
images: [],
content: ["Some Content Here"] } ] }
The only difference is that when I retrieve it, there are three fewer modules, and I append some new ones to the array.
Would love to hear any thoughts, at least as to what the error means!
This is probably because the updated object is still a Mongoose object.
Try to convert it to a JS object before the findOneAndUpdate
object = object.toString()
And delete any potential ID attribute
delete object._id
Or simply
object = object.toObject();
I had the same problem and it turned out I was using $push incorrectly. I was doing
{$push:thing_to_push}
But it needed to be
{$push:{list_to_push_into:thing_to_push}}
#Magrelo and #plus led me to an answer that worked. Something like:
MyModel.findOneAndUpdate({ section_id: '51e427ac550dabbb0900000d' }, mongooseObject.toObject(), { upsert: true }, function(err, results) {
//...
});
Try passing update parameter value as string instead of mongoose model object. I was getting same error when I use to pass model object. Below is the code difference.
Code that was having issue:
updateUser: function(req, res) {
**var updatedUserModel = new Users(req.body);**
Users.findOneAndUpdate({tpx_id:req.params.id}, {$set:**updatedUserModel**}, function(err, User){
...
}
}
Working code:
updateUser: function(req, res) {
Users.findOneAndUpdate({tpx_id:req.params.id}, {$set:**req.body**}, function(err, User) {
...
}
I had the same issue. I ended up by using finOne() method
create a new one if no found, update the existing one if found.
I know there are two operations. but I just haven't find any way to do it in one step.
Another thing to check is if you are sending passing an array of changes to $set. The error message that I received when calling something like this:
db.products.update( { sku: "abc123" },
{ $set: [
{quantity: 500},
{instock: true}
]
}
)
Gave me the [Error: Invalid atomic update value for $set. Expected an object, received object]
Changing it to an object worked.
db.products.update( { sku: "abc123" },
{ $set: {
quantity: 500,
instock: true
}
}
)

Resources