In CouchDB _users, I'm making user id's and roles with emails in them.
Are you aware of any special CouchDB problems this causes?
I can't find any docs on valid values for these fields. They appear to be docs, so it seems ok.
{
"_id": "org.couchdb.user:some#email.com",
"_rev": "1-0bb5ba9dd3e989a28bc8282efaf32aa2",
"password_scheme": "pbkdf2",
"iterations": 10,
"type": "user",
"name": "some#email.com",
"roles": [
"f#soddddddddddddddddddddddddddme#examddddddddddple.com"
],
"derived_key": "f1f41961688ffd35addebdd0ece7714b08242c5e",
"salt": "3d299831afccb98c39ddeb3308275acb"
}
CouchDB Core Dev here. There are no semantic limits to either the _id field roles.
The only thing is that the _id needs to start with org.couchdb.user:.
Roles are just arrays of strings, anything that goes into a string can be a role.
General advice is to keep things short, but email-addresses are totally within the realm of applicable values.
Related
I would like to return items where a nested key exists. I have the following table:
"users": [
{
"active": true,
"apps": {
"app-name-1": {
"active": true,
"group": "aaaaaaaaa",
"settings": {}
}
},
"username: "user1"
},
{
"active": true,
"apps": {
"app-name-2": {
"active": true,
"group": "bbbbbb",
"settings": {}
}
},
"username: "user2"
]
So I want to return all users that have "app-name-1" under "apps". Which operation is the best for this purpose?
The question you need to ask yourself isn't just the "operation", but also how do you model your data in DynamoDB. I.e., how does that JSON array you showed translates into a DynamoDB table, with hash and sort keys?
While DynamoDB nominally does supports nested attributes, this support is actually only partial, with some features (notably secondary indexes) not supporting them, so as I'll show now it is better not to use them. To model your data without nested attributes, what you can do is to use a hash key "username" and sort key "appname". Each item in this table is one app belonging to one user. The user's "active" flag is a bit of a problem in this modeling, but you can implement it by using a fake appname for storing such user parameters.
This modeling makes it efficient to list all applications belonging to one user (I assume you need this feature as well...) but not all users with a certain application. But you were looking for the reverse operation - to get a list of users given an app name.
You can get this reverse with a Scan operation but this is a full-table scan, and accordingly can be very slow and expensive (you'll be paying to read the entire database, even if only part of the data is actually returned to the user).
If efficient search by app is important, you should create a secondary index (GSI) whose hash key is app-name and sort key is user (i.e., the opposite key order from that of the base table). You can then query this index to get - efficiently - the list of usernames that have this app.
Note that such a GSI wouldn't have been possible if you were to insist of modeling your "user" item with nested attributes, because GSIs don't support nested attributes as the key.
I am trying to determine the best document schema for a project for couchdb (2.3.1). In researching this I am finding some conflicting information and no relevant guidelines for the latest version of couchdb and similar scenarios. If this data does not lend itself to couchdb or a different method other than whats detailed below is prefered, I would like to better understand why.
My scenario is to track the manufacturing details of widgets:
100,000-300,000 widget types must be tracked
Each widget type is manufactured between 200-1,800 times a day
Widget type manufacturing may burst to ~10,000 in a day
Each widget creation and its associated details must be recorded and updated
Widget creation is stored for 30 days
Query widget details by widget type and creationStartTime/creationEndTime
I am not concerned with revisions, and can just update and use the same _rev if this may increase performance
Method 1:
{
"_id": "*",
"_rev": "*",
"widgetTypeId": "1831",
"creation": [{
"creationId" "da17faef-3591-4579-b5f6-ff0a719a6da7",
"creationStartTime": 1556471139,
"creationEndTime": 1556471173,
"color": "#ffffff",
"styleId": "92811",
"creatorId": "82812"
},{
"creationId" "893fede7-3874-44ed-b290-7001b4901bc9",
"creationStartTime": 1556471481,
"creationEndTime": 1556471497,
"color": "#cccccc",
"styleId": "75343",
"creatorId": "3211"
}]
}
Using method one would limit my document creation to 100,000-300,000 documents. However, these documents would be very tall and frequently updated.
Method 2:
{
"_id": "*",
"_rev": "*",
"widgetTypeId": "1831",
"creationId" "da17faef-3591-4579-b5f6-ff0a719a6da7",
"creationStartTime": 1556471139,
"creationEndTime": 1556471173,
"color": "#ffffff",
"styleId": "92811",
"creatorId": "82812"
},{
"_id": "*",
"_rev": "*",
"widgetTypeId": "1831",
"creationId" "893fede7-3874-44ed-b290-7001b4901bc9",
"creationStartTime": 1556471481,
"creationEndTime": 1556471497,
"color": "#cccccc",
"styleId": "75343",
"creatorId": "3211"
}
Method 2 creates a tall database
It's a common problem to be faced with. In general terms, small, immutable documents will likely be more performant than few, huge, mutable documents. The reasons for this include:
There is no support for partial updates (patch) in CouchDB. So if you need to insert data into an array in a big document, you need to fetch all of the data, unpack the json, insert the data, repack the json and send the whole thing back to CouchDB over the wire.
Larger documents provide for more internal overheads, too, especially when it comes to indexing.
It's best to let the data that change as a unit make up a document. Ever-growing lists in documents is a bad idea.
It seems to me that your second alternative is a perfect fit for what you want to achieve: a set of small documents that can be made immutable. Then make a set of views so you can query on time ranges and widget type.
we are building a product on LUIS / Microsoft Bot framework and one of the doubt we have is Person Name understanding. The product is set to use by anyone by just signing up to our website. Which means any company who is signing up can have any number of employees with any name obviously.
What we understood is the user entity is not able to recognize all names. We have created a phrase list but as per we know there is a limit to phrase list (10K or even if its 100K) and names in the world can never have a limit. The other way we are thinking is to not train the entity with utterances. However if we have 100s of customers with 1000s of users each, the utterances will not be a good idea in that case.
I do not see any other way of handling this situation. Probably I am missing something here? Anyone faced similar problem and how it is handled?
The worst case would be to create a separate LUIS instance for each customer but that's really a big task to do only because we cant handle names.
As you might already know, a person's name could literally be anything: e.g. an animal, car, month, or color. So, there isn't any definitive way to identify something as a name. The closest you can come is via text analysis parts of speech and either taking a guess or comparing to an existing list. LUIS or any other NLP tool is unlikely to help with this. Here's one approach that might work out better. Try something like Microsoft's Text Analytics cognitive service, with a POST to the Key Phrases endpoint, like this:
https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/keyPhrases
and the body:
{
"documents": [
{
"language": "en-us",
"id": "myid",
"text": "Please book a flight for John Smith at 2:30pm on Wednesday."
}
]
}
That returns:
{
"languageDetection": {
"documents": [
{
"id": "e4263091-2d54-4ab7-b660-d2b393c4a889",
"detectedLanguages": [
{
"name": "English",
"iso6391Name": "en",
"score": 1.0
}
]
}
],
"errors": []
},
"keyPhrases": {
"documents": [
{
"id": "e4263091-2d54-4ab7-b660-d2b393c4a889",
"keyPhrases": [
"John Smith",
"flight"
]
}
],
"errors": []
},
"sentiment": {
"documents": [
{
"id": "e4263091-2d54-4ab7-b660-d2b393c4a889",
"score": 0.5
}
],
"errors": []
}
}
Notice that you get "John Smith" and "flight" back as key phrases. "flight" is definitely not a name, but "John Smith" might be, giving you a better idea of what the name is. Additionally, if you have a database of customer names, you can compare the value to a customer name, either exact or soundex, to increase your confidence in the name.
Sometimes, the services don't give you an 100% answer and you have to be creative with work-arounds. Please see the Text Analytics API docs for more info.
Have asked this question to few MS guys in my local region however it seems there is no way LUIS at moment can identify names.
Its not good as being NLP, it is not able to handle such things :(
I found wit.ai (best so far) in identifying names and IBM Watson is also good upto some level. Lets see how they turn out in future but for now I switched to https://wit.ai
When designing the endpoints for an activity resource that provides information on the activity of other resources such as users and organisations we are struggling with naming conventions.
What would be more semantic:
/organisations/activity
/organisations/activity/${activityId}
/users/activity
/users/activity/${activityId}
OR
/activity/users/${activityId}
/activity/users
/activity/organisations/${activityId}
/activity/organisations
There's not a generic answer for this, especially since the mechanisms doing the lookup/retrieval at the other end, and associated back-ends vary so drastically, not to mention the use case purpose and intended application.
That said, assuming for all intents and purposes the "schema" (or ... endpoint convention from the point of view of the end user) was just going to be flat, I have seen many more of the latter activity convention, as that is the actual resource, which is what many applications and APIs are developed around.
I've come to expect the following style of representation from APIs today (how they achieve the referencings and mappings is a different story, but from the point of view of API reference)
-
{
"Activity": [
{
"date": "1970-01-01 08:00:00",
"some_other_resource_reference_uuid": "f1c4a41e-1639-4e35-ba98-e7b169d1c92d",
"user": "b3ababc4-461b-404a-a1a2-83b4ca8c097f",
"uuid": "0ccf1b41-aecf-45f9-a963-178128096c97"
}
],
"Users": [
{
"email": "johnanderson#mycompany.net",
"first": "John",
"last": "Anderson",
"user_preference_1": "somevalue",
"user_property_1": "somevalue",
"uuid": "b3ababc4-461b-404a-a1a2-83b4ca8c097f"
}
]
}
The StackExchange API allows retrieving objects through multiple methods also:
For example, the User type look like this:
-
{
"view_count": 1000,
"user_type": "registered",
"user_id": 9999,
"link": "http://example.stackexchange.com/users/1/example-user",
"profile_image": "https://www.gravatar.com/avatar/a007be5a61f6aa8f3e85ae2fc18dd66e?d=identicon&r=PG",
"display_name": "Example User"
}
And on the Question type, the same user is shown underneath the owner object :
-
{
"owner": {
"user_id": 9999,
"user_type": "registered",
"profile_image": "https://www.gravatar.com/avatar/a007be5a61f6aa8f3e85ae2fc18dd66e?d=identicon&r=PG",
"display_name": "Example User",
"link": "https://example.stackexchange.com/users/1/example-user"
},
"is_answered": false,
"view_count": 31415,
"favorite_count": 1,
"down_vote_count": 2,
"up_vote_count": 3,
"answer_count": 0,
"score": 1,
"last_activity_date": 1494871135,
"creation_date": 1494827935,
"last_edit_date": 1494896335,
"question_id": 1234,
"link": "https://example.stackexchange.com/questions/1234/an-example-post-title",
"title": "An example post title",
"body": "An example post body"
}
On the Posts Type reference (Using this as a separate example because there is only a handful of methods to reach this type), you'll see an example down the bottom :
Methods That Return This Type
posts
posts/{ids}
users/{ids}/posts 2.2
me/posts 2.2
So whilst you can access resources (or "types" as it is on StackExchange), through a number of ways including filters and complex queries, there still exists the ability to see the desired resource through a number of more direct transparent URI conventions.
Different applications will clearly have different requirements. For example, the Gmail API is user based all the way - this makes sense from a users point of view given that in the context of the authenticated credential, you're separating one users objects from another.
This doesn't mean google uses the same convention for all of their APIs, their Activities API resource is all about the activity
Even looking at the Twitter API, there is a Direct Messages endpoint resource that has sender and receiver objects within.
I've not seen many API's at all that are limited to accessing resources purely via a user endpoint, unless the situation obviously calls for it, i.e. the Gmail example above.
Regardless of how flexible a REST API can be, the minimum I have come to expect is that some kind of Activity, location, physical object, or other entity is usually it's own resource, and the user association is plugged in and referenced at various degrees of flexibility (at a minimum, the example given at the top of this post).
It should be pointed out that in a true REST api the uri holds no meaning. It's the link relationships from your organizations and users resources that matter.
Clients should just discover those urls, and should also adapt to the new situation if you decide that you want a different url structure after all.
That being said, it's nice to have a logical structure for this type of thing. However, either is fine. You're asking for an opinion, there is not really a standard or best practice. That said, I would choose option #1.
I am trying to find an nth object using '_id', which is in the same document.
Any suggestions or references or code samples would be appreciated.
(e.g)
Document will look as below:
{
"_id": "xxxxx",
"name": "One",
"pocket": [{
"_id": "xxx123",
"name": "NestedOne",
"pocket": []
}, {
"_id": "xxx1234",
"name": "NestedTwo",
"pocket": [{
"_id": "xxx123456",
"name": "NestedTwoNested",
"pocket": [{"_id": "xxx123666",
"name": "NestedNestedOne",
"pocket": []
}]
}]
}]
}
The pockets shall hold more pockets and it is dynamic.
Here, I would like to search "pocket" using "_id" , say "xxx123456", but without using static reference.
Thanks again.
I highly recommend you change your document structure to something easier to manage/search, as this will only become more of a pain to work with.
Why not use multiple collections, like explained in this answer?
So an easy way to think about this for your situation, which I hope is easier for you to reason about than dropping some schema code...
Store all of your things as children in the same document. Give them unique _ids.
Store all of the contents of their pockets as collections. The collections simply hold all the ids that would normally be inside the pocket (instead of referencing the pockets themselves).
That way, a lot of your work can happen outside of DB calls. You just batch pull out the items you need when you need them, instead of searching nested documents!
However, if you can work with the entire document:
Looks like you want to do a recursive search a certain of number of levels deep. I'll give you a general idea with some pseudocode, in hopes that you'll be able to figure the rest out.
Say your function will be:
function SearchNDeep(obj, n, id){
/**
You want to go down 1 level, to pocket
see if pocket has any things in it. If so:
Check all of the things...for further pockets...
Once you've checked one level of things, increment the counter.
When the counter reaches the right level, you'd want to then see if the object you're checking has a `'_id'` of `id`.
**/
}
That's the general idea. There is a cleaner, recursive way to do this where you call SearchNDeep while passing a number for how deep you are, base case being no more levels to go, or the object is found.
Remember to return false or undefined if you don't find it, and the right object if you do! Good luck!