Multiple key search in CouchDB - couchdb

Given the following object structure:
{
key1: "...",
key2: "...",
data: "..."
}
Is there any way to get this object from a CouchDB by quering both key1 and key2 without setting up two different views (one for each key) like:
select * from ... where key1=123 or key2=123
Kind regards,
Artjom
edit:
Here is a better description of the problem:
The object described above is a serialized game state. A game has exactly one creator user (key1) and his opponent (key2). For a given user I would like to get all games where he is involved (both as creator and opponent).

Emit both keys (or only one if equal):
function(doc) {
if (doc.hasOwnProperty('key1')) {
emit(doc.key1, 1);
}
if (doc.hasOwnProperty('key2') && doc.key1 !== doc.key2) {
emit(doc.key2, 1);
}
}
Query with (properly url-encoded):
?include_docs=true&key=123
or with multiple values:
?include_docs=true&keys=[123,567,...]
UPDATE: updated to query multiple values with a single query.

You could create a CouchDB view which produces output such as:
["key1", 111],
["key1", 123],
["key2", 111],
["key2", 123],
etc.
It is very simple to write a map view in javascript:
function(doc) {
emit(["key1", doc["key1"]], null);
emit(["key2", doc["key2"]], null);
}
When querying, you can query using multiple keys:
{"keys": [["key1", 123], ["key2", 123]]}
You can send that JSON as the data in a POST to the view. Or preferably use an API for your programming language. The results of this query will be each row in the view that matches either key. So, every document which matches on both key1 and key2 will return two rows in the view results.

I also was struggling with simular question, how to use
"select * from ... where key1=123 or key2=123".
The following view would allow you to lookup customer documents by the LastName or FirstName fields:
function(doc) {
if (doc.Type == "customer") {
emit(doc.LastName, {FirstName: doc.FirstName, Address: doc.Address});
emit(doc.FirstName, {LastName: doc.LastName, Address: doc.Address});
}
}

I am using this for a web service that queries all my docs and returns every doc that matches both the existence of a node and the query. In this example I am using the node 'detail' for the search. If you would like to search a different node, you need to specify.
This is my first Stack Overflow post, so I hope I can help someone out :)
***Python Code
import tornado.httpserver
import tornado.ioloop
import tornado.options
import tornado.web
import httplib, json
from tornado.options import define,options
define("port", default=8000, help="run on the given port", type=int)
class MainHandler(tornado.web.RequestHandler):
def get(self):
db_host = 'YOUR_COUCHDB_SERVER'
db_port = 5984
db_name = 'YOUR_COUCHDB_DATABASE'
node = self.get_argument('node',None)
query = self.get_argument('query',None)
cleared = None
cleared = 1 if node else self.write('You have not supplied an object node.<br>')
cleared = 2 if query else self.write('You have not supplied a query string.<br>')
if cleared is 2:
uri = ''.join(['/', db_name, '/', '_design/keysearch/_view/' + node + '/?startkey="' + query + '"&endkey="' + query + '\u9999"'])
connection = httplib.HTTPConnection(db_host, db_port)
headers = {"Accept": "application/json"}
connection.request("GET", uri, None, headers)
response = connection.getresponse()
self.write(json.dumps(json.loads(response.read()), sort_keys=True, indent=4))
class Application(tornado.web.Application):
def __init__(self):
handlers = [
(r"/", MainHandler)
]
settings = dict(
debug = True
)
tornado.web.Application.__init__(self, handlers, **settings)
def main():
tornado.options.parse_command_line()
http_server = tornado.httpserver.HTTPServer(Application())
http_server.listen(options.port)
tornado.ioloop.IOLoop.instance().start()
if __name__ == '__main__':
main()
***CouchDB Design View
{
"_id": "_design/keysearch",
"language": "javascript",
"views": {
"detail": {
"map": "function(doc) { var docs = doc['detail'].match(/[A-Za-z0-9]+/g); if(docs) { for(var each in docs) { emit(docs[each],doc); } } }"
}
}
}

Related

Problems integrating Python graphene with Apollo Federation

Using python to implement GraphQL across multiple microservices, some use Ariadne, and some use graphene (and graphene-Django). Because of the microservice architecture, it's chosen that Apollo Federation will merge the schemas from the different microservices.
With Ariadne, it's very simple (being schema first), and a small example:
from ariadne import QueryType, gql, make_executable_schema, MutationType, ObjectType
from ariadne.asgi import GraphQL
query = QueryType()
mutation = MutationType()
sdl = """
type _Service {
sdl: String
}
type Query {
_service: _Service!
hello: String
}
"""
#query.field("hello")
async def resolve_hello(_, info):
return "Hello"
#query.field("_service")
def resolve__service(_, info):
return {
"sdl": sdl
}
schema = make_executable_schema(gql(sdl), query)
app = GraphQL(schema, debug=True)
Now this is picked up with no problem with Apollo Federation:
const { ApolloServer } = require("apollo-server");
const { ApolloGateway } = require("#apollo/gateway");
const gateway = new ApolloGateway({
serviceList: [
// { name: 'msone', url: 'http://192.168.2.222:9091' },
{ name: 'mstwo', url: 'http://192.168.2.222:9092/graphql/' },
]
});
(async () => {
const { schema, executor } = await gateway.load();
const server = new ApolloServer({ schema, executor });
// server.listen();
server.listen(
3000, "0.0.0.0"
).then(({ url }) => {
console.log(`🚀 Server ready at ${url}`);
});
})();
For which I can run graphql queries against the server on 3000.
But, with using graphene, trying to implement the same functionality as Ariadne:
import graphene
class _Service(graphene.ObjectType):
sdl = graphene.String()
class Query(graphene.ObjectType):
service = graphene.Field(_Service, name="_service")
hello = graphene.String()
def resolve_hello(self, info, **kwargs):
return "Hello world!"
def resolve_service(self, info, **kwargs):
from config.settings.shared import get_loaded_sdl
res = get_loaded_sdl() # gets the schema defined later in this file
return _Service(sdl=res)
schema = graphene.Schema(query=Query)
# urls.py
urlpatterns = [
url(r'^graphql/$', GraphQLView.as_view(graphiql=True)),
]
,... now results in an error from the Apollo Federation:
GraphQLSchemaValidationError: Type Query must define one or more fields.
As I checked into this matter, I found that apollo calls the microservice with a graphql query of:
query GetServiceDefinition { _service { sdl } }
Running it on the microservice via Insomnia/Postman/GraphiQL with Ariadne gives:
{
"data": {
"_service": {
"sdl": "\n\ntype _Service {\n sdl: String\n}\n\ntype Query {\n _service: _Service!\n hello: String\n}\n"
}
}
}
# Which expanding the `sdl` part:
type _Service {
sdl: String
}
type Query {
_service: _Service!
hello: String
}
and on the microservice with Graphene:
{
"data": {
"_service": {
"sdl": "schema {\n query: Query\n}\n\ntype Query {\n _service: _Service\n hello: String\n}\n\ntype _Service {\n sdl: String\n}\n"
}
}
}
# Which expanding the `sdl` part:
schema {
query: Query
}
type Query {
_service: _Service
hello: String
}
type _Service {
sdl: String
}
So, they both are the same thing for defining how to get sdl, I checked into the microservice response, and found that graphene response is sending the correct data too,
with the Json response "data" being equal to:
execution_Result: OrderedDict([('_service', OrderedDict([('sdl', 'schema {\n query: Query\n}\n\ntype Query {\n _service: _Service\n hello: String\n}\n\ntype _Service {\n sdl: String\n}\n')]))])
So what could the reason be for Apollo Federation not being able to successfully get this microservice schema?
This pip library can help https://pypi.org/project/graphene-federation/
Just use build_schema, and it'll add _service{sdl} for you:
import graphene
from graphene_federation import build_schema
class Query(graphene.ObjectType):
...
pass
schema = build_schema(Query) # add _service{sdl} field in Query
You are on the good path on the other answer, but it looks like you are going to need to strip out some stuff from the printed version.
here is the way I have used in a github issue
i sum up my code here:
schema = ""
class ServiceField(graphene.ObjectType):
sdl = String()
def resolve_sdl(parent, _):
string_schema = str(schema)
string_schema = string_schema.replace("\n", " ")
string_schema = string_schema.replace("type Query", "extend type Query")
string_schema = string_schema.replace("schema { query: Query mutation: MutationQuery }", "")
return string_schema
class Service:
_service = graphene.Field(ServiceField, name="_service", resolver=lambda x, _: {})
class Query(
# ...
Service,
graphene.ObjectType,
):
pass
schema = graphene.Schema(query=Query, types=CUSTOM_ATTRIBUTES_TYPES)
The solution is actually a slight hack the schema that is automatically generated via graphene. I thought I had tried this already and it still worked, but I just did it again now but it broke.
So if in Ariadne, I add
schema {
query: Query
}
into the sdl, Apollo Federation also raises Type Query must define one or more fields.. Without it, it works fine. So then I also went to graphene and in the resolve_service function I did:
def resolve_service(self, info, **kwargs):
from config.settings.shared import get_loaded_sdl
res = get_loaded_sdl()
res = res.replace("schema {\n query: Query\n}\n\n", "")
return _Service(sdl=res)
And now graphene works too, so I guess the problem was something I overlooked, it seems that Apollo Federation cannot handle schema grammar of:
schema {
query: Query
}
Update 1
A line I didn't notice on Apollo's website is that:
This SDL does not include the additions of the federation spec above. Given an input like this:
This is clear when combining the services together in Federation as it will raise the error:
GraphQLSchemaValidationError: Field "_Service.sdl" can only be defined once.
So, although in the full schema for the microservice with define _Service.sdl, we want that information gone for the string of the full-schema that is returned as the return String for _Service.sdl
Update 2
The Apollo Federation is now working fine, with making sure that the string returned by the sdl field does not contain federation specs.
In graphene, I think each implementation might differ, but in general you want to replace the following:
res = get_loaded_sdl()
res = res.replace("schema {\n query: Query\n}\n\n", "")
res = res.replace("type _Service {\n sdl: String\n}", "")
res = res.replace("\n _service: _Service!", "")
And in Ariadne, just need to define two sdl's, one containing the federation specs (for the schema returned by the service), and one without federation specs (the one returned by the sdl field)
In case anyone is wondering, this is because graphene v2 uses commas instead of ampersands in interfaces
interface x implements y, z {
...
}
and this syntax no longer works, a workaround is to monkey-patch get_sdl
import re
from myproject import Query, Mutation
from graphene_federation import service, build_schema
# monkey patch old get_sdl
old_get_sdl = service.get_sdl
def get_sdl(schema, custom_entities):
string_schema = old_get_sdl(schema, custom_entities)
string_schema = string_schema.replace('\n', ' ')
pattern_types_interfaces = r'type [A-Za-z]* implements ([A-Za-z]+\s*,?\s*)+'
pattern = re.compile(pattern_types_interfaces)
string_schema = pattern.sub(lambda matchObj: matchObj.group().replace(',', ' &'), string_schema)
return string_schema
service.get_sdl = get_sdl
schema = build_schema(Query, mutation=Mutation)
and it works.

How to obtain virtual user id/details in gatling?

I am new to Gatling and Scala and I need your advice.
I would like to obtain load test for n-users. Each user have to send request for creating different accounts. This is obtained by sending json file with appropriate array of objects ('entries' in our case).
Each single user must send different login as our backend system is checking if username is unique. Somehow we have to be sure that gatling is sending different data for each virtual user and also for each entries as well.
We noticed that there us session element which represents virtual user's state. Problem is that code showed below will not work as Exec structure used with expression function does not send any request.
There is section that could work but I do not know how to determine third parameter to distinguish virtual user id. Please find below simple json file structure used for this test
{
"entries": [
{
"userName": "some user name",
"password": "some password"
}
}
and scala code with my comments
import io.gatling.core.Predef._
import io.gatling.http.Predef._
class UserCreationTest extends Simulation {
val profilesNumber = 2
val virtualUsers = 2
val httpConf = http
.baseURL("some url")
.acceptHeader("application/json")
.basicAuth("username", "password")
// This method will multiply 'entries' section in JSON 'entriesNumber' times
def createJsonUserEntries(entriesNumber: Int, users: List[String], userId : Long): String = {
val header = """{"entries": ["""
val footer = """]}"""
val builder = StringBuilder.newBuilder
for (i <- 0 until entriesNumber) {
val userIndex = (userId.toInt - 1) * entriesNumber + i
val userName = users(userIndex).get
val apiString =
s"""{
"userName": "${userName}"
"password": "password"
}"""
builder.append(apiString)
if (i != entriesNumber) {
builder.append(",")
}
}
header + builder.toString() + footer
}
// We do have method for generating user names based on profilesNumber and virtualUsers variables
// but for sake of this example lets hardcode 4 (profilesNumber * virtualUsers) user names
val usersList = List("user-1", "user-2", "user-3", "user-4")
//This will throw exception as no request was send. According to documentation function block is used to debugging/editing session
val scn = scenario("Create WiFi User Profile")
.exec(session => {
http("CreateUserProfile")
.post("/userProfiles/create/")
.body(StringBody(
createJsonUserEntries(profilesNumber, userslList, session.userId).toString
)
).asJSON
session})
// This exec block will send a request but I do not know how to determine third param that should be virtual user Id
// To run this section please comment previous whole scenario block
/*
val scn = scenario("")
.exec(http("CreateUserProfile")
.post("/userProfiles/create/")
.body(StringBody(
createJsonUserEntries(profilesNumber, emailList, ???).toString
)
).asJSON
)
*/
setUp(scn.inject(atOnceUsers(virtualUsers)).protocols(httpConf))
}
Can you help me on that please? Is there any other way to do that in gatling? Thank you very much in advance
so you are trying to have each user have a unique userId?
you could create a feeder that does this
var userIdFeeder = (1 to 999999).toStream.map(i => Map("userId" -> i)).toIterator
val scn = scenario("")
.feed(userIdFeeder)
.exec(http("CreateUserProfile")
.post("/userProfiles/create/")
.body(StringBody(
createJsonUserEntries(profilesNumber, emailList, "${userId}").toString
)
).asJSON
)

Remove hyphens from keys in deeply nested map

I posted this question in the Groovy mailing lists, but I've not yet gotten an answer. I was wondering if someone can help here. I am re-posting relevant text from my original question.
I have an input json that’s nested, that is read via a JsonSlurper, and some of the keys have hyphens in them. I need to replace those keys that have hyphens with underscores and convert it back to json for downstream processing. I looked at the JsonGenerator.Options documentation and I could not find any documentation for this specific requirement.
I also looked through options to iterate through the Map that is produced from JsonSlurper, but unfortunately I’m not able to find an effective solution that iterates through a nested Map, changes the keys and produces another Map which could be converted to a Json string.
Example Code
import groovy.json.*
// This json can be nested many levels deep
def inputJson = """{
"database-servers": {
"dc-1": [
"server1",
"server2"
]
},
"discovery-servers": {
"dc-3": [
"discovery-server1",
"discovery-server2"
]
}
}
"""
I need to convert the above to json that looks like the example below. I can iterate through and convert using the collectEntries method which only works on the first level, but I need to do it recursively, since the input json can be an nested many levels deep.
{
"database_servers": {
"dc_1": [
"server1",
"server2"
]
},
"discovery_servers": {
"dc_3": [
"discovery-server1",
"discovery-server2"
]
}
}
Seems like you just need a recursive method to process the slurped Map and its sub-Maps.
import groovy.json.JsonSlurper
JsonSlurper slurper = new JsonSlurper()
def jsonmap = slurper.parseText( inputJson )
Map recurseMap( def inputMap ) {
return inputMap.collectEntries { key, val ->
String newkey = key.replace( "-", "_" )
if ( val instanceof Map ) {
return [ newkey, recurseMap( val ) ]
}
return [ newkey, val ]
}
}
def retmap = recurseMap( jsonmap )
println retmap // at this point you can use output this however you like

Finding all properties for a schema-less vertex class

I have a class Node extends V. I add instances to Node with some set of document type information provided. I want to query the OrientDB database and return some information from Node; to display this in a formatted way I want a list of all possible field names (in my application, there are currently 115 field names, only one of which is a property used as an index)
To do this in pyorient, the only solution I found so far is (client is the name of the database handle):
count = client.query("SELECT COUNT(*) FROM Node")[0].COUNT
node_records = client.query("SELECT FROM Node LIMIT {0}".format(count))
node_key_list = set([])
for node in node_records:
node_key_list |= node.oRecordData.keys()
I figured that much out pretty much through trial and error. It isn't very efficient or elegant. Surely there must be a way to have the database return a list of all possible fields for a class or any other document-type object. Is there a simple way to do this through either pyorient or the SQL commands?
I tried your case with this dataset:
And this is the structure of my class TestClass:
As you can see from my structure only name, surname and timeStamp have been created in schema-full mode, instead nameSchemaLess1 and nameSchemaLess1 have been inserted into the DB in schema-less mode.
After having done that, you could create a Javascript function in OrientDB Studio or Console (as explained here) and subsequently you can recall it from pyOrient by using a SQL command.
The following posted function retrieves all the fields names of the class TestClass without duplicates:
Javascript function:
var g = orient.getGraph();
var fieldsList = [];
var query = g.command("sql", "SELECT FROM TestClass");
for (var x = 0; x < query.length; x++){
var fields = query[x].getRecord().fieldNames();
for (var y = 0; y < fields.length; y++) {
if (fieldsList == false){
fieldsList.push(fields[y]);
} else {
var fieldFound = false;
for (var z = 0; z < fieldsList.length; z++){
if (fields[y] == fieldsList[z]){
fieldFound = true;
break;
}
}
if (fieldFound != true){
fieldsList.push(fields[y]);
}
}
}
}
return fieldsList;
pyOrient code:
import pyorient
db_name = 'TestDatabaseName'
print("Connecting to the server...")
client = pyorient.OrientDB("localhost", 2424)
session_id = client.connect("root", "root")
print("OK - sessionID: ", session_id, "\n")
if client.db_exists(db_name, pyorient.STORAGE_TYPE_PLOCAL):
client.db_open(db_name, "root", "root")
functionCall = client.command("SELECT myFunction() UNWIND myFunction")
for idx, val in enumerate(functionCall):
print("Field name: " + val.myFunction)
client.db_close()
Output:
Connecting to the server...
OK - sessionID: 54
Field name: name
Field name: surname
Field name: timeStamp
Field name: out_testClassEdge
Field name: nameSchemaLess1
Field name: in_testClassEdge
Field name: nameSchemaLess2
As you can see all of the fields names, both schema-full and schema-less, have been retrieved.
Hope it helps
Luca's answer worked. I modified it to fit my tastes/needs. Posting here to increase the amount of OrientDB documentation on Stack Exchange. I took Luca's answer and translated it to groovy. I also added a parameter to select the class to get fields for and removed the UNWIND in the results. Thank you to Luca for helping me learn.
Groovy code for function getFieldList with 1 parameter (class_name):
g = orient.getGraph()
fieldList = [] as Set
ret = g.command("sql", "SELECT FROM " + class_name)
for (record in ret) {
fieldList.addAll(record.getRecord().fieldNames())
}
return fieldList
For the pyorient part, removing the database connection it looks like this:
node_keys = {}
ret = client.command("SELECT getFieldList({0})".format("'Node'"))
node_keys = ret[0].oRecordData['getFieldList']
Special notice to the class name; in the string passed to client.command(), the parameter must be encased in quotes.

Retrieving all Documents from couchdb using Node.js

I am writing a simple test app to experiment with the functionality of node.js and couchdb, so far i am loving it, but i ran in a snag. i have looked for and wide but can't seem to find an answer. My test server(a simple address book) does 2 things:
if the user goes to localhost:8000/{id} then my app returns the name and address of the user with that id.
if the user goes to localhost:8000/ then my app needs to return a list a names that are hyperlinks and takes them to the page localhost:8000/{id}.
I was able to get the first requirement working. i cant not seem to find how to retrieve a list of all names from my couchdb. that is what i need help with. here is my code:
var http = require('http');
var cradle = require('cradle');
var conn = new(cradle.Connection)();
var db = conn.database('users');
function getUserByID(id) {
var rv = "";
db.get(id, function(err,doc) {
rv = doc.name;
rv += " lives at " + doc.Address;
});
return rv;
}
function GetAllUsers() {
var rv = ""
return rv;
}
var server = http.createServer(function(req,res) {
res.writeHead(200, {'Content-Type':'text/plain'});
var rv = "" ;
var id = req.url.substr(1);
if (id != "")
rv = getUserByID(id);
else
rv = GetAllUsers();
res.end(rv);
});
server.listen(8000);
console.log("server is runnig");
As you can see, I need to fill in the GetAllUsers() function. Any help would be appreciated. Thanks in advance.
I would expect you to be doing something like (using nano, which is a library I authored):
var db = require('nano')('http://localhost:5984/my_db')
, per_page = 10
, params = {include_docs: true, limit: per_page, descending: true}
;
db.list(params, function(error,body,headers) {
console.log(body);
});
I'm not pretty sure what you are trying to accomplish with http over there but feel free to head to my blog if you are looking for some more examples. Just wrote a blog post for people getting started with node and couch
As said above it will come a time when you will need to create your own view. Check up the CouchDB API Wiki, then scan thru the book, check what are design documents, then if you like you can go and check the test code I have for view generation and querying.
You can create a CouchDB view which will list the users. Here are several resources on CouchDB views which you should read in order to get a bigger picture on this topic:
Introduction to CouchDB Views
Finding Your Data with Views
View Cookbook for SQL Jockeys
HTTP View API
So let's say you have documents structured like this:
{
"_id": generated by CouchDB,
"_rev": generated by CouchDB,
"type": "user",
"name": "Johny Bravo",
"isHyperlink": true
}
Then you can create a CouchDB view (the map part) which would look like this:
// view map function definition
function(doc) {
// first check if the doc has type and isHyperlink fields
if(doc.type && doc.isHyperlink) {
// now check if the type is user and isHyperlink is true (this can also inclided in the statement above)
if((doc.type === "user") && (doc.isHyperlink === true)) {
// if the above statements are correct then emit name as it's key and document as value (you can change what is emitted to whatever you want, this is just for example)
emit(doc.name, doc);
}
}
}
When a view is created you can query it from your node.js application:
// query a view
db.view('location of your view', function (err, res) {
// loop through each row returned by the view
res.forEach(function (row) {
// print out to console it's name and isHyperlink flag
console.log(row.name + " - " + row.isHyperlink);
});
});
This is just an example. First I would recommend to go through the resources above and learn the basics of CouchDB views and it's capabilities.

Resources