how to pass complex string to pymongo for query by python? - python-3.x

I need to filter the webserver requests and setting a query for pymongo, its not so simple as I need to have "and", or "or" functionality for multiple fields.
I have filtered the get request, got the parameters, built the string to be passed to db..find. But it throws error. I have identified the error as because I am forming a string like this to passed to the function, now as its a string and not actually a dict, its throwing an error. What is the right way of doing it?
Actually, I have to get something like: {$and:[{Title:{"$regex":"Hong Kong"}},{Url:{"$regex":"hong"}}]}{'_id':0, 'Body':0}
The get request I am sending is: http://127.0.0.1:5000/getRequest?Title="Hong Kong protest"&Url="hong" Now the below thing gives the exact required string, but it throws an error as its not supposed to be string. Please help.
#app.route('/getRequest', methods=['GET'])
def request():
global connection
args = request.args
if len(args) > 1:
search_str = ""
for key, val in args.items():
search_str += '{'+key+':{"$regex":'+str(val)+'}},'
search_str = search_str[:-1]
display_dict={'id':0, 'Body':0}
final_search_str = "{$and:["+search_str+"]},{'_id':0, 'Body':0}"
#return(final_search_str)
# query_str = request.args.get('query_string')
db = connection['test']
collection = db['collect1']
output = []
for s in collection.find(final_search_str):
output.append({'Title' : s['Title'], 'Url' : s['Url']})
It should be dict which should be passed to the function. Any better way to do this complex query via pymongo?

You can do this using re and bson.regex.Regex module.
http://api.mongodb.com/python/current/api/bson/regex.html
import re
from bson.regex import Regex
query = {}
for key, val in args.items():
pattern = re.compile(val)
regex = Regex.from_native(regex)
query[key] = regex
for s in collection.find(query):
output.append({'Title' : s['Title'], 'Url' : s['Url']})

Related

DynamoDB Query Reserved Words using boto3 Resource?

I've been playing around with some "weirder" querying of DynamoDB with Reserved Words using boto3.resource method, and came across a pretty annoying issue which I can't resolve for quite some time (Always the same error sigh), and can't seem to find the answer anywhere.
My code is the following:
import logging
import boto3
from boto3.dynamodb.conditions import Key
logger = logging.getLogger()
logger.setLevel(logging.INFO)
TABLE_NAME = "some-table"
def getItems(record, table=None):
if table is None:
dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table(TABLE_NAME)
record = str(record)
get_item = table.query(
KeyConditionExpression=Key("#pk").eq(":pk"),
ExpressionAttributeNames={"#pk": "PK"},
ExpressionAttributeValues={":pk": record},
)
logger.info(
f"getItem parameters\n{json.dumps(get_item, indent=4,sort_keys=True, default=str)}"
)
return get_item
if __name__ == "__main":
record = 5532941
getItems(record)
It's nothing fancy, as I mentioned I'm just playing around, but I'm constantly getting the following error no matter what I try:
"An error occurred (ValidationException) when calling the Query operation: Value provided in ExpressionAttributeNames unused in expressions: keys: {#pk}"
As far as I understand in order to "replace" the reserved keys/values with something arbitrary you put it into ExpressionAttributeNames and ExpressionAttributeValues, but I can't wrap my head around as to why it's telling me that this key is not used.
I should mention that this Primary Key exists with this value in the record var in DynamoDB.
Any suggestions?
Thanks
If you're using Key then just provide the string values and don't be fancy with substitution. See this example:
https://github.com/aws-samples/aws-dynamodb-examples/blob/master/DynamoDB-SDK-Examples/python/WorkingWithQueries/query_equals.py
If you're writing an equality expression as a single string, then you need the substitution. See this example:
https://github.com/aws-samples/aws-dynamodb-examples/blob/master/DynamoDB-SDK-Examples/python/WorkingWithQueries/query-consistent-read.py

use jsonSlurper.parseText for text that has dash

How can I use jsonSlurper.parseText to parse "807-000" that has dash in it with groovy ?
You are generating the below string for parsing:
[807-000]
What I think you wanted is an json array containing a string:
["807-000"]
You could generate that json yourself:
def arr2 = "[" + arr.collect({ '"' + it + '"' }).join(",") + "]"
But why reinvent the wheel, when you can do it like this:
def arr2 = groovy.json.JsonOutput.toJson(arr)
It's not entirely clear what exactly do you want to do. parseText() is waiting for json to be input. I suggest several options for parsing.
def text = jsonSlurper.parseText("""{ "key": "807-000" } """)
Or did you mean that before the dash is the key, and after it is the value? If so then you can try this:
def map = "807-000".split("-").toSpreadMap()
map.each {row ->
def parsedText = jsonSlurper.parseText("""{ "${row.key}": "${row.value}" } """)
println(parsedText)
}
output is = [807:000]
How can I use jsonSlurper.parseText to parse "807-000" that has dash
in it with groovy ?
I am not sure what the challenge actually is. Something I can think of is possibly you are having trouble using Groovy property access to retrieve the value of a key when the key has a hyphen in it. You can do that by quoting the property name:
String jsonString = '''
{"807-000":"Eight O Seven"}
'''
def slurper = new JsonSlurper()
def json = slurper.parseText(jsonString)
// quote the property name which
// contains a hyphen...
String description = json.'807-000'
assert description == 'Eight O Seven'

groovy extract value from string

I got a string from a server response:
responseString:"{"session":"vvSbMInXHRJuZQ==","age":7200,"prid":"901Vjmx9qenYKw","userid":"user_1"}"
then I do:
responseString[1..-2].tokenize(',')
got:
[""session":"vvSbMInXHRJuZQ=="", ""age":7200", ""prid":"901Vjmx9qenYKw"", ""userid":"user_1""]
get(3) got:
""userid":"user_1""
what I need is the user_1, is there anyway I can actually get it? I have been stuck here, other json methods get similar result, how to remove the outside ""?
Thanks.
If you pull out the proper JSON from responseStr, then you can use JsonSlurper, as shown below:
def s = 'responseString:"{"session":"vvSbMInXHRJuZQ==","age":7200,"prid":"901Vjmx9qenYKw","userid":"user_1"}"'
def matcher = (s =~ /responseString:"(.*)"/)
assert matcher.matches()
def responseStr = matcher[0][1]
import groovy.json.JsonSlurper
def jsonSlurper = new JsonSlurper()
def json = jsonSlurper.parseText(responseStr)
assert "user_1" == json.userid
This code can help you get you to the userid.
def str= 'responseString:"{:"session":"vvSbMInXHRJuZQ==","age":7200,"prid":"901Vjmx9qenYKw","userid":"user_1","hdkshfsd":"sdfsdfsdf"}'
def match = (str=~ /"userid":"(.*?)"/)
log.info match[0][1]
this pattern can help you getting any of the values you want from the string. Try replacing userid with age, you will get that
def match = (str=~ /"age":"(.*?)"/)
#Michael code is also correct. Its just that you have clarified that you want the user Name to be specific

Getting the correct information from differently formulated queries

Howdo people,
I'm to put together a limited Q&A program that will allow the user to query Wikidata using SPARQL with very specific/limited query structures.
I've got the program going, but I'm running into issues when entering queries that are formulated differently.
def sparql_query(line):
m = re.search('What is the (.*) of (.*)?', line)
relation = m.group(1)
entity = m.group(2)
wdparams['search'] = entity
json = requests.get(wdapi, wdparams).json()
for result in json['search']:
entity_id = result['id']
wdparams['search'] = relation
wdparams['type'] = 'property'
json = requests.get(wdapi, wdparams).json()
for result in json['search']:
relation_id = result['id']
fire_sparql(entity_id, relation_id)
return fire_sparql
As you can see, this only works with queries following that specific structure, for example "What is the color of night?" Queries along the lines of 'What are the ingredients of pizza?' simply would cause the program to crash because it doesn't follow the 'correct' structure as set in the code. As such, I would like it to be able to differentiate between different types of query structures ("What is.." and "What are.." for example) and still collect the needed information (relation/property and entity).
This setup is required, insofar as I can determine, seeing as the property and entity need to be extracted from the query in order to get the proper results from Wikidata. This is unfortunately also what I'm running into problems with; I can't seem to use 'if' or 'while-or' statements without the code returning all sorts of issues.
So the question being: How can I make the code accept differently formulated queries whilst still retrieving the needed information from them?
Many thanks in advance.
The entirety of the code in case required:
#!/usr/bin/python3
import sys
import requests
import re
def main():
example_queries()
for line in sys.stdin:
line = line.rstrip()
answer = sparql_query(line)
print(answer)
def example_queries():
print("Example query?\n\n Ask your question.\n")
wdapi = 'https://www.wikidata.org/w/api.php'
wdparams = {'action': 'wbsearchentities', 'language': 'en', 'format': 'json'}
def sparql_query(line):
m = re.search('What is the (.*) of (.*)', line)
relation = m.group(1)
entity = m.group(2)
wdparams['search'] = entity
json = requests.get(wdapi, wdparams).json()
for result in json['search']:
entity_id = result['id']
wdparams['search'] = relation
wdparams['type'] = 'property'
json = requests.get(wdapi, wdparams).json()
for result in json['search']:
relation_id = result['id']
fire_sparql(entity_id, relation_id)
return fire_sparql
url = 'https://query.wikidata.org/sparql'
def fire_sparql(ent, rel):
query = 'SELECT * WHERE { wd:' + ent + ' wdt:' + rel + ' ?answer.}'
print(query)
data = requests.get(url, params={'query': query, 'format': 'json'}).json()
for item in data['results']['bindings']:
for key in item:
if item[key]['type'] == 'literal':
print('{} {}'.format(key, item[key]['value']))
else:
print('{} {}'.format(key, item[key]))
if __name__ == "__main__":
main()

Substituting Variables Value in Mongodb statement

My main intention is to dynamically change the Employees collection while using pymongo, and i was able to do it for insert commands, I am facing problems with the find command, no matter what i do exec() always returns None. but if i copy the string and run it value gets assigned to the variable.
can someone throw some light on why the exec is unable to return a resultset or assign a the resultset to a variable?
db.Employees.update_one(
{"id": criteria},
{
"$set": {
"name":name,
"age":age,
"country":country
}
}
)
from pymongo import MongoClient
import ast
client = MongoClient('localhost:27017')
db = client.TextClassifier
insert works
def mongo_insert_one(COLLECTION_NAME, JSON):
QUERY = """db.%(COLLECTION_NAME)s.insert_one( %(JSON)s )""" % locals();
exec(QUERY)
def mongo_retrive(COLLECTION_NAME, JSON):
resultset = None
query = """resultset = db.%(COLLECTION_NAME)s.find( %(JSON)s )""" % locals();
return resultset
print(mongo_retrive('hungry_intent', "{'Intent':'Hungry'}"))
neither this would work
resultset = exec(""" db.%(COLLECTION_NAME)s.find( %(JSON)s )""" % locals();)
this would not work for an entirely different reason,it says If you meant to call the 'locals' method on a 'Database' object it is failing because no such method exists.
resultset = db.locals()[COLLECTION_NAME].find()
PyMongo Database objects support bracket notation to access a named collection, and PyMongo's included bson module provides a much better JSON decoder than "eval":
from bson import json_util
COLLECTION_NAME = 'hungry_intent'
JSON = "{'Intent':'Hungry'}"
print(list(db[COLLECTION_NAME].find(json_util.loads(JSON))))
This will be faster and more reliable than your "eval" code, and also prevents the injection attack that your "eval" code is vulnerable to.
If you can avoid using JSON at all it could be preferable:
COLLECTION_NAME = 'hungry_intent'
QUERY = {'Intent':'Hungry'}
print(list(db[COLLECTION_NAME].find(QUERY)))

Resources