DHT InfoHash Lookup sequence. PeerID vs InfoHash - bittorrent

I know there is a previous question about this somewhere on SO, but I cannot find it again. The relationship between NodeId and InfoHash.
Is the following diagram roughly correct?
Background (no need to read)
I'm trying to implement my own DHT/bittorrent application in Java.
I know that there are already some excellent implementations that I will never better. But this is purely a hedonistic pursuit.
What was it Kennedy said? "we choose to do this not because it is easy.."
I have conquered the easy part, which is to write the low level socket handling and Remote Procedure Call syntax etc.
Now I am on to the hard part, I must behave responsibly and service incoming requests on the DHT. (maintain KBuckets etc.)

Yes that diagram is correct. Here some python code that implements FIND_VALUE algorithm you describe:
async def _get(self, key):
"""Fetch the value associated with KEY from the network"""
uid = pack(key)
queried = set()
while True:
# retrieve the k nearest peers and remove already queried peers
peers = await self.peers((None, None), uid)
peers = [address for address in peers if address not in queried]
# no more peer to query, the key is not found in the dht
if not peers:
raise KeyError(unpack(uid))
# query selected peers
queries = dict()
for address in peers:
query = self._protocol.rpc(address, "value", uid)
queries[address] = query
responses = await gather(queries, return_exceptions=True)
for (address, response) in responses.items():
queried.add(address)
if isinstance(response, Exception):
continue
elif response[0] == b"VALUE":
value = response[1]
if hash(value) == unpack(uid):
# store it
#h.transactional
def add(tr, key, value):
tr.add("QADOM:MAPPING", key, "value", value)
await self._run(add, self._hoply, key, value)
# at last!
return value
else:
log.warning(
"[%r] bad value returned from %r", self._uid, address
)
await self.blacklist(address)
continue
elif response[0] == b"PEERS":
await self._welcome_peers(response[1])
else:
await self.blacklist(address)
log.warning(
"[%r] unknown response %r from %r",
self._uid,
response[0],
address,
This is extract from qadom project peer.py.

Related

How can I use something stored in a DataBase? SQLite / Python

So, I am new at DataBases and I have a question. I first made a re-search in the internet but could not find anything or actually I could not understand correctly what they were explaining. Before starting with my question I want to say that I am currently working in a Discord Bot in Python as Main Language and I am trying to create a per-server-prefix system. I have already made it once but in .JSON format. Then I heard that using .JSON to save this kind of Data is not actually good so I moved to DataBases. Now, my question is:
I have stored the guild_id and prefix in my DB, how could I use that for a per-server-prefix system? I have not tried anything yet except writing the Code to store both Texts. I would really love if anyone could explain to me not just tell me the Code! Any answer will be appreciated <3.
Main.py:
def get_prefix(client, message):
db = sqlite3.connect("main.sqlite")
cursor = db.cursor()
cursor.execute(f"SELECT prefix FROM prefixes WHERE guild_id = {message.guild.id}")
result = cursor.fetchone()
if result is None:
return "/"
else:
client = commands.Bot(command_prefix = get_prefix)
Prefixes.py (COGS):
#commands.command()
#commands.has_permissions(administrator=True)
async def prefix(self, ctx, prefix=None):
db = sqlite3.connect("main.sqlite")
cursor = db.cursor()
cursor.execute(f"SELECT prefix FROM prefixes WHERE guild_id = ?", ctx.guild.id)
result = cursor.fetchone()
if result is None:
sql = ("INSERT INTO prefixes(guild_id, prefix) VALUES(?,?)")
val = (ctx.guild.id, prefix)
await ctx.channel.send(f"{prefix}")
elif result is not None:
sql = ("UPDATE prefixes SET prefix = ? WHERE guild_id = ?")
val = (prefix, ctx.guild.id)
await ctx.channel.send(f"update `{prefix}`")
cursor.execute(sql, val)
db.commit()
cursor.close()
db.close()
That is pretty much the whole code. If you think anything should be changed or have any suggestions, answer in the comments!
All you need to do is, after the else, put return result. For example:
result = cursor.fetchone()
if result is None:
return "/"
else:
return result
cursor.fetchone() returns a tuple with each element requested in the row, as you only requested the prefix, it will contain just that (e.g: ("!",)) which is permitted as your command_prefix callable can return a string or tuple of strings.
Warning:
You may want to add a check to ensure that someone doesn't specify an empty string (A zero length string with nothing in it) as their prefix, otherwise your bot will attempt to run every message it sees as a command
References: discord.ext.commands.Bot.command_prefix

Signing and Verifying of Signature using Pycryptodome always fails

Hi I'm using the Pycryptodome package to try and verify signatures of transactions in a Blockchain. My issue is that when trying to add a new transaction, I first create a signature to be passed into a verify transaction method but for some reason it always fails even though the logic seems to be right when I compare it to the documentation. If anyone could point me where I'm going wrong it would be much appreciated. I have 3 methods that handle all of this and i'm not sure where the issue is
The generate keys method
def generate_keys(self):
# generate private key pair
private_key = RSA.generate(1024, Crypto.Random.new().read)
# public key comes as part of private key generation
public_key = private_key.publickey()
# return keys as hexidecimal representation of binary data
return (binascii.hexlify(public_key.exportKey(format='DER')).decode('ascii'), binascii.hexlify(private_key.exportKey(format='DER')).decode('ascii'))
The sign transaction method
def sign_transaction(self, sender, recipient, amount, key):
# convert transaction data to SHA256 string
hash_signer = SHA256.new(
(str(sender) + str(recipient) + str(amount)).encode('utf-8'))
# sign transaction
signature = pkcs1_15.new(RSA.importKey(
binascii.unhexlify(key))).sign(hash_signer)
# return hexidecimal representation of signature
return binascii.hexlify(signature).decode('ascii')
and the verify transaction method
#staticmethod
def verify_transaction(transaction):
# convert public key back to binary representation
public_key = RSA.importKey(binascii.unhexlify(
transaction.sender))
try:
# create signature from transaction data
hash_signer = SHA256.new(
(str(transaction.sender) + str(transaction.recipient) + str(transaction.amount)).encode('utf-8'))
pkcs1_15.new(public_key).verify(
hash_signer, binascii.unhexlify(transaction.signature))
return True
except ValueError:
return False
Once i've generated my key pair and attempt to use them to sign and verify transactions it always fails. I know this because it always returns false from the verify method leading me to believe a value error is always raised. Thanks in advance hopefully someone can help me out.

How to find nodes with exact info_hash in Bittorrent DHT?

In the documentation of DHT protocol for bittorrent,it is given that get_peers method is used for finding nodes with given info_hash.It says that if response contains "values" key,the queried node has returned the information about nodes containing exact info_hash.If node returns "nodes" key,it has returned K nodes closest to the result.Should we recursively call get_peers on returned nodes(closest) in order to reach till exact nodes(with same info_hash)?
Should we recursively call get_peers on returned nodes(closest) in order to reach till exact nodes(with same info_hash)?
Yes and no. You could use a recursive function if you are the LISP kind. That said, a while loop will do the job. Here is some python code that implement the FIND_VALUE algorithm with some comments:
async def _get(self, key):
"""Fetch the value associated with KEY from the network"""
uid = key
queried = set()
while True:
# retrieve the k nearest peers and remove already queried peers
peers = nearest(k, self.peers)
# no more peer to query, the key is not found in the dht
if not peers:
raise KeyError(uid)
# query selected peers
responses = dict()
for address in peers:
response = self._protocol.rpc(address, "value", uid)
responses[address] = response
# check responses: look for the value or add peers more near KEY
for (address, response) in responses.items():
queried.add(address)
if isinstance(response, Exception):
continue
elif response[0] == b"VALUE":
# value is found, return it
value = response[1]
if hash(value) == unpack(uid):
# at last!
return value
else:
log.warning(
"[%r] bad value returned from %r", self._uid, address
)
await self.blacklist(address)
continue
elif response[0] == b"PEERS":
# value not found but we got peers that are more near KEY
await self._welcome_peers(response[1])
This code is based on qadom's peer.py

Getting the correct information from differently formulated queries

Howdo people,
I'm to put together a limited Q&A program that will allow the user to query Wikidata using SPARQL with very specific/limited query structures.
I've got the program going, but I'm running into issues when entering queries that are formulated differently.
def sparql_query(line):
m = re.search('What is the (.*) of (.*)?', line)
relation = m.group(1)
entity = m.group(2)
wdparams['search'] = entity
json = requests.get(wdapi, wdparams).json()
for result in json['search']:
entity_id = result['id']
wdparams['search'] = relation
wdparams['type'] = 'property'
json = requests.get(wdapi, wdparams).json()
for result in json['search']:
relation_id = result['id']
fire_sparql(entity_id, relation_id)
return fire_sparql
As you can see, this only works with queries following that specific structure, for example "What is the color of night?" Queries along the lines of 'What are the ingredients of pizza?' simply would cause the program to crash because it doesn't follow the 'correct' structure as set in the code. As such, I would like it to be able to differentiate between different types of query structures ("What is.." and "What are.." for example) and still collect the needed information (relation/property and entity).
This setup is required, insofar as I can determine, seeing as the property and entity need to be extracted from the query in order to get the proper results from Wikidata. This is unfortunately also what I'm running into problems with; I can't seem to use 'if' or 'while-or' statements without the code returning all sorts of issues.
So the question being: How can I make the code accept differently formulated queries whilst still retrieving the needed information from them?
Many thanks in advance.
The entirety of the code in case required:
#!/usr/bin/python3
import sys
import requests
import re
def main():
example_queries()
for line in sys.stdin:
line = line.rstrip()
answer = sparql_query(line)
print(answer)
def example_queries():
print("Example query?\n\n Ask your question.\n")
wdapi = 'https://www.wikidata.org/w/api.php'
wdparams = {'action': 'wbsearchentities', 'language': 'en', 'format': 'json'}
def sparql_query(line):
m = re.search('What is the (.*) of (.*)', line)
relation = m.group(1)
entity = m.group(2)
wdparams['search'] = entity
json = requests.get(wdapi, wdparams).json()
for result in json['search']:
entity_id = result['id']
wdparams['search'] = relation
wdparams['type'] = 'property'
json = requests.get(wdapi, wdparams).json()
for result in json['search']:
relation_id = result['id']
fire_sparql(entity_id, relation_id)
return fire_sparql
url = 'https://query.wikidata.org/sparql'
def fire_sparql(ent, rel):
query = 'SELECT * WHERE { wd:' + ent + ' wdt:' + rel + ' ?answer.}'
print(query)
data = requests.get(url, params={'query': query, 'format': 'json'}).json()
for item in data['results']['bindings']:
for key in item:
if item[key]['type'] == 'literal':
print('{} {}'.format(key, item[key]['value']))
else:
print('{} {}'.format(key, item[key]))
if __name__ == "__main__":
main()

Compound dictionary keys

I have a particular case where using compound dictionary keys would make a task easier. I have a working solution, but feel it is inelegant. How would you do it?
context = {
'database': {
'port': 9990,
'users': ['number2', 'dr_evil']
},
'admins': ['number2#virtucon.com', 'dr_evil#virtucon.com'],
'domain.name': 'virtucon.com'
}
def getitem(key, context):
if hasattr(key, 'upper') and key in context:
return context[key]
keys = key if hasattr(key, 'pop') else key.split('.')
k = keys.pop(0)
if keys:
try:
return getitem(keys, context[k])
except KeyError, e:
raise KeyError(key)
if hasattr(context, 'count'):
k = int(k)
return context[k]
if __name__ == "__main__":
print getitem('database', context)
print getitem('database.port', context)
print getitem('database.users.0', context)
print getitem('admins', context)
print getitem('domain.name', context)
try:
getitem('database.nosuchkey', context)
except KeyError, e:
print "Error:", e
Thanks.
>>> def getitem(context, key):
try:
return context[key]
except KeyError:
pass
cur, _, rest = key.partition('.')
rest = int(rest) if rest.isdigit() else rest
return getitem(context[cur], rest)
>>> getitem(context, 'admins.0')
'number2#virtucon.com'
>>> getitem(context, 'database.users.0')
'number2'
>>> getitem(context, 'database.users.1')
'dr_evil'
I've changed the order of the arguments, because that's how most Python's functions work, cf. getattr, operator.getitem, etc.
The accepted solution (as well as my first attempt) failed due to the ambiguity inherent in the specs: '.' may be "just a separator" or a part of the actual key string. Consider, for example, that key may be 'a.b.c.d.e.f' and the actual key to use at the current level is 'a.b.c.d' with 'e.f' left over for the next-most-indented level. Also, the spec is ambiguous in another sense: if more than one dot-joined prefix of 'key' is present, which one to use?
Assume the intention is to try every such feasible prefix: this would possibly produce multiple solutions but we can arbitrarily return the first solution found in this case.
def getitem(key, context):
stk = [(key.split('.'), context)]
while stk:
kl, ctx = stk.pop()
if not kl: return ctx
if kl[0].isdigit():
ik = int(kl[0])
try: stk.append((kl[1:], ctx[ik]))
except LookupError: pass
for i in range(1, len(kl) + 1):
k = '.'.join(kl[:i])
if k in ctx: stk.append((kl[i:], ctx[k]))
raise KeyError(key)
I was originally trying to avoid all try/excepts (as well as recursion and introspection via hasattr, isinstance, etc), but one snuck back in: it's hard to check if an integer is an acceptable index/key into what might be either a dict or a list, without either some introspection to distinguish the cases, or (and it looks simpler here) a try/except, so I went fir te latter, simplicity being always near the top of my concerns. Anyway...
I believe variants on this approach (where all the "possible continuation-context pairs" that might still be feasible at any point are kept around) are the only working way to deal with the ambiguities I've explained above (of course, one might choose to collect all possible solutions, arbitrarily pick one of them according to whatever heuristic criterion is desire, or maybe raise if the ambiguity is biting so there are multiple solutions, etc, etc, but these are minor variants of this general idea).
The following code works. It checks for the special case of a single key having a period in it. Then, it splits the key apart. For each subkey, it tries to fetch the value from a list-like context, then it tries from a dictionary-type context, then it gives up.
This code also shows how to use unittest/nose, which is highly recommended. Test with "nosetests mysource.py".
Lastly, consder using Python's built-in ConfigParser class, which is really useful for this type of configuration task: http://docs.python.org/library/configparser.html
#!/usr/bin/env python
from nose.tools import eq_, raises
context = {
'database': {
'port': 9990,
'users': ['number2', 'dr_evil']
},
'admins': ['number2#virtucon.com', 'dr_evil#virtucon.com'],
'domain.name': 'virtucon.com'
}
def getitem(key, context):
if isinstance(context, dict) and context.has_key(key):
return context[key]
for key in key.split('.'):
try:
context = context[int(key)]
continue
except ValueError:
pass
if isinstance(context, dict) and context.has_key(key):
context = context[key]
continue
raise KeyError, key
return context
def test_getitem():
eq_( getitem('database', context), {'port': 9990, 'users': ['number2', 'dr_evil']} )
eq_( getitem('database.port', context), 9990 )
eq_( getitem('database.users.0', context), 'number2' )
eq_( getitem('admins', context), ['number2#virtucon.com', 'dr_evil#virtucon.com'] )
eq_( getitem('domain.name', context), 'virtucon.com' )
#raises(KeyError)
def test_getitem_error():
getitem('database.nosuchkey', context)
As the key to getitem must be a string (or a list which is passed in the recursive call) I've come up with the following:
def getitem(key, context, first=True):
if not isinstance(key, basestring) and not isinstance(key, list) and first:
raise TypeError("Compound key must be a string.")
if isinstance(key, basestring):
if key in context:
return context[key]
else:
keys = key.split('.')
else:
keys = key
k = keys.pop(0)
if key:
try:
return getitem(keys, context[k], False)
except KeyError, e:
raise KeyError(key)
# is it a sequence type
if hasattr(context, '__getitem__') and not hasattr(context, 'keys'):
# then the index must be an integer
k = int(k)
return context[k]
I am on the fence as to whether this is an improvement.
I'm leaving my original solution for posterity:
CONTEXT = {
"database": {
"port": 9990,
"users": ["number2", "dr_evil"]},
"admins": ["number2#virtucon.com", "dr_evil#virtucon.com"],
"domain": {"name": "virtucon.com"}}
def getitem(context, *keys):
node = context
for key in keys:
node = node[key]
return node
if __name__ == "__main__":
print getitem(CONTEXT, "database")
print getitem(CONTEXT, "database", "port")
print getitem(CONTEXT, "database", "users", 0)
print getitem(CONTEXT, "admins")
print getitem(CONTEXT, "domain", "name")
try:
getitem(CONTEXT, "database", "nosuchkey")
except KeyError, e:
print "Error:", e
But here's a version that implements an approach similar to the getitem interface suggested by doublep. I am specifically not handling dotted keys, but rather forcing the keys into separate nested structures because that seems cleaner to me:
CONTEXT = {
"database": {
"port": 9990,
"users": ["number2", "dr_evil"]},
"admins": ["number2#virtucon.com", "dr_evil#virtucon.com"],
"domain": {"name": "virtucon.com"}}
if __name__ == "__main__":
print CONTEXT["database"]
print CONTEXT["database"]["port"]
print CONTEXT["database"]["users"][0]
print CONTEXT["admins"]
print CONTEXT["domain"]["name"]
try:
CONTEXT["database"]["nosuchkey"]
except KeyError, e:
print "Error:", e
You might notice that what I've really done here is eliminate all ceremony regarding accessing the data structure. The output of this script is the same as the original except that it does not contain a dotted key. This seems like a more natural approach to me but if you really wanted to be able to handle dotted keys, you could do something like this I suppose:
CONTEXT = {
"database": {
"port": 9990,
"users": ["number2", "dr_evil"]},
"admins": ["number2#virtucon.com", "dr_evil#virtucon.com"],
"domain": {"name": "virtucon.com"}}
def getitem(context, dotted_key):
keys = dotted_key.split(".")
value = context
for key in keys:
try:
value = value[key]
except TypeError:
value = value[int(key)]
return value
if __name__ == "__main__":
print getitem(CONTEXT, "database")
print getitem(CONTEXT, "database.port")
print getitem(CONTEXT, "database.users.0")
print getitem(CONTEXT, "admins")
print getitem(CONTEXT, "domain.name")
try:
CONTEXT["database.nosuchkey"]
except KeyError, e:
print "Error:", e
I'm not sure what the advantage of this type of approach would be though.

Resources