I'm trying to form a Marvel API Call.
Here's a link on authorization:
https://developer.marvel.com/documentation/authorization
I'm attempting to create a server-side application, so according to the link above, I need a timestamp, apikey, and hash url parameters. The hash needs be a md5 hash of the form: md5(timestamp + privateKey + publicKey) and the apikey url param is my public key.
Here's my code, I'm making the request in Python 3, using the request library to form the request, the time library to form the timestamp, and the hashlib library to form the hash.
#request.py: making a http request to marvel api
import requests;
import time;
import hashlib;
#timestamp
ts = time.time();
ts_str = str(float(ts));
#keys
public_key = 'a3c785ecc50aa21b134fca1391903926';
private_key = 'my_private_key';
#hash and encodings
m_hash = hashlib.md5();
ts_str_byte = bytes(ts_str, 'utf-8');
private_key_byte = bytes(private_key, 'utf-8');
public_key_byte = bytes(public_key, 'utf-8');
m_hash.update(ts_str_byte + private_key_byte + public_key_byte);
m_hash_str = str(m_hash.digest());
#all request parameters
payload = {'ts': ts_str, 'apikey': 'a3c785ecc50aa21b134fca1391903926', 'hash': m_hash_str};
#make request
r = requests.get('https://gateway.marvel.com:443/v1/public/characters', params=payload);
#for debugging
print(r.url);
print(r.json());
Here's the output:
$python3 request.py
https://gateway.marvel.com:443/v1/public/characters...${URL TRUNCATED FOR READABILITY)
{'code': 'InvalidCredentials', 'message': 'That hash, timestamp, and key combination is invalid'}
$
I'm not sure what exactly is causing the combination to be invalid.
I can provide more info on request. Any info would be appreciated. Thank you!
EDIT:
I'm a little new to API calls in general. Are there any resources for understanding more about how to perform them? So far with my limited experience they seem very specific, and getting each one to work takes a while. I'm a college student and whenever I work in hackathons it takes me a long time just to figure out how to perform the API call. I admit I'm not experienced, but in general does figuring out new API's require a large learning curve, even for individuals who have done 10 or so of them?
Again, thanks for your time :)
I've also had similar issues when accessing the Marvel API key. For those that are still struggling, here is my templated code (that I use in a jupyter notebook).
# import dependencies
import hashlib #this is needed for the hashing library
import time #this is needed to produce a time stamp
import json #Marvel provides its information in json format
import requests #This is used to request information from the API
#Constructing the Hash
m = hashlib.md5() #I'm assigning the method to the variable m. Marvel
#requires md5 hashing, but I could also use SHA256 or others for APIS other
#than Marvel's
ts = str(time.time()) #This creates the time stamp as a string
ts_byte = bytes(ts, 'utf-8') #This converts the timestamp into a byte
m.update(ts_byte) # I add the timestamp (in byte format) to the hash
m.update(b"my_private_key") #I add the private key to
#the hash.Notice I added the b in front of the string to convert it to byte
#format, which is required for md5
m.update(b"b2aeb1c91ad82792e4583eb08509f87a") #And now I add my public key to
#the hash
hasht = m.hexdigest() #Marvel requires the string to be in hex; they
#don't say this in their API documentation, unfortunately.
#constructing the query
base_url = "https://gateway.marvel.com" #provided in Marvel API documentation
api_key = "b2aeb1c91ad82792e4583eb08509f87a" #My public key
query = "/v1/public/events" +"?" #My query is for all the events in Marvel Uni
#Building the actual query from the information above
query_url = base_url + query +"ts=" + ts+ "&apikey=" + api_key + "&hash=" +
hasht
print(query_url) #I like to look at the query before I make the request to
#ensure that it's accurate.
#Making the API request and receiving info back as a json
data = requests.get(query_url).json()
print(data) #I like to view the data to make sure I received it correctly
Give credit where credit is due, I relied on this blog a lot. You can go here for more information on the hashlib library. https://docs.python.org/3/library/hashlib.html
I noticed in your terminal your MD5 hash is uppercase. MD5 should output in lowercase. Make sure you convert to that.
That was my issue, I was sending an uppercase hash.
As mentioned above, the solution was that the hash wasn't formatted properly. Needed to be a hexadecimal string and the issue is resolved.
Your final URL should be like this:
http:// gateway.marvel.com/v1/public/characters?apikey=(public_key)&ts=1&hash=(md5_type_hash)
So, you already have public key in developer account. However, how can you produce md5_type_hash?
ts=1 use just 1. So, your pattern should be this:
1 + private_key(ee7) + public_key(aa3). For example: Convert 1ee7aa3 to MD5-Hash = 1ca3591360a252817c30a16b615b0afa (md5_type_hash)
You can create from this website: https://www.md5hashgenerator.com
Done, you can use marvel api now!
Related
I want to validate the signature of an S3 (hosted at digitalocean) presigned URL via Python. As far as I know, the signature consists of the full URL, with the secret key.
I've already tried things like AWS S3 presigned urls with boto3 - Signature mismatch, but that results in a different signature.
I want to check the signature given in the URL (of an image for example) by recreating it with the hashing algorithm.
How would I go about doing this?
I had the same problem and was hoping the boto package would provide an easy way to do this, but unfortunately it doesn't.
I also tried to use boto to create the same signature base on the url, but the problem is the timestamp (X-Amz-Date in the url)
To get the exact same signature, the timestamp provided in the url needs to be used for generating.
I went down the rabbit hole trying to 'override' the datetime but it seems like it's impossible.
So what's left is generating the signature from scratch, like you said you tried. The code in the question you linked does work but it's not straightforward.
Inspired by that link and the boto3 source, this is what I've created and it seems to work:
from urllib.parse import urlparse, parse_qs, urlencode, quote
import hashlib
import hmac
from django.conf import settings
from datetime import datetime, timedelta
def validate_s3_url(url: str, method='GET'):
"""
This check whether the signature in the given S3 url is valid,
considering the other parts of the url.
This requires that we have access to the (secret) access key
that was used to sign the request (the access key ID is
available in the url).
"""
parts = urlparse(url)
querydict = parse_qs(parts.query)
# get relevant query parameters
url_signature = querydict['X-Amz-Signature'][0]
credentials = querydict['X-Amz-Credential'][0]
algorithm = querydict['X-Amz-Algorithm'][0]
timestamp = querydict['X-Amz-Date'][0]
signed_headers = querydict['X-Amz-SignedHeaders'][0]
expires = querydict['X-Amz-Expires'][0]
timestamp_datetime = datetime.strptime(timestamp, "%Y%m%dT%H%M%SZ")
if timestamp_datetime + timedelta(
seconds=int(expires) if expires.isdigit() else 0) < datetime.utcnow():
return False
# if we have multiple access keys we could use access_key_id to get the right one
access_key_id, credential_scope = credentials.split("/", maxsplit=1)
host = parts.netloc
# important: in Python 3 this dict is sorted which is essential
canonical_querydict = {
'X-Amz-Algorithm': [algorithm],
'X-Amz-Credential': [credentials],
'X-Amz-Date': [timestamp],
'X-Amz-Expires': querydict['X-Amz-Expires'],
'X-Amz-SignedHeaders': [signed_headers],
}
# this is optional (to force download with specific name)
# if used, it's passed in as 'ResponseContentDisposition' Param when signing.
if 'response-content-disposition' in querydict:
canonical_querydict['response-content-disposition'] = querydict['response-content-disposition']
canonical_querystring = urlencode(canonical_querydict, doseq=True, quote_via=quote)
# build the request, hash it and build the string to sign
canonical_request = f"{method}\n{parts.path}\n{canonical_querystring}\nhost:{host}\n\n{signed_headers}\nUNSIGNED-PAYLOAD"
hashed_request = hashlib.sha256(canonical_request.encode('utf-8')).hexdigest()
string_to_sign = f"{algorithm}\n{timestamp}\n{credential_scope}\n{hashed_request}"
# generate signing key from credential scope.
signing_key = f"AWS4{settings.AWS_SECRET_ACCESS_KEY}".encode('utf-8')
for message in credential_scope.split("/"):
signing_key = hmac.new(signing_key, message.encode('utf-8'), hashlib.sha256).digest()
# sign the string with the key and check if it's the same as the one provided in the url
signature = hmac.new(signing_key, string_to_sign.encode('utf-8'), hashlib.sha256).hexdigest()
return url_signature == signature
This uses django settings to get the secret key but really it could come from anywhere.
Hi I'm using the Pycryptodome package to try and verify signatures of transactions in a Blockchain. My issue is that when trying to add a new transaction, I first create a signature to be passed into a verify transaction method but for some reason it always fails even though the logic seems to be right when I compare it to the documentation. If anyone could point me where I'm going wrong it would be much appreciated. I have 3 methods that handle all of this and i'm not sure where the issue is
The generate keys method
def generate_keys(self):
# generate private key pair
private_key = RSA.generate(1024, Crypto.Random.new().read)
# public key comes as part of private key generation
public_key = private_key.publickey()
# return keys as hexidecimal representation of binary data
return (binascii.hexlify(public_key.exportKey(format='DER')).decode('ascii'), binascii.hexlify(private_key.exportKey(format='DER')).decode('ascii'))
The sign transaction method
def sign_transaction(self, sender, recipient, amount, key):
# convert transaction data to SHA256 string
hash_signer = SHA256.new(
(str(sender) + str(recipient) + str(amount)).encode('utf-8'))
# sign transaction
signature = pkcs1_15.new(RSA.importKey(
binascii.unhexlify(key))).sign(hash_signer)
# return hexidecimal representation of signature
return binascii.hexlify(signature).decode('ascii')
and the verify transaction method
#staticmethod
def verify_transaction(transaction):
# convert public key back to binary representation
public_key = RSA.importKey(binascii.unhexlify(
transaction.sender))
try:
# create signature from transaction data
hash_signer = SHA256.new(
(str(transaction.sender) + str(transaction.recipient) + str(transaction.amount)).encode('utf-8'))
pkcs1_15.new(public_key).verify(
hash_signer, binascii.unhexlify(transaction.signature))
return True
except ValueError:
return False
Once i've generated my key pair and attempt to use them to sign and verify transactions it always fails. I know this because it always returns false from the verify method leading me to believe a value error is always raised. Thanks in advance hopefully someone can help me out.
I found pretty cool ASN API tool that allows me to supply an AS # and it will go out and pull down the subnets that relate with that ASN.
Here is (rough) but partial code. I am defining a function ASNNUMBER (to which I will supply the number through another file)
When I call url here, it just gives me an n...
What I'm trying to do here, is append my str(ASNNUMBER) to the end of the ?q= parameter in the URL.
Once I do that, I'd like to display my results and output it to a file
import requests
def asnfinder(ASNNUMBER):
print('n\n######## Running ASNFinder ########\n')
url = 'https://api.hackertarget.com/aslookup?q=' + str(ASNNUMBER)
response = requests.get(url)
My results I'd like to get is an output of the get request I'm performing
## Running ASNFinder
n
Try to write something like that:
import requests
def asnfinder(ASNNUMBER):
print('n\n######## Running ASNFinder ########\n')
url = 'https://api.hackertarget.com/aslookup?q=' + str(ASNNUMBER)
response = requests.get(url)
data = response.text
print(data)
with open('filename', 'r') as f:
f.write(data)
It must works fine
P.S. If it helped ya, please make sure you mark this as the answer :)
I have the following question, I want to set up a routine to perform iterations inside a dataframe (pandas) to extract longitude and latitude data, after supplying the address using the 'geopy' library.
The routine I created was:
import time
from geopy.geocoders import GoogleV3
import os
arquivo = pd.ExcelFile('path')
df = arquivo.parse("Table1")
def set_proxy():
proxy_addr = 'http://{user}:{passwd}#{address}:{port}'.format(
user='usuario', passwd='senha',
address='IP', port=int('PORTA'))
os.environ['http_proxy'] = proxy_addr
os.environ['https_proxy'] = proxy_addr
def unset_proxy():
os.environ.pop('http_proxy')
os.environ.pop('https_proxy')
set_proxy()
geo_keys = ['AIzaSyBXkATWIrQyNX6T-VRa2gRmC9dJRoqzss0'] # API Google
geolocator = GoogleV3(api_key=geo_keys )
for index, row in df.iterrows():
location = geolocator.geocode(row['NO_LOGRADOURO'])
time.sleep(2)
lat=location.latitude
lon=location.longitude
timeout=10)
address = location.address
unset_proxy()
print(str(lat) + ', ' + str(lon))
The problem I'm having is that when I run the code the following error is thrown:
GeocoderQueryError: Your request was denied.
I tried the creation without passing the key to the google API, however, I get the following message.
KeyError: 'http_proxy'
and if I remove the unset_proxy () statement from within the for, the message I receive is:
GeocoderQuotaExceeded: The given key has gone over the requests limit in the 24 hour period or has submitted too many requests in too short a period of time.
But I only made 5 requests today, and I'm putting a 2-second sleep between requests. Should the period be longer?
Any idea?
api_key argument of the GoogleV3 class must be a string, not a list of strings (that's the cause of your first issue).
geopy doesn't guarantee the http_proxy/https_proxy env vars to be respected (especially the runtime modifications of the os.environ). The advised (by docs) usage of proxies is:
geolocator = GoogleV3(proxies={'http': proxy_addr, 'https': proxy_addr})
PS: Please don't ever post your API keys to the public. I suggest to revoke the key you've posted in the question and generate a new one, to prevent the possibility of it being abused by someone else.
I was trying to download a .csv file from this url for the history of a stock. Here's my code:
import requests
r = requests.get("https://query1.finance.yahoo.com/v7/finance/download/CHOLAFIN.BO?period1=1514562437&period2=1517240837&interval=1d&events=history&crumb=JaCfCutLNr7")
file = open(r"history_of_stock.csv", 'w')
file.write(r.text)
file.close()
But when I opened the file history_of_stock.csv, this was what I found: {
"finance": {
"error": {
"code": "Unauthorized",
"description": "Invalid cookie"
}
}
}
I couldn't find anything that could fix my problem. I found this thread in which someone has the same problem except that it is in C#: C# Download price data csv file from https instead of http
To complement the earlier answer and provide a concrete completed code, I wrote a script which accomplishes the task of getting historical stock prices in Yahoo Finance. Tried to write it as simply as possible. To give a summary: when you use requests to get a URL, in many instances you don't need to worry about crumbs or cookies. However, with Yahoo finance, you need to get the crumbs and the cookies. Once you get the cookies, then you are good to go! Make sure to set a timeout on the requests.get call.
import re
import requests
import sys
from pdb import set_trace as pb
symbol = sys.argv[-1]
start_date = '1442203200' # start date timestamp
end_date = '1531800000' # end date timestamp
crumble_link = 'https://finance.yahoo.com/quote/{0}/history?p={0}'
crumble_regex = r'CrumbStore":{"crumb":"(.*?)"}'
cookie_regex = r'set-cookie: (.*?);'
quote_link = 'https://query1.finance.yahoo.com/v7/finance/download/{}?period1={}&period2={}&interval=1d&events=history&crumb={}'
link = crumble_link.format(symbol)
session = requests.Session()
response = session.get(link)
# get crumbs
text = str(response.content)
match = re.search(crumble_regex, text)
crumbs = match.group(1)
# get cookie
cookie = session.cookies.get_dict()
url = "https://query1.finance.yahoo.com/v7/finance/download/%s?period1=%s&period2=%s&interval=1d&events=history&crumb=%s" % (symbol, start_date, end_date, crumbs)
r = requests.get(url,cookies=session.cookies.get_dict(),timeout=5, stream=True)
out = r.text
filename = '{}.csv'.format(symbol)
with open(filename,'w') as f:
f.write(out)
There was a service for exactly this but it was discontinued.
Now you can do what you intend but first you need to get a Cookie. On this post there is an example of how to do it.
Basically, first you need to make a useless request to get the Cookie and later, with this Cookie in place, you can query whatever else you actually need.
There's also a post about another service which might make your life easier.
There's also a Python module to work around this inconvenience and code to show how to do it without it.