I tried below code based on the coinbase documentaion coinbase doc
The documentation is given for Python2 but i have modified and used it for Python3 because i am trying to connect to advanced trade API in Coinbase Coinbase Advanced trade doc
import datetime
import time
import hmac
import hashlib
import http.client
secret_key='***' #hidden
api_key='***' #hidden
date_time = datetime.datetime.utcnow()
timestamp=int(time.mktime(date_time.timetuple())) # timestamp should be from UTC time and no decimal allowed
method = "GET" # method can be GET or POST. Only capital is allowed
request_path = 'api/v3/brokerage/accounts'
body=''
message= str(timestamp) + method + request_path + body
signature = hmac.new(secret_key.encode('utf-8'), message.encode('utf-8'), hashlib.sha256).hexdigest()
headers={
'accept':'application/json',
'CB-ACCESS-KEY': api_key,
'CB-ACCESS-TIMESTAMP': timestamp,
'CB-ACCESS-SIGN': signature
}
conn = http.client.HTTPSConnection("api.coinbase.com")
payload = ''
conn.request("GET", "/api/v3/brokerage/accounts", payload, headers)
res = conn.getresponse()
data = res.read()
print(data.decode("utf-8"))
When executing this code i was expecting account details. But i am getting unauthoried error and error code 401 as return from API.
I was able to connect to Coinbase Pro API earlier and everything was fine till the merger of coinbase and Coinbase Pro.Now unable to figure out how to connect to the Advanced trade feature in coinbase.
OK, so I made two updates to get this running.
Insert a / before api in request_path = '/api/v3/brokerage/accounts' :)
Change the way you generate your timestamp to timestamp = str(int(time.time())).
I thought updating your timestamp to a string would fix it, but it didn't, so I reverted to the way I was generating it. Maybe someone can tell us why one works and one doesn't. I'll certainly update this post if I figure it out.
Here's the full working code. I kept everything else the same but replaced your comments with mine.
import datetime
import time
import hmac
import hashlib
import http.client
secret_key = '***'
api_key = '***'
# This is the first part where you were getting time
#date_time = datetime.datetime.utcnow()
# And this is the second part where you format it as an integer
#timestamp=int(time.mktime(date_time.timetuple()))
# I cast your timestamp as a string, but it still doesn't work, and I'm having a hard time figuring out why.
#timestamp=str(int(time.mktime(date_time.timetuple())))
# So I reverted to the way that I'm getting the timestamp, and it works
timestamp = str(int(time.time()))
method = "GET"
request_path = '/api/v3/brokerage/accounts' # Added a forward slash before 'api'
body=''
message= str(timestamp) + method + request_path + body
signature = hmac.new(secret_key.encode('utf-8'), message.encode('utf-8'), hashlib.sha256).hexdigest()
headers={
'accept':'application/json',
'CB-ACCESS-KEY': api_key,
'CB-ACCESS-TIMESTAMP': timestamp,
'CB-ACCESS-SIGN': signature
}
conn = http.client.HTTPSConnection("api.coinbase.com")
payload = ''
conn.request("GET", "/api/v3/brokerage/accounts", payload, headers)
# You were probably troubleshooting, but the above line is redundant and can be simplified to:
# conn.request("GET", request_path, body, headers)
res = conn.getresponse()
data = res.read()
print(data.decode("utf-8"))
I see someone open-sourced a python library for the Advanced Trade endpoints: https://github.com/KmiQ/coinbase-advanced-python
I am trying to debug random ConnectTimeout happening in our infrastructure.
The symptom: Every now and then we receive a ConnectTimeout.
What I have tested:
Change host file: We changed the host file to route to the public IP, to avoid DNS lookups which may cause similar behaviour. This did not provide any significant benefit.
Change client:
Using aiohttp.ClientSession() - We get that occasionally, instead of a ConnectTimeout that the connection never terminates and remains open forever.
async with aiohttp.ClientSession() as session:
async with session.request(method, url, json=data, headers=headers, timeout=timeout, raise_for_status=False) as resp:
if resp.content_type == "application/json":
data = await resp.json()
else:
data = await resp.text()
if resp.ok:
return data
if isinstance(data, dict) and "detail" in data:
raise RawTextError(data["detail"])
resp.raise_for_status()
Replacing the client with httpx.AsyncClient, I get a ConnectTimeout sometimes and increasing connect timeout removes the connectTimeouts received. Sample Code
async with httpx.AsyncClient(http2=True, timeout=30.0) as client:
result = await client.post(cmdurl, data=json.dumps(wf_event["stages"], cls=JSONEncoder))
res = result.read()
Running the following script without timeouts (Default Httpx timeout is 5 s) I can reproduce the behaviour from multiple servers and locations
import httpx
import asyncio
import datetime
import sys, traceback
async def testurl():
cmdurl = "url
headers = {"authorization" : "token"}
while True:
try:
t0 = datetime.datetime.now()
async with httpx.AsyncClient(http2=True) as client:
#print("Fetching from URL: {}".format(cmdurl))
result = await client.get(cmdurl,headers=headers)
res = result.json()
#print(res)
except Exception as e:
print("Start : ", t0)
print("End : ", datetime.datetime.now())
print(e)
traceback.print_exc(file=sys.stdout)
loop = asyncio.get_event_loop()
loop.run_until_complete(testurl())
Upgrade NGINX: The original Nginx is version 1.18. We created a separate Nginx on 1.21, we created a new URL for the new Nginx and could replicate the ConnectTimeouts with the above httpx script on the new endpoint, indicating that NGINX version is fine and that the load on Nginx is probably not the issue.
Regarding aiohttp.web.Application: Here we have 14 load-balancing dockers in play. I couldn't find anything suggesting a max connection count could be the issue here. I am also not sure if the upstream aiohttp is the issue as I don't see any related issues in the nginx error logs for the corresponding ConnectTimeout.
So while using the Httpx with timeouts almost completely resolved the symptom I still have no Idee why a ConnectTimeout would occur and why it sometimes takes longer than 5 seconds to connect. I haven't been able to reproduce this locally, but then our live service does handle 5000 concurrent connections at any given time.
Hope someone can point me in the direction of where to look.
Thanks for all the help in advance.
How do we post a GraphQL request through AWS AppSync using boto?
Ultimately I'm trying to mimic a mobile app accessing our stackless/cloudformation stack on AWS, but with python. Not javascript or amplify.
The primary pain point is authentication; I've tried a dozen different ways already. This the current one, which generates a "401" response with "UnauthorizedException" and "Permission denied", which is actually pretty good considering some of the other messages I've had. I'm now using the 'aws_requests_auth' library to do the signing part. I assume it authenticates me using the stored /.aws/credentials from my local environment, or does it?
I'm a little confused as to where and how cognito identities and pools will come into it. eg: say I wanted to mimic the sign-up sequence?
Anyways the code looks pretty straightforward; I just don't grok the authentication.
from aws_requests_auth.boto_utils import BotoAWSRequestsAuth
APPSYNC_API_KEY = 'inAppsyncSettings'
APPSYNC_API_ENDPOINT_URL = 'https://aaaaaaaaaaaavzbke.appsync-api.ap-southeast-2.amazonaws.com/graphql'
headers = {
'Content-Type': "application/graphql",
'x-api-key': APPSYNC_API_KEY,
'cache-control': "no-cache",
}
query = """{
GetUserSettingsByEmail(email: "john#washere"){
items {name, identity_id, invite_code}
}
}"""
def test_stuff():
# Use the library to generate auth headers.
auth = BotoAWSRequestsAuth(
aws_host='aaaaaaaaaaaavzbke.appsync-api.ap-southeast-2.amazonaws.com',
aws_region='ap-southeast-2',
aws_service='appsync')
# Create an http graphql request.
response = requests.post(
APPSYNC_API_ENDPOINT_URL,
json={'query': query},
auth=auth,
headers=headers)
print(response)
# this didn't work:
# response = requests.post(APPSYNC_API_ENDPOINT_URL, data=json.dumps({'query': query}), auth=auth, headers=headers)
Yields
{
"errors" : [ {
"errorType" : "UnauthorizedException",
"message" : "Permission denied"
} ]
}
It's quite simple--once you know. There are some things I didn't appreciate:
I've assumed IAM authentication (OpenID appended way below)
There are a number of ways for appsync to handle authentication. We're using IAM so that's what I need to deal with, yours might be different.
Boto doesn't come into it.
We want to issue a request like any regular punter, they don't use boto, and neither do we. Trawling the AWS boto docs was a waste of time.
Use the AWS4Auth library
We are going to send a regular http request to aws, so whilst we can use python requests they need to be authenticated--by attaching headers.
And, of course, AWS auth headers are special and different from all others.
You can try to work out how to do it
yourself, or you can go looking for someone else who has already done it: Aws_requests_auth, the one I started with, probably works just fine, but I have ended up with AWS4Auth. There are many others of dubious value; none endorsed or provided by Amazon (that I could find).
Specify appsync as the "service"
What service are we calling? I didn't find any examples of anyone doing this anywhere. All the examples are trivial S3 or EC2 or even EB which left uncertainty. Should we be talking to api-gateway service? Whatsmore, you feed this detail into the AWS4Auth routine, or authentication data. Obviously, in hindsight, the request is hitting Appsync, so it will be authenticated by Appsync, so specify "appsync" as the service when putting together the auth headers.
It comes together as:
import requests
from requests_aws4auth import AWS4Auth
# Use AWS4Auth to sign a requests session
session = requests.Session()
session.auth = AWS4Auth(
# An AWS 'ACCESS KEY' associated with an IAM user.
'AKxxxxxxxxxxxxxxx2A',
# The 'secret' that goes with the above access key.
'kwWxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxgEm',
# The region you want to access.
'ap-southeast-2',
# The service you want to access.
'appsync'
)
# As found in AWS Appsync under Settings for your endpoint.
APPSYNC_API_ENDPOINT_URL = 'https://nqxxxxxxxxxxxxxxxxxxxke'
'.appsync-api.ap-southeast-2.amazonaws.com/graphql'
# Use JSON format string for the query. It does not need reformatting.
query = """
query foo {
GetUserSettings (
identity_id: "ap-southeast-2:8xxxxxxb-7xx4-4xx4-8xx0-exxxxxxx2"
){
user_name, email, whatever
}}"""
# Now we can simply post the request...
response = session.request(
url=APPSYNC_API_ENDPOINT_URL,
method='POST',
json={'query': query}
)
print(response.text)
Which yields
# Your answer comes as a JSON formatted string in the text attribute, under data.
{"data":{"GetUserSettings":{"user_name":"0xxxxxxx3-9102-42f0-9874-1xxxxx7dxxx5"}}}
Getting credentials
To get rid of the hardcoded key/secret you can consume the local AWS ~/.aws/config and ~/.aws/credentials, and it is done this way...
# Use AWS4Auth to sign a requests session
session = requests.Session()
credentials = boto3.session.Session().get_credentials()
session.auth = AWS4Auth(
credentials.access_key,
credentials.secret_key,
boto3.session.Session().region_name,
'appsync',
session_token=credentials.token
)
...<as above>
This does seem to respect the environment variable AWS_PROFILE for assuming different roles.
Note that STS.get_session_token is not the way to do it, as it may try to assume a role from a role, depending where it keyword matched the AWS_PROFILE value. Labels in the credentials file will work because the keys are right there, but names found in the config file do not work, as that assumes a role already.
OpenID
In this scenario, all the complexity is transferred to the conversation with the openid connect provider. The hard stuff is all the auth hoops you jump through to get an access token, and thence using the refresh token to keep it alive. That is where all the real work lies.
Once you finally have an access token, assuming you have configured the "OpenID Connect" Authorization Mode in appsync, then you can, very simply, drop the access token into the header:
response = requests.post(
url="https://nc3xxxxxxxxxx123456zwjka.appsync-api.ap-southeast-2.amazonaws.com/graphql",
headers={"Authorization": ACCESS_TOKEN},
json={'query': "query foo{GetStuff{cat, dog, tree}}"}
)
You can set up an API key on the AppSync end and use the code below. This works for my case.
import requests
# establish a session with requests session
session = requests.Session()
# As found in AWS Appsync under Settings for your endpoint.
APPSYNC_API_ENDPOINT_URL = 'https://vxxxxxxxxxxxxxxxxxxy.appsync-api.ap-southeast-2.amazonaws.com/graphql'
# setup the query string (optional)
query = """query listItemsQuery {listItemsQuery {items {correlation_id, id, etc}}}"""
# Now we can simply post the request...
response = session.request(
url=APPSYNC_API_ENDPOINT_URL,
method='POST',
headers={'x-api-key': '<APIKEYFOUNDINAPPSYNCSETTINGS>'},
json={'query': query}
)
print(response.json()['data'])
Building off Joseph Warda's answer you can use the class below to send AppSync commands.
# fileName: AppSyncLibrary
import requests
class AppSync():
def __init__(self,data):
endpoint = data["endpoint"]
self.APPSYNC_API_ENDPOINT_URL = endpoint
self.api_key = data["api_key"]
self.session = requests.Session()
def graphql_operation(self,query,input_params):
response = self.session.request(
url=self.APPSYNC_API_ENDPOINT_URL,
method='POST',
headers={'x-api-key': self.api_key},
json={'query': query,'variables':{"input":input_params}}
)
return response.json()
For example in another file within the same directory:
from AppSyncLibrary import AppSync
APPSYNC_API_ENDPOINT_URL = {YOUR_APPSYNC_API_ENDPOINT}
APPSYNC_API_KEY = {YOUR_API_KEY}
init_params = {"endpoint":APPSYNC_API_ENDPOINT_URL,"api_key":APPSYNC_API_KEY}
app_sync = AppSync(init_params)
mutation = """mutation CreatePost($input: CreatePostInput!) {
createPost(input: $input) {
id
content
}
}
"""
input_params = {
"content":"My first post"
}
response = app_sync.graphql_operation(mutation,input_params)
print(response)
Note: This requires you to activate API access for your AppSync API. Check this AWS post for more details.
graphql-python/gql supports AWS AppSync since version 3.0.0rc0.
It supports queries, mutation and even subscriptions on the realtime endpoint.
The documentation is available here
Here is an example of a mutation using the API Key authentication:
import asyncio
import os
import sys
from urllib.parse import urlparse
from gql import Client, gql
from gql.transport.aiohttp import AIOHTTPTransport
from gql.transport.appsync_auth import AppSyncApiKeyAuthentication
# Uncomment the following lines to enable debug output
# import logging
# logging.basicConfig(level=logging.DEBUG)
async def main():
# Should look like:
# https://XXXXXXXXXXXXXXXXXXXXXXXXXX.appsync-api.REGION.amazonaws.com/graphql
url = os.environ.get("AWS_GRAPHQL_API_ENDPOINT")
api_key = os.environ.get("AWS_GRAPHQL_API_KEY")
if url is None or api_key is None:
print("Missing environment variables")
sys.exit()
# Extract host from url
host = str(urlparse(url).netloc)
auth = AppSyncApiKeyAuthentication(host=host, api_key=api_key)
transport = AIOHTTPTransport(url=url, auth=auth)
async with Client(
transport=transport, fetch_schema_from_transport=False,
) as session:
query = gql(
"""
mutation createMessage($message: String!) {
createMessage(input: {message: $message}) {
id
message
createdAt
}
}"""
)
variable_values = {"message": "Hello world!"}
result = await session.execute(query, variable_values=variable_values)
print(result)
asyncio.run(main())
I am unable to add a comment due to low rep, but I just want to add that I tried the accepted answer and it didn't work. I was getting an error saying my session_token is invalid. Probably because I was using AWS Lambda.
I got it to work pretty much exactly, but by adding to the session token parameter of the aws4auth object. Here's the full piece:
import requests
import os
from requests_aws4auth import AWS4Auth
def AppsyncHandler(event, context):
# These are env vars that are always present in an AWS Lambda function
# If not using AWS Lambda, you'll need to add them manually to your env.
access_id = os.environ.get("AWS_ACCESS_KEY_ID")
secret_key = os.environ.get("AWS_SECRET_ACCESS_KEY")
session_token = os.environ.get("AWS_SESSION_TOKEN")
region = os.environ.get("AWS_REGION")
# Your AppSync Endpoint
api_endpoint = os.environ.get("AppsyncConnectionString")
resource = "appsync"
session = requests.Session()
session.auth = AWS4Auth(access_id,
secret_key,
region,
resource,
session_token=session_token)
The rest is the same.
Hope this Helps Everyone
import requests
import json
import os
from dotenv import load_dotenv
load_dotenv(".env")
class AppSync(object):
def __init__(self,data):
endpoint = data["endpoint"]
self.APPSYNC_API_ENDPOINT_URL = endpoint
self.api_key = data["api_key"]
self.session = requests.Session()
def graphql_operation(self,query,input_params):
response = self.session.request(
url=self.APPSYNC_API_ENDPOINT_URL,
method='POST',
headers={'x-api-key': self.api_key},
json={'query': query,'variables':{"input":input_params}}
)
return response.json()
def main():
APPSYNC_API_ENDPOINT_URL = os.getenv("APPSYNC_API_ENDPOINT_URL")
APPSYNC_API_KEY = os.getenv("APPSYNC_API_KEY")
init_params = {"endpoint":APPSYNC_API_ENDPOINT_URL,"api_key":APPSYNC_API_KEY}
app_sync = AppSync(init_params)
mutation = """
query MyQuery {
getAccountId(id: "5ca4bbc7a2dd94ee58162393") {
_id
account_id
limit
products
}
}
"""
input_params = {}
response = app_sync.graphql_operation(mutation,input_params)
print(json.dumps(response , indent=3))
main()
I am trying to use requests in a Python3 script to update a deploy key in a gitlab project with write access. Unfortunately, I am receiving a 404 error when trying to connect via the requests module. The code is below:
project_url = str(url)+('/deploy_keys/')+str(DEPLOY_KEY_ID)
headers = {"PRIVATE-TOKEN" : "REDACTED"}
payload = {"can_push" : "true"}
r = requests.put(project_url, headers=headers, json=payload)
print(r)
Is there something that I am doing wrong where in the syntax of my Private Key/headers?
I have gone through gitlab api and requests documentation. I have also confirmed that my Private Token is working outside of the script.
I am expecting it to update the deploy key with write access, but am receiving a upon exit, making me think the issue is with the headers/auth.
This is resolved, it was actually not an auth problem. You must use the project ID instead of url, for example:
project_url = f'https://git.REDACTED.com/api/v4/projects/{project_id}/deploy_keys/{DEPLOY_KEY_ID}'
headers = {"PRIVATE-TOKEN" : "REDACTED"}
payload = {'can_push' : 'true'}
try:
r = requests.put(project_url, headers=headers, data=payload)
So i have a command where it sends whatever the user said after the command to a website api and sends the file the site generates. However i'm changing over to aiohttp as it doesn't block like the standered requests functions
This is how i do it with normal requests and it works fine:
elif (data[0].lower() == ">signgirl"):
await bot.send_typing(message.channel)
tmp = message.content.replace(">signgirl", "")
m = hashlib.md5()
m.update(tmp.encode('utf-8'))
print(tmp, m.hexdigest())
r = requests.post("http://localhost/sign.php", stream=True, data={'text': tmp})
if (r.status_code() == 200):
await bot.send_file(destination=message.channel, filename=str(m.hexdigest()+".png"), fp=r.raw)
However when i try with aiohttp i have no idea how to actually get the raw file data..
So i made this function to get it. but it doesn't let me return an image and i cannot check the http status code without it causing an error.
async def post_data2(url, payload):
async with aiohttp.ClientSession() as session2:
async with session2.post(url, data=payload) as response2:
output = {}
output['data'] = await Image.open(BytesIO(response2.read()))
output['status'] = 200 #await str(response2.status()) #Why is this object not callable?
return output
How else could i do this? Is this possible? aiohttp doesn't seem as easy to understand.
Mister Day "V" Own from the discord.py discord server sent a perfect example of getting and sending the data
async with aiohttp.ClientSession() as session:
# or use a session you already have
async with session.get("http://example.com") as resp:
buffer = io.BytesIO(await resp.read())
# buffer is a file-like
await client.send_file(channel, fp=buffer, filename="whatever")