I tried below code based on the coinbase documentaion coinbase doc
The documentation is given for Python2 but i have modified and used it for Python3 because i am trying to connect to advanced trade API in Coinbase Coinbase Advanced trade doc
import datetime
import time
import hmac
import hashlib
import http.client
secret_key='***' #hidden
api_key='***' #hidden
date_time = datetime.datetime.utcnow()
timestamp=int(time.mktime(date_time.timetuple())) # timestamp should be from UTC time and no decimal allowed
method = "GET" # method can be GET or POST. Only capital is allowed
request_path = 'api/v3/brokerage/accounts'
body=''
message= str(timestamp) + method + request_path + body
signature = hmac.new(secret_key.encode('utf-8'), message.encode('utf-8'), hashlib.sha256).hexdigest()
headers={
'accept':'application/json',
'CB-ACCESS-KEY': api_key,
'CB-ACCESS-TIMESTAMP': timestamp,
'CB-ACCESS-SIGN': signature
}
conn = http.client.HTTPSConnection("api.coinbase.com")
payload = ''
conn.request("GET", "/api/v3/brokerage/accounts", payload, headers)
res = conn.getresponse()
data = res.read()
print(data.decode("utf-8"))
When executing this code i was expecting account details. But i am getting unauthoried error and error code 401 as return from API.
I was able to connect to Coinbase Pro API earlier and everything was fine till the merger of coinbase and Coinbase Pro.Now unable to figure out how to connect to the Advanced trade feature in coinbase.
OK, so I made two updates to get this running.
Insert a / before api in request_path = '/api/v3/brokerage/accounts' :)
Change the way you generate your timestamp to timestamp = str(int(time.time())).
I thought updating your timestamp to a string would fix it, but it didn't, so I reverted to the way I was generating it. Maybe someone can tell us why one works and one doesn't. I'll certainly update this post if I figure it out.
Here's the full working code. I kept everything else the same but replaced your comments with mine.
import datetime
import time
import hmac
import hashlib
import http.client
secret_key = '***'
api_key = '***'
# This is the first part where you were getting time
#date_time = datetime.datetime.utcnow()
# And this is the second part where you format it as an integer
#timestamp=int(time.mktime(date_time.timetuple()))
# I cast your timestamp as a string, but it still doesn't work, and I'm having a hard time figuring out why.
#timestamp=str(int(time.mktime(date_time.timetuple())))
# So I reverted to the way that I'm getting the timestamp, and it works
timestamp = str(int(time.time()))
method = "GET"
request_path = '/api/v3/brokerage/accounts' # Added a forward slash before 'api'
body=''
message= str(timestamp) + method + request_path + body
signature = hmac.new(secret_key.encode('utf-8'), message.encode('utf-8'), hashlib.sha256).hexdigest()
headers={
'accept':'application/json',
'CB-ACCESS-KEY': api_key,
'CB-ACCESS-TIMESTAMP': timestamp,
'CB-ACCESS-SIGN': signature
}
conn = http.client.HTTPSConnection("api.coinbase.com")
payload = ''
conn.request("GET", "/api/v3/brokerage/accounts", payload, headers)
# You were probably troubleshooting, but the above line is redundant and can be simplified to:
# conn.request("GET", request_path, body, headers)
res = conn.getresponse()
data = res.read()
print(data.decode("utf-8"))
I see someone open-sourced a python library for the Advanced Trade endpoints: https://github.com/KmiQ/coinbase-advanced-python
I've been using KuCoin API with python for a while now, but stuck trying to get advanced orders form the API. For reference, 'advanced orders' are what stop limit orders are called within KuCoin dash.
I've used the endpoint: api/v1/orders
. I've experimented with this but it only shows 'normal' orders or the order history if 'status' is set to 'done'. Ref: https://docs.kucoin.com/#list-orders
I'm wondering if instead I should use: /api/v1/limit/orders. This endpoint has an option to return stop_limit orders. But when I try, I get response:
{"code":"400007","msg":"Access denied, require more permission."}'
I checked my API permissions and its 'General' permission is set to 'read only' which is the only option. Read only would be fine for me because I only want to get the list of all 'Advanced orders'
See https://docs.kucoin.com/#recent-orders
Does anyone know how to get all open limit orders please? The reason I want this is I have placed many limit orders. I want to collect the orderid's of all stop limit orders so I can clean out just the ones not associated to an active trade.
my code is as follows:
import requests
import json
import hmac
import hashlib
import base64
from urllib.parse import urlencode
import time
from urllib.request import urlretrieve
'''working code to get active orders'''
api_key = 'api_key'
api_secret = 'api_secret'
api_passphrase = 'api_passphrase' # this is NOT trading password
base_uri = 'https://api.kucoin.com'
def get_headers(method, endpoint):
now = int(time.time() * 1000)
str_to_sign = str(now) + method + endpoint
signature = base64.b64encode(hmac.new(api_secret.encode(), str_to_sign.encode(), hashlib.sha256).digest()).decode()
passphrase = base64.b64encode(hmac.new(api_secret.encode(), api_passphrase.encode(), hashlib.sha256).digest()).decode()
return {'KC-API-KEY': api_key,
'KC-API-KEY-VERSION': '2',
'KC-API-PASSPHRASE': passphrase,
'KC-API-SIGN': signature,
'KC-API-TIMESTAMP': str(now)
}
#List Orders
method = 'GET'
endpoint = '/api/v1/limit/orders/' # main
# params = {'type': 'stop_limit'}
params = {}
# # if params :
qstr = '?' + urlencode(params)if params else ''
active_orders = requests.get(url=base_uri+endpoint+qstr, headers=get_headers(method,endpoint+qstr))
I've seen many tutorials on how to deploy just a Discord Bot or just a basic Flask Web App, however, this project packages both together where the Bot relies completely on interfacing with a REST api.
Presentation Layer: Discord UI. Handles user command input and bot output
Application Layer: Discord handlers that interface with and formats output based on data received from requests (bot.py)
Business Logic Layer: Flask interface that handles data from arguments passed from Discord user input. Handles CRUD requests (app.py)
Data Access Layer: MongoDB database
I currently have it set up to run the app using the routine in this answer for threading: run discord bot inside flask
$ python3 app.py
I've tried uploading the project to a google VM which works but when I exit, it goes offline. I've also tried following a digital ocean tutorial on Dockerizing a Flask and MongoDB app to deploy it to the cloud but it throws errors like
File "/var/www/wsgi.py", line 1, in <module>
flask | from app import app
flask | File "/var/www/app.py", line 5
flask | from import client
flask | ^
flask | SyntaxError: invalid syntax
even though there is no such error...
db.py
from flask import Flask
import pymongo
from pymongo import MongoClient
from app import app
CONNECTION_STRING = "mongodb+srv://<user>:<password>#cluster0.43ypd.mongodb.net/book-bot?retryWrites=true&w=majority"
client = MongoClient(CONNECTION_STRING)
db = client["book-bot"]
books_collection = db["books"]
app.py
from flask import Flask, render_template, request
from flask_mail import *
from random import *
from json import dumps, loads
from bot import client
from threading import Thread
from functools import partial
import time
import os
import db
from book import Book, Seller
app = Flask(__name__)
app.config["MAIL_SERVER"]='smtp.gmail.com'
app.config["MAIL_PORT"] = 465
app.config["MAIL_USERNAME"] = 'email#gmail.com'
app.config['MAIL_PASSWORD'] = 'password'
app.config['MAIL_USE_TLS'] = False
app.config['MAIL_USE_SSL'] = True
mail = Mail(app)
otp = randint(000000,999999)
# localhost:8000/book?course=<course>&number=<course-number>
#app.route('/book/<department>/<course_num>', methods=['GET'])
def book(department, course_num):
query = {"department": department, "course": course_num}
document = db.db.books_collection.find(query, {"_id": 0})
for book in document:
return dumps(book)
return dumps(None)
#app.route('/book/insert/<dep>/<cnum>/<name>/<link>/<buy_price>/<rent_price>', methods=["POST"])
def insert_book(dep, cnum, name, link, buy_price, rent_price):
# Initialize book with the GMU bookstore seller information
seller = Seller("GMU Bookstore", link, float(buy_price), float(rent_price), "Fairfax Campus", True)
# Initialize seller and book data with request params and seller data
bookstore_dict = {"name": seller.name, "link": seller.link,
"buy": seller.buy,
"buy_price": seller.buy_price,
"rent": seller.rent,
"rent_price": seller.rent_price,
"location": seller.location,
"verified": seller.verified
}
book = Book(name, dep, cnum, bookstore_dict)
# create dict struct to insert proper format into Mongo collection
book_dict = {
"name": book.name,
"department": book.department,
"course": book.course_name,
"sellers": book.sellers
}
# Insert book into Database
db.db.books_collection.insert_one(book_dict)
return f"Book name: {book.name} | Department: {book.department} | Course: {book.course_name}"
# example: localhost:8000/book/insert/seller/Mostafa/mostaf#gmu.edu/true/false/Faifax
# "http://localhost:8000/book/insert/seller/{dep}/{cnum}/{name}/{link}/{buy_price}/{rent_price}/{location}
#app.route('/book/insert/seller/<dep>/<cnum>/<name>/<link>/<buy_price>/<rent_price>/<location>', methods=["POST"])
def insert_seller(dep, cnum, name, link, buy_price, rent_price, location, methods=["POST"]):
# temporary solution to interrupted POST request due to # tag in string passed into request
name_arg = name.split("#")
name = name_arg[0] + "#" + name_arg[1]
seller = Seller(name, link, float(buy_price), float(rent_price), location, False)
seller_dict = {
"name": seller.name,
"link": seller.link,
"buy": seller.buy,
"buy_price": seller.buy_price,
"rent": seller.rent,
"rent_price": seller.rent_price,
"location": seller.location,
"verified": seller.verified,
}
query = {"department": dep, "course": cnum}
book = db.db.books_collection.find(query)
for doc in book:
db.db.books_collection.update_one({"_id": doc["_id"]}, {"$push": {"sellers": seller_dict}})
return "Added seller to book"
#app.route('/verify/<email>/<dep>/<cnum>', methods=["POST"])
def verify(email, dep, cnum):
send_otp = f"{otp}-{dep}-{cnum}"
msg = Message('Verify Student Seller', sender='email#gmail.com', recipients=[email])
msg.body = f"Please click on the link to verify your student seller status\n {send_otp}\nhttp://localhost:8000/"
mail.send(msg)
return "Success"
#app.route('/', methods=["GET"])
def load_validation():
return render_template("email.html")
#app.route('/validate', methods=["POST"])
def validate():
user_email = request.form['email']
user_otp = request.form['otp']
args = user_otp.split('-')
# check if numbervalue is equal to otp generated
if int(args[0]) == otp:
query = {"department": args[1], "course": args[2]}
book = db.db.books_collection.find(query)
for doc in book:
db.db.books_collection.update_one(query, {"$set": {"sellers.$[t].verified": True}},
array_filters=[{"t.link": user_email}])
return "<h3>Your seller status has been activated.</h3>"
return "<h3>failure, OTP does not match</h3>"
def flask_thread(func):
thread = Thread(target=func)
print('Start Separate Thread From Bot')
thread.start()
def run():
app.run(host='0.0.0.0', port=8000, use_reloader=False)
if __name__ == '__main__':
flask_thread(func=run)
client.run('bot-token', bot=True)
bot.py
import discord
from discord.ext import commands
import os
import requests
import json
import asyncio
client = discord.Client()
# client = commands.Bot(command_prefix="!", case_insensitive=True)
def insert_seller(dep, cnum, link, buy_price, rent_price, location, name):
response = requests.post(f"http://localhost:8000/book/insert/seller/{dep}/{cnum}/{name}/{link}/{buy_price}/{rent_price}/{location}")
return response
def verify_student(email, dep, cnum):
if ('#gmu.edu' in email):
response = requests.post(f"http://localhost:8000/verify/{email}/{dep}/{cnum}")
return response
return "Please enter a valid email"
def get_book(department, course_num):
response = requests.get(f"http://localhost:8000/book/{department}/{course_num}")
if (response == None):
return None
json_data = json.loads(response.text)
return json_data
#client.event
async def on_ready():
print("We have logged in as {0.user}".format(client))
#client.event
async def on_message(message):
if (message.content.startswith("!bot help")):
msg = "To check for a course textbook enter:\n'!bot check <department> <course_num>\n\nTo add yourself as a seller enter:\n'!bot add <department> <course_num> <your_name> <gmu_email> <buy_price> <rent_price> <city>'"
await message.channel.send(msg)
if (message.content.startswith('!bot check')):
args = message.content.split()
book = get_book(args[2], args[3])
embed = discord.Embed(title=f"{book['department']} {book['course']} Textbook", description=f"\n{book['name']}\n", color=0xFFD700)
for i in book['sellers']:
# hyperlink the bookstore link because it's long
if i['buy']:
options = "buy"
if i['rent']:
options += "/rent"
if i['name'] == "GMU Bookstore":
embed.add_field(name=f"{i['name']}", value=f"[Bookstore link]({i['link']})\nOptions: {options}\nLocation: {i['location']}",inline=False)
else:
embed.add_field(name=f"{i['name']} ({i['link']})", value=f"Options: {options}\nLocation: {i['location']}",inline=False)
embed.set_author(name=message.author.display_name, icon_url=message.author.avatar_url)
embed.set_footer(text="Powered by students. This is not an official GMU service.")
await message.channel.send(embed=embed)
if (message.content.startswith('!bot add')):
args = message.content.split()
book = get_book(args[2], args[3])
if (book == None):
await message.channel.send("Course requirements not found.")
return
def check(msg):
return msg.author == message.author and 'Y' in msg.content
embed = discord.Embed(title=f"Is this your book (enter Y/N)?\n\n", description=f"\n{book['name']}\n", color=0xFFD700)
await message.channel.send(embed=embed)
try:
msg = await client.wait_for("message", check=check, timeout=30)
except asyncio.TimeoutError:
await message.channel.send("Sorry, you didn't reply in time.")
if (msg.content):
# name will save the seller with their discord user handle
name = f"{message.author}"
# Request will 404 because of '#' symbol
regs = name.split('#')
insert_seller(args[2], args[3], args[4], args[5], args[6], args[7], regs[0] + "#" + regs[1])
verify_student(args[4], args[2], args[3])
await message.channel.send("Sending verification email!")
else:
await message.channel.send("Sorry you can't add that.")
Hello jr data engineer here!
For some strange reason my task_fail_slack_alert module is triggering the Slack API request a ridiculous amount of times, which is then showing up in our Slack channel that many times and is really annoying. My module should only run and show up in in Slack channel the same amount as the number of tasks that failed.
What am I missing?
import os
from airflow.models
import Variable
import json import requests
def get_channel_name():
channel = '#airflow_alerts_local'
env = Variable.get('env', None)
if env == 'prod':
channel = '#airflow_alerts'
elif env == 'dev':
channel = '#airflow_alerts_dev'
return channel
def task_fail_slack_alert(context):
webhook_url = os.environ.get('SLACK_URL')
slack_data = {
'channel': get_channel_name(),
'text':
""" :red_circle: Task Failed.
*Task*: {task}
*Dag*: {dag}
*Execution Time*: {exec_date}
*Log Url*: {log_url}
""".format(
task=context.get('task_instance').task_id,
dag=context.get('task_instance').dag_id,
ti=context.get('task_instance'),
exec_date=context.get('execution_date'),
log_url=context.get('task_instance').log_url,
)}
response = requests.post(webhook_url, data=json.dumps(slack_data),
headers={'Content-Type': 'application/json'})
if response.status_code != 200:
raise ValueError( 'Request to slack returned an error %s,
the response is:\n%s'(response.status_code, response.text))
task_fail_slack_alert(context)
This is how I have it showing up in the arguments for each dag:
default_args = {
'on_failure_callback': task_fail_slack_alert,
}
The code you provided is recursive:
def task_fail_slack_alert(context):
......
task_fail_slack_alert(context)
Remove the recursion as it's not needed.
import json
import base64
from google.cloud import bigquery
import ast
import pandas as pd
import sys
import pandas_gbq
def process_data(data):
#msg = str(data)
df = pd.DataFrame({"Data":data},index=[0])
df['time'] = pd.datetime.now()
lst = list(df)
df[lst] = df[lst].astype(str)
pandas_gbq.to_gbq(df,'datasetid.tableid',project_id='project_id',if_exists='append')
def receive_messages(project_id, subscription_name):
"""Receives messages from a pull subscription."""
# [START pubsub_subscriber_async_pull]
# [START pubsub_quickstart_subscriber]
import time
from google.cloud import pubsub_v1
# TODO project_id = "Your Google Cloud Project ID"
# TODO subscription_name = "Your Pub/Sub subscription name"
subscriber = pubsub_v1.SubscriberClient()
# The `subscription_path` method creates a fully qualified identifier
# in the form `projects/{project_id}/subscriptions/{subscription_name}`
subscription_path = subscriber.subscription_path(
project_id, subscription_name)
def callback(message):
#print('Received message: {}'.format(message))
process_data(message)
message.ack()
subscriber.subscribe(subscription_path, callback=callback)
# The subscriber is non-blocking. We must keep the main thread from
# exiting to allow it to process messages asynchronously in the background.
# print('Listening for messages on {}'.format(subscription_path))
while True:
time.sleep(60)
# [END pubsub_subscriber_async_pull]
# [END pubsub_quickstart_subscriber]
receive_messages(project-id,sub-id)
I'm streaming the real time data from Pub/Sub to bigquery using cloud functions.
Here the following error:
Deployment failure:
Function failed on loading user code. Error message: Error: function load attempt timed out.
Your code is in a while True loop. Cloud Functions considers that your code has crashed because it does not return. Then your function is killed.
Redesign so that Pub/Sub is calling your Cloud Function using events (triggers). Follow this guide on how to implement a correct design:
Google Cloud Pub/Sub Triggers