why would python3 recursive function return null - python-3.x

I have this function that when hitting a rate limit will call itself again. It should eventually succeed and return the working data. It works normally then rate limiting works as expected, and finally when the data goes back to normal I get:
TypeError: 'NoneType' object is not subscriptable
def grabPks(pageNum):
# cloudflare blocks bots...use scraper library to get around this or build your own logic to store and use a manually generated cloudflare session cookie... I don't care 😎
req = scraper.get("sumurl.com/"+str(pageNum)).content
if(req == b'Rate Limit Exceeded'):
print("adjust the rate limiting because they're blocking us :(")
manPenalty = napLength * 3
print("manually sleeping for {} seconds".format(manPenalty))
time.sleep(manPenalty)
print("okay let's try again... NOW SERVING {}".format(pageNum))
grabPks(pageNum)
else:
tree = html.fromstring(req)
pk = tree.xpath("/path/small/text()")
resCmpress = tree.xpath("path/a//text()")
resXtend = tree.xpath("[path/td[2]/small/a//text()")
balance = tree.xpath("path/font//text()")
return pk, resCmpress, resXtend, balance
I've tried to move the return to outside of the else scope but then it throws:
UnboundLocalError: local variable 'pk' referenced before assignment

Your top level grabPks doesnt return anything if it is rate limited.
Think about this:
Call grabPks()
You're rate limited so you go into the if statement and call grabPks() again.
This time it succeeds so grabPks() returns the value to the function above it.
The first function now falls out of the if statement, gets to the end of the function and returns nothing.
Try return grabPks(pageNum) instead inside your if block.

well okay... I needed to return grabPKs to make it play nice...:
def grabPks(pageNum):
# cloudflare blocks bots...use scraper library to get around this or build your own logic to store and use a manually generated cloudflare session cookie... I don't care 😎
req = scraper.get("sumurl.com/"+str(pageNum)).content
if(req == b'Rate Limit Exceeded'):
print("adjust the rate limiting because they're blocking us :(")
manPenalty = napLength * 3
print("manually sleeping for {} seconds".format(manPenalty))
time.sleep(manPenalty)
print("okay let's try again... NOW SERVING {}".format(pageNum))
return grabPks(pageNum)
else:
tree = html.fromstring(req)
pk = tree.xpath("/path/small/text()")
resCmpress = tree.xpath("path/a//text()")
resXtend = tree.xpath("[path/td[2]/small/a//text()")
balance = tree.xpath("path/font//text()")
return pk, resCmpress, resXtend, balance

Related

Why does transact and wait behave differently when within a function is behaving different?

Within my project I intend to send large volumes of transactions therefore for simplicity I am building a wrapper function for the following functions to be executed together: contractName.functions.functionName(params).transact() and w3.eth.wait_for_transaction(tx_hash). However when I write the functions transact_and_wait with the above implemented within in the transactions do not get executed!
Implementation of Transact and wait
def transact_and_wait(contract_function, transaction_params= {"gas": 100000}):
# Send the transaction
if transaction_params != {"gas": 100000}:
transaction_params["gas"] = 100000
transaction_hash = contract_function.transact(transaction_params)
# Wait for the transaction to be mined
transaction_receipt = w3.eth.wait_for_transaction_receipt(transaction_hash)
return transaction_receipt
Where it is called via: Transact_and_wait(contractName.functions.functionName(account.address))
For example this should set the a role for a user defined via index 1
However when I call. print(contractName.functions.stateVariable(account.address).call()) it returns 0
If i do the same process above but not within a functions:
tx_hash = contractName.functions.functionName(account.address).transact({"gas": 100000}))
transaction_receipt = w3.eth.wait_for_transaction_receipt(tx_hash)
Then I can call the same getter: print(contractName.functions.stateVariable(account.address).call()
It returns 1.

write the main function that takes no parameters and returns nothing

I need to write the main function that takes no parameters and returns nothing. The main function should call the make_periodic_table function and print the returned table with each individual element on a separate line and add a call to the main function. Be certain to protect the call to main with an if statement. but when I tried My output it's not the expected. How Can I solve this ? please.
periodic_table_list = [
# [symbol, name, atomic_mass]
["Ac","Actinium",227],
["Ag","Silver",107.8682],
["Al","Aluminum",26.9815386],
["Am","Americium",243],
["Ar","Argon",39.948],
["As","Arsenic",74.9216],
["At","Astatine",210],
["Au","Gold",196.966569],
["B","Boron",10.811],
["Ba","Barium",137.327],
["Be","Beryllium",9.012182],
["Bh","Bohrium",272],
["Bi","Bismuth",208.9804],
["Bk","Berkelium",247],
["Br","Bromine",79.904],
["C","Carbon",12.0107],
["Ca","Calcium",40.078],
["Cd","Cadmium",112.411],
["Ce","Cerium",140.116],
["Cf","Californium",251],
["Cl","Chlorine",35.453],
["Cm","Curium",247],
["Cn","Copernicium",285],
["Co","Cobalt",58.933195],
["Cr","Chromium",51.9961],
["Cs","Cesium",132.9054519],
["Cu","Copper",63.546],
["Db","Dubnium",268],
["Ds","Darmstadtium",281],
["Dy","Dysprosium",162.5],
["Er","Erbium",167.259],
["Es","Einsteinium",252],
["Eu","Europium",151.964],
["F","Fluorine",18.9984032],
["Fe","Iron",55.845],
["Fl","Flerovium",289],
["Fm","Fermium",257],
["Fr","Francium",223],
["Ga","Gallium",69.723],
["Gd","Gadolinium",157.25],
["Ge","Germanium",72.64],
["H","Hydrogen",1.00794],
["He","Helium",4.002602],
["Hf","Hafnium",178.49],
["Hg","Mercury",200.59],
["Ho","Holmium",164.93032],
["Hs","Hassium",270],
["I","Iodine",126.90447],
["In","Indium",114.818],
["Ir","Iridium",192.217],
["K","Potassium",39.0983],
["Kr","Krypton",83.798],
["La","Lanthanum",138.90547],
["Li","Lithium",6.941],
["Lr","Lawrencium",262],
["Lu","Lutetium",174.9668],
["Lv","Livermorium",293],
["Mc","Moscovium",288],
["Md","Mendelevium",258],
["Mg","Magnesium",24.305],
["Mn","Manganese",54.938045],
["Mo","Molybdenum",95.96],
["Mt","Meitnerium",276],
["N","Nitrogen",14.0067],
["Na","Sodium",22.98976928],
["Nb","Niobium",92.90638],
["Nd","Neodymium",144.242],
["Ne","Neon",20.1797],
["Nh","Nihonium",284],
["Ni","Nickel",58.6934],
["No","Nobelium",259],
["Np","Neptunium",237],
["O","Oxygen",15.9994],
["Og","Oganesson",294],
["Os","Osmium",190.23],
["P","Phosphorus",30.973762],
["Pa","Protactinium",231.03588],
["Pb","Lead",207.2],
["Pd","Palladium",106.42],
["Pm","Promethium",145],
["Po","Polonium",209],
["Pr","Praseodymium",140.90765],
["Pt","Platinum",195.084],
["Pu","Plutonium",244],
["Ra","Radium",226],
["Rb","Rubidium",85.4678],
["Re","Rhenium",186.207],
["Rf","Rutherfordium",267],
["Rg","Roentgenium",280],
["Rh","Rhodium",102.9055],
["Rn","Radon",222],
["Ru","Ruthenium",101.07],
["S","Sulfur",32.065],
["Sb","Antimony",121.76],
["Sc","Scandium",44.955912],
["Se","Selenium",78.96],
["Sg","Seaborgium",271],
["Si","Silicon",28.0855],
["Sm","Samarium",150.36],
["Sn","Tin",118.71],
["Sr","Strontium",87.62],
["Ta","Tantalum",180.94788],
["Tb","Terbium",158.92535],
["Tc","Technetium",98],
["Te","Tellurium",127.6],
["Th","Thorium",232.03806],
["Ti","Titanium",47.867],
["Tl","Thallium",204.3833],
["Tm","Thulium",168.93421],
["Ts","Tennessine",294],
["U","Uranium",238.02891],
["V","Vanadium",50.9415],
["W","Tungsten",183.84],
["Xe","Xenon",131.293],
["Y","Yttrium",88.90585],
["Yb","Ytterbium",173.054],
["Zn","Zinc",65.38],
["Zr","Zirconium",91.224],
]
# [symbol, name, atomic_mass]
SYMBOl_INDEX = 0
NAME_INDEX = 1
ATOMIC_MASS_INDEX = 2
def main():
print(f'{SYMBOl_INDEX}, {NAME_INDEX}, {ATOMIC_MASS_INDEX}')
if __name__ == "__main__":
main() ```
It sounds like you will have to create a function (make_periodic_table) and then call it within main.
def make_periodic_table(PTL):
for element in PTL:
print(element)
main():
make_periodic_table(periodic_table_list)
To better explain why yours wasn't working, (I'm not 100% on the error you're getting but) in your print statement in main, you're not referencing anything, it's like pointing to thin air and saying print... it won't work, but if you point to an employee and say print this, they may go print something for you. (not the best analogy but you probably only care about the solution anyways)
also, I'm not sure what your professor meant by "Be certain to protect the call to main with an if statement". I would add something like this in your main function (replace everything in the main function with this) if no one else replies to this:
if periodic_table_list:
make_periodic_table(periodic_table_list)
else:
print("cannot find periodic table")

Python3 threading on AWS Lambda

I am using flask, and have a route that sends emails to people. I am using threading to send them faster. When i run it on my local machine it takes about 12 seconds to send 300 emails. But when I run it on lambda thorough API Gateway it times out.
here's my code:
import threading
def async_mail(app, msg):
with app.app_context():
mail.send(msg)
def mass_mail_sender(order, user, header):
html = render_template('emails/pickup_mail.html', bruger_order=order.ordre, produkt=order.produkt)
msg = Message(recipients=[user],
sender=('Sender', 'infor#example.com'),
html=html,
subject=header)
thread = threading.Thread(target=async_mail, args=[create_app(), msg])
thread.start()
return thread
#admin.route('/lager/<url_id>/opdater', methods=['POST'])
def update_stock(url_id):
start = time.time()
if current_user.navn != 'Admin':
abort(403)
if request.method == 'POST':
produkt = Produkt.query.filter_by(url_id=url_id)
nyt_antal = int(request.form['bestilt_hjem'])
produkt.balance = nyt_antal
produkt.bestilt_hjem = nyt_antal
db.session.commit()
orders = OrdreBog.query.filter(OrdreBog.produkt.has(func.lower(Produkt.url_id == url_id))) \
.filter(OrdreBog.produkt_status == 'Ikke klar').all()
threads = []
for order in orders:
if order.antal <= nyt_antal:
nyt_antal -= order.antal
new_thread = mass_mail_sender(order, order.ordre.bruger.email, f'Din bog {order.produkt.titel} er klar til afhentning')
threads.append(new_thread)
order.produkt_status = 'Klar til afhentning'
db.session.commit()
for thread in threads:
try:
thread.join()
except Exception:
pass
end = time.time()
print(end - start)
return 'Emails sendt'
return ''
AWS lambda functions designed to run functions within these constraints:
Memory– The amount of memory available to the function during execution. Choose an amount between 128 MB and 3,008 MB in 64-MB increments.
Lambda allocates CPU power linearly in proportion to the amount of memory configured. At 1,792 MB, a function has the equivalent of one full vCPU (one vCPU-second of credits per second).
Timeout – The amount of time that Lambda allows a function to run before stopping it. The default is 3 seconds. The maximum allowed value is 900 seconds.
To put this in your mail sending multi threaded python code. The function will terminate automatically either when your function execution completes successfully or it reaches configured timeout.
I understand you want single python function to send n number of emails "concurrently". To achieve this with lambda try the "Concurrency" setting and trigger your lambda function through a local script, S3 hosted html/js triggered by cloud watch or EC2 instance.
Concurrency – Reserve concurrency for a function to set the maximum number of simultaneous executions for a function. Provision concurrency to ensure that a function can scale without fluctuations in latency.
https://docs.aws.amazon.com/lambda/latest/dg/configuration-console.html
Important: All above settings will affect your lambda function execution cost significantly. So plan and compare before applying.
If you need any more help, let me know.
Thank you.

Avoid memory leaks with promises and loop in coffee-script (no await)

I am currently trying to perform some operations using promises in a loop but I ended up with huge memory leaks.
My problem is exactly the one pointed out in this article but as opposite to author, I am writing in coffee-script (yes, with hyphen. Which means coffeescript 1.12 and not the latest version). Thus, I am not able to use "await" key word (this is a casual guess since each time I want to use it, I got "await is not defined" error).
This is my original code (with memory leaks) :
recursiveFunction: (next = _.noop) ->
_data = #getSomeData()
functionWithPromise(_data).then (_enrichedData) =>
#doStuffWithEnrichedData(_enrichedData)
#recursiveFunction()
.catch (_err) =>
#log.error _err.message
#recursiveFunction()
So according to the article I linked, I would have to do something like that :
recursiveFunction: (next = _.noop) ->
_data = #getSomeData()
_enrichedData = await functionWithPromise(_data)
#recursiveFunction()
But then again, I am stuck because I can't use "await" key word. What would be the best approach then ?
EDIT:
Here is my real original code. What I am trying to achieve is a face-detection application. This function is located in a lib and I am using "Service" variable to expose variables between libs. In order to get frame from webcam, I am using opencv4nodejs.
faceapi = require('face-api.js')
tfjs = require('#tensorflow/tfjs-node')
(...)
# Analyse the new frame
analyseFrame: (next = _.noop) ->
# Skip if not capturing
return unless Service.isCapturing
# get frame
_frame = Service.videoCapture.getFrame()
# get frame date, and
#currentFrameTime = Date.now()
# clear old faces in history
#refreshFaceHistory(#currentFrameTime)
#convert frame to a tensor
try
_data = new Uint8Array(_frame.cvtColor(cv.COLOR_BGR2RGB).getData().buffer)
_tensorFrame = tfjs.tensor3d(_data, [_frame.rows, _frame.cols, 3])
catch _err
#log.error "Error instantiating tensor !!!"
#log.error _err.message
# find faces on frames
faceapi.detectAllFaces(_tensorFrame, #faceDetectionOptions).then (_detectedFaces) =>
#log.debug _detectedFaces
# fill face history with detceted faces
_detectedFaces = #fillFacesHistory(_detectedFaces)
# draw boxes on image
Service.videoCapture.drawFaceBoxes(_frame, _detectedFaces)
# Get partial time
Service.frameDuration = Date.now() - #currentFrameTime
# write latency on image
Service.videoCapture.writeLatency(_frame, Service.frameDuration)
# show image
Service.faceRecoUtils.showImage(_frame)
# Call next
_delayNextFrame = Math.max(0, 1000/#options.fps - Service.frameDuration)
setTimeout =>
# console.log "Next frame : #{_delayNextFrame}ms - TOTAL : #{_frameDuration}ms"
#analyseFrame()
, (_delayNextFrame)
The solution was to dispose the tensor copy sent to detectFaces.
faceapi.detectAllFaces(_tensorFrame, #faceDetectionOptions).then (_detectedFaces) =>
(...)
_tensorFrame.dispose()
(...)

How to reference/return a value from SignalR?

This is the code I have:
from signalr_aio import Connection
if __name__ == "__main__":
# Create connection
# Users can optionally pass a session object to the client, e.g a cfscrape session to bypass cloudflare.
connection = Connection('https://beta.bittrex.com/signalr', session=None)
hub = connection.register_hub('c2')
hub.server.invoke('GetAuthContext', API_KEY) #Invoke 0 Creates the challenge that needs to be signed by the create_signature coroutine
signature = await create_signature(API_SECRET, challenge) #Creates the signature that needs to authenticated in the Authenticate query
hub.server.invoke('Authenticate', API_KEY, signature) #Invoke 1 authenticates user to account level information
connection.start()
What I have to do is verify my identity by getting a string-type challenge by the GetAuthContext call, then create a string-type signature using that challenge, and then pass that signature to the Authenticatecall. The problem I'm having is that that I need to enter the return value of the GetAuthContext into the challenge parameter of the create_signature coroutine. I'm guessing from the comment next to the below example that every invoke method gets marked as I([index of method]), so I would have to do signature = await create_signature(API_SECRET, 'I(0)')
async def on_debug(**msg):
# In case of `queryExchangeState` or `GetAuthContext`
if 'R' in msg and type(msg['R']) is not bool:
# For the simplicity of the example I(1) corresponds to `Authenticate` and I(0) to `GetAuthContext`
# Check the main body for more info.
if msg['I'] == str(2):
decoded_msg = await process_message(msg['R'])
print(decoded_msg)
elif msg['I'] == str(3):
signature = await create_signature(API_SECRET, msg['R'])
hub.server.invoke('Authenticate', API_KEY, signature)
Later this example gets assigned to connection.received ( connection.received += on_debug ) so I'm guessing that after connection.start() I have to put connection.recieved() to call the on_debug coroutine which will verify me, but for now I just want to understand how to reference the .invoke() methods to use within a function or coroutine.
I am far from an expert, but the feed from Bittrex is indeed a Dictionary.
for i in range(0, len(decoded_msg['D'])):
print('The Currency pair is:{0} the Bid is:{1} and the Ask is :{2}'.format(decoded_msg['D'][i]['M'], decoded_msg['D'][i]['B'], decoded_msg['D'][i]['A']))

Resources