I am facing this error I tried everything, I deployed this function as a google cloud function, but when I am running the triggering URL I am getting an error
Request cannot be handled
Logs are
TypeError: process_session() missing 1 required positional argument: 'session'
at call_user_function (/env/local/lib/python3.7/site-packages/google/cloud/functions/worker_v2.py:261)
at invoke_user_function (/env/local/lib/python3.7/site-packages/google/cloud/functions/worker_v2.py:268)
at run_http_function (/env/local/lib/python3.7/site-packages/google/cloud/functions/worker_v2.py:402)
Function code
def process_session(self, session, utc_offset=0):
s = {}
try:
edfbyte, analysis = process_session(session, utc_offset)
report_json, quality = process_analysis(analysis, session.ref.id)
# save the EDF
path = 'Users/' + session.ref.get("uid") + '/session-' + session.ref.get("sessionId") + '.edf'
path_report = 'Users/' + session.ref.get("uid") + '/session-' + session.ref.get("sessionId") + '.json'
bucket = storage.bucket("......")
bucket.blob(path).upload_from_string(edfbyte, content_type='application/octet-stream')
bucket.blob(path_report).upload_from_string(report_json, content_type='application/json')
# update session
s = session.to_dict()
s[u'macid'] = analysis['header']['macid']
s[u'quality'] = quality
s[u'edfPath'] = path
s[u'reportPath'] = path_report
s[u'timestamp'] = dateutil.parser.parse(analysis['header']['startdate'])
self.db.collection(u'ProcessedSessions').document(session.ref.id).set(s)
try:
self.db.collection(u'UnprocessedSessions').document(session.ref.id).delete()
#session.ref.reference.delete()
except:
pass
return True, 0
except Exception as e:
traceback.print_exc()
s = session.to_dict()
if u'attempt' in s:
attempt = s['attempt']
else:
attempt = 0
self.db.collection(u'UnprocessedSessions').document(session.ref.id).set({u'attempt': attempt + 1}, merge=True)
return False, attempt + 1
You are trying to pass 3 parameters in a HTTP Trigger, Python only accepts 1 parameter wich is a Flask Request Object.
HTTP Functions
Your function is passed a single parameter, (request), which is a
Flask Request object. Return any value from your function that can be
handled by the Flask make_response method. The result will be the HTTP
response.
You can try this to invoke your function:
def functionx (request):
# the GCF always receive only 1 paramater an HTTP request object (flask request)
# you need to get the parameter from the request
# after that you can call your method
process_session(param1,param2,param3)
def process_session(self, session, utc_offset=0):
#dosomenthing
print ("some code here")
Or if you need to use more parameters for your entry point, you can use a Background function.
Background Functions
Background functions are passed arguments holding
data associated with the event that triggered the function's
execution. In the Python runtime, your function is passed the
arguments (data, context)
Is this all part of def process_session? I can't see indentation.
Is it inside a class-object?
Maybe try in line 4:
edfbyte, analysis = self.process_session(session, utc_offset)
The given error signals that you haven't referenced the class-object.
Related
Below is a module for querying and caching AWS STS tokens, the intention is to avoid querying STS if there is a valid token.
class Credentials:
def __init__(self):
self.sts_credentials = None
self.token_expiry_time = None
def is_token_expired(self):
current_time_with_buffer = datetime.now() + timedelta(minutes=2)
return not self.token_expiry_time or self.token_expiry_time < current_time_with_buffer
CREDENTIALS_ = Credentials()
def get_credentials():
if CREDENTIALS_.is_token_expired():
sts_client = boto3.client('sts')
LOGGER.info("The credentials are either empty or expiring, refreshing")
try:
sts_token = sts_client.assume_role(
RoleArn=os.environ["KINESIS_ASSUME_ROLE"],
RoleSessionName=str(uuid.uuid4()))
except Exception as e:
LOGGER.error(f"Error occurred while trying to assume role with {os.environ['KINESIS_ASSUME_ROLE']}", e)
raise e
CREDENTIALS_.sts_credentials = {
"aws_access_key_id": sts_token['Credentials']['AccessKeyId'],
"aws_secret_access_key": sts_token['Credentials']['SecretAccessKey'],
"aws_session_token": sts_token['Credentials']['SessionToken']
}
CREDENTIALS_.token_expiry_time = sts_token["Credentials"]["Expiration"]
return CREDENTIALS_.sts_credentials
One of the unit tests is as below, this passes in isolation, but fails when run alongside other tests, the reason being CREDENTIALS_ variable, which is modified by other tests, I can set this value to None, but I want to know what is the cleaner way of clearing the cached value
def test_get_credentials_refreshes_token_if_about_to_expire(sts_response, credentials):
with mock.patch("boto3.client") as mock_boto_client:
mock_assume_role = mock_boto_client.return_value.assume_role
mock_assume_role.return_value = sts_response
get_credentials()
actual_credentials = get_credentials()
calls = [call('sts'),
call().assume_role(RoleArn='arn:aws:iam::000000000000:role/dummyarn', RoleSessionName=ANY),
call('sts'),
call().assume_role(RoleArn='arn:aws:iam::000000000000:role/dummyarn', RoleSessionName=ANY)]
assert credentials == actual_credentials
mock_boto_client.assert_has_calls(calls)
The cleaner way would be to make sure that your unit tests are performing unit tests. This means that for every unit there should be no interaction with other units. Since you are using a global variable CREDENTIALS_, this is going to be nearly impossible.
1) easy fix
An easy fix would be to pass CREDENTIALS_ as input argument. Then you can create a fake CREDENTIALS_ object during each of the tests, that are tailored to your test conditions.
2) Better fix
A better solution would be, besides using the credential input argument, to break up the logic inside the get_credentials. By splitting it into smaller functions, you can separate the server logic and the credential updating. Making it easier to Mock and test. A possible division of the whole function would be:
get_sts_token
update_credentials
get_credentials
Now the get_sts_token has connections to the server, but the update_credentials and get_credentials do not have to directly interact with it.
Code
Example 1)
def update_credentials(credentials):
if credentials.is_token_expired():
sts_client = boto3.client('sts')
LOGGER.info("The credentials are either empty or expiring, refreshing")
try:
sts_token = sts_client.assume_role(
RoleArn=os.environ["KINESIS_ASSUME_ROLE"],
RoleSessionName=str(uuid.uuid4()))
except Exception as e:
LOGGER.error(f"Error occurred while trying to assume role with {os.environ['KINESIS_ASSUME_ROLE']}", e)
raise e
credentials.sts_credentials = {
"aws_access_key_id": sts_token['Credentials']['AccessKeyId'],
"aws_secret_access_key": sts_token['Credentials']['SecretAccessKey'],
"aws_session_token": sts_token['Credentials']['SessionToken']
}
credentials.token_expiry_time = sts_token["Credentials"]["Expiration"]
return credentials
# Where you need the credentials
CREDENTIALS_ = update_credentials(CREDENTIALS_)
CREDENTIALS_.sts_credentials
Now you can insert your own CREDENTIALS_ object in the test.
Example 2)
def get_sts_token():
sts_client = boto3.client('sts')
LOGGER.info("The credentials are either empty or expiring, refreshing")
try:
sts_token = sts_client.assume_role(
RoleArn=os.environ["KINESIS_ASSUME_ROLE"],
RoleSessionName=str(uuid.uuid4()))
except Exception as e:
LOGGER.error(f"Error occurred while trying to assume role with {os.environ['KINESIS_ASSUME_ROLE']}", e)
raise e
return sts_token
def update_credentials(credentials, sts_token):
credentials.sts_credentials = {
"aws_access_key_id": sts_token['Credentials']['AccessKeyId'],
"aws_secret_access_key": sts_token['Credentials']['SecretAccessKey'],
"aws_session_token": sts_token['Credentials']['SessionToken']
}
return credentials
def get_credentials(credentials: Credentials):
if credentials.is_token_expired():
sts_token = get_sts_token()
credentials = update_credentials(credentials, sts_token)
return credentials.sts_credentials
Unit-Test code for a function which does a validation operation and updates the Global dict - result_count = {'test_method':{'Total_tested': 0, 'passed': 0, 'failed': 0}
below is the function
def validate_response(testmethod, response, expected_data):
ra = response.json()
expected = expected_data['payload']
if (response.status_code == expected['response_status']) \
and (result_count[testmethod['folder']]["FAILED"] < 10):
------code logic - checks using jsondiff and re.expressions--
else:
missmatch = 'response status code missmatch'
update_result(testmethod, 'status_code', expected_data, response, missmatch, fail=True)
result_count[testmethod['folder']]['FAILED'] += 1
Need to write test for the above function. to check if the result_count is updated properly.
Regular expression library & jsondiff is also used in the mentioned function.
help required for mocking the global variable and using the same for testing.
During executing the test script below i was getting a key error for the global variable result_count. That implies that the code is unable to access the result_count.. the key error after the update - is throwing TypeError: '<=' not supported between instances of 'MagicMock' and 'int' as Error
The current issue is that the result_count is not updated when the below line code is executed.
partner_test.validate_response(test_input_mocker.method, response, expected_data)
my unit test script is as below
#patch("tests.p_test.result")
#patch("tests.p_test.result_count")
def test_validate_response_pass(result_count_mocker, monkeypatch, result_mocker, test_input_mocker):
# Build data for validate response function
response = Resp(200, {'message': 'pong'})
response_data = response.json()
expected_data = {some_test_data}
# global variable import and initialize result, result_count
from p_test import result, result_count
result_count.update(result_count_mocker.data)
result.update(result_mocker.data)
result_count_mocker.return_value = result_count_mocker.data
def update_result_mocker(*args):
mock operations here
return None
monkeypatch.setattr(partner_test, "update_result", update_result_mocker)
p_test.validate_response(test_input_mocker.method, response, expected_data)
Resp() in the test fnction is a response class created to mock the response object.
the issue can be resolved using #patch.dict(dict_name, values) below is the solution
#patch.dict(p_test.result, {dict_values })
#patch.dict(p_test.result_count, {dict_values})
def test_validate_response(monkeypatch, test_input_mocker, result_count_test):
response = Resp(200, {'message': 'pong'})
response_data = response.json()
expected_data = {----your data-----}
def update_result_mocker(*args):
# mock_operations
monkeypatch.setattr(p_test, "update_result", update_result_mocker)
p_test.validate_response(test_input_mocker.method, response, expected_data)
print(p_test.result_count)
I have this function that when hitting a rate limit will call itself again. It should eventually succeed and return the working data. It works normally then rate limiting works as expected, and finally when the data goes back to normal I get:
TypeError: 'NoneType' object is not subscriptable
def grabPks(pageNum):
# cloudflare blocks bots...use scraper library to get around this or build your own logic to store and use a manually generated cloudflare session cookie... I don't care 😎
req = scraper.get("sumurl.com/"+str(pageNum)).content
if(req == b'Rate Limit Exceeded'):
print("adjust the rate limiting because they're blocking us :(")
manPenalty = napLength * 3
print("manually sleeping for {} seconds".format(manPenalty))
time.sleep(manPenalty)
print("okay let's try again... NOW SERVING {}".format(pageNum))
grabPks(pageNum)
else:
tree = html.fromstring(req)
pk = tree.xpath("/path/small/text()")
resCmpress = tree.xpath("path/a//text()")
resXtend = tree.xpath("[path/td[2]/small/a//text()")
balance = tree.xpath("path/font//text()")
return pk, resCmpress, resXtend, balance
I've tried to move the return to outside of the else scope but then it throws:
UnboundLocalError: local variable 'pk' referenced before assignment
Your top level grabPks doesnt return anything if it is rate limited.
Think about this:
Call grabPks()
You're rate limited so you go into the if statement and call grabPks() again.
This time it succeeds so grabPks() returns the value to the function above it.
The first function now falls out of the if statement, gets to the end of the function and returns nothing.
Try return grabPks(pageNum) instead inside your if block.
well okay... I needed to return grabPKs to make it play nice...:
def grabPks(pageNum):
# cloudflare blocks bots...use scraper library to get around this or build your own logic to store and use a manually generated cloudflare session cookie... I don't care 😎
req = scraper.get("sumurl.com/"+str(pageNum)).content
if(req == b'Rate Limit Exceeded'):
print("adjust the rate limiting because they're blocking us :(")
manPenalty = napLength * 3
print("manually sleeping for {} seconds".format(manPenalty))
time.sleep(manPenalty)
print("okay let's try again... NOW SERVING {}".format(pageNum))
return grabPks(pageNum)
else:
tree = html.fromstring(req)
pk = tree.xpath("/path/small/text()")
resCmpress = tree.xpath("path/a//text()")
resXtend = tree.xpath("[path/td[2]/small/a//text()")
balance = tree.xpath("path/font//text()")
return pk, resCmpress, resXtend, balance
This code acts as an early warning system for ADFS failures, which works fine when run locally. Problem is that when I run it in Lambda, it loops non stop.
In short:
lambda_handler() runs pagecheck()
pagecheck() produces the info needed then passes 2 lists (msgdet_list, error_list) and an int (error_count) to notification().
notification() collates and prints the output. The output is two key variables (notificationheader and notificationbody).
I've #commentedOut the SNS piece which would usually email the info, and am using print() to instead send the info to CloudWatch logs until I can get the loop sorted. Logs:
CloudWatch logs
If I run this locally, it produces a clean single output. In Lambda, the function will loop until it times out. It's almost like every time the lists are updated, they're passed to the notification() module and it's run. I can limit the function time, but would rather fix the code!
Cheers,
tac
# This python/boto3/lambda script sends a request to an Office 365 landing page, parses return details to confirm a successful redirect to /
# the organisation ADFS homepage, authenticates homepge is correct, raises any errors, and sends a consolodated report to /
# an AWS SNS topic.
# Run once to produce pageserver and htmlchar values for global variables.
# Import required modules
import boto3
import urllib.request
from urllib.request import Request, urlopen
from datetime import datetime
import time
import re
import sys
# Global variables to be set
url = "https://outlook.com/CONTOSSO.com"
adfslink = "https://sts.CONTOSSO.com/adfs/ls/?client-request-id="
# Input after first run
pageserver = "Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0"
htmlchar = 18600
# Input AWS SNS ARN
snsarn = 'arn:aws:sns:ap-southeast-2:XXXXXXXXXXXXX:Daily_Check_Notifications_CONTOSSO'
sns = boto3.client('sns')
def pagecheck():
# Present the request to the webpage as if coming from a user in a browser
user_agent = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)'
values = {'name' : 'user'}
headers = { 'User-Agent' : user_agent }
data = urllib.parse.urlencode(values)
data = data.encode('ascii')
# "Null" the Message Detail and Error lists
msgdet_list = []
error_list = []
request = Request(url)
req = urllib.request.Request(url, data, headers)
response = urlopen(request)
with urllib.request.urlopen(request) as response:
# Get the URL. This gets the real URL.
acturl = response.geturl()
msgdet_list.append("\nThe Actual URL is:")
msgdet_list.append(str(acturl))
if adfslink not in acturl:
error_list.append(str("Redirect Fail"))
# Get the HTTP resonse code
httpcode = response.code
msgdet_list.append("\nThe HTTP code is: ")
msgdet_list.append(str(httpcode))
if httpcode//200 != 1:
error_list.append(str("No HTTP 2XX Code"))
# Get the Headers as a dictionary-like object
headers = response.info()
msgdet_list.append("\nThe Headers are:")
msgdet_list.append(str(headers))
if response.info() == "":
error_list.append(str("Header Error"))
# Get the date of request and compare to UTC (DD MMM YYYY HH MM)
date = response.info()['date']
msgdet_list.append("The Date is: ")
msgdet_list.append(str(date))
returndate = str(date.split( )[1:5])
returndate = re.sub(r'[^\w\s]','',returndate)
returndate = returndate[:-2]
currentdate = datetime.utcnow()
currentdate = currentdate.strftime("%d %b %Y %H%M")
if returndate != currentdate:
date_error = ("Date Error. Returned Date: ", returndate, "Expected Date: ", currentdate, "Times in UTC (DD MMM YYYY HH MM)")
date_error = str(date_error)
date_error = re.sub(r'[^\w\s]','',date_error)
error_list.append(str(date_error))
# Get the server
headerserver = response.info()['server']
msgdet_list.append("\nThe Server is: ")
msgdet_list.append(str(headerserver))
if pageserver not in headerserver:
error_list.append(str("Server Error"))
# Get all HTML data and confirm no major change to content size by character lenth (global var: htmlchar).
html = response.read()
htmllength = len(html)
msgdet_list.append("\nHTML Length is: ")
msgdet_list.append(str(htmllength))
msgdet_list.append("\nThe Full HTML is: ")
msgdet_list.append(str(html))
msgdet_list.append("\n")
if htmllength // htmlchar != 1:
error_list.append(str("Page HTML Error - incorrect # of characters"))
if adfslink not in str(acturl):
error_list.append(str("ADFS Link Error"))
error_list.append("\n")
error_count = len(error_list)
if error_count == 1:
error_list.insert(0, 'No Errors Found.')
elif error_count == 2:
error_list.insert(0, 'Error Found:')
else:
error_list.insert(0, 'Multiple Errors Found:')
# Pass completed results and data to the notification() module
notification(msgdet_list, error_list, error_count)
# Use AWS SNS to create a notification email with the additional data generated
def notification(msgdet_list, error_list, errors):
datacheck = str("\n".join(msgdet_list))
errorcheck = str("\n".join(error_list))
notificationbody = str(errorcheck + datacheck)
if errors >1:
result = 'FAILED!'
else:
result = 'passed.'
notificationheader = ('The daily ADFS check has been marked as ' + result + ' ' + str(errors) + ' ' + str(error_list))
if result != 'passed.':
# message = sns.publish(
# TopicArn = snsarn,
# Subject = notificationheader,
# Message = notificationbody
# )
# Output result to CloudWatch logstream
print('Response: ' + notificationheader)
else:
print('passed')
sys.exit()
# Trigger the Lambda handler
def lambda_handler(event, context):
aws_account_ids = [context.invoked_function_arn.split(":")[4]]
pagecheck()
return "Successful"
sys.exit()
Your CloudWatch logs contain the following error message:
Process exited before completing request
This is caused by invoking sys.exit() in your code. Locally your Python interpreter will just terminate when encountering such a sys.exit().
AWS Lambda on the other hand expects a Python function to just return and handles sys.exit() as an error. As your function probably got invoked asynchronously AWS Lambda retries to execute it twice.
To solve your problem, you can replace the occurences of sys.exit() with return or even better, just remove the sys.exit() calls, as there would be already implicit returns in the places where you use sys.exit().
I have a problem with Burpsuite API that I can't find a proper function to print out the response for edited requests . I'm developing a new plugin for burpsuite with python . myscript is simply takes requests from proxy then it edit headers and send it again .
from burp import IBurpExtender
from burp import IHttpListener
import re,urllib2
class BurpExtender(IBurpExtender, IHttpListener):
def registerExtenderCallbacks(self, callbacks):
self._callbacks = callbacks
self._helpers = callbacks.getHelpers()
callbacks.setExtensionName("Burp Plugin Python Demo")
callbacks.registerHttpListener(self)
return
def processHttpMessage(self, toolFlag, messageIsRequest, currentRequest):
# only process requests
if messageIsRequest:
requestInfo = self._helpers.analyzeRequest(currentRequest)
#timestamp = datetime.now()
#print "Intercepting message at:", timestamp.isoformat()
headers = requestInfo.getHeaders()
#print url
if(requestInfo.getMethod() == "GET"):
print "GET"
print requestInfo.getUrl()
response = urllib2.urlopen(requestInfo.getUrl())
print response
elif(requestInfo.getMethod() == "POST"):
print "POST"
print requestInfo.getUrl()
#for header in headers:
#print header
bodyBytes = currentRequest.getRequest()[requestInfo.getBodyOffset():]
bodyStr = self._helpers.bytesToString(bodyBytes)
bodyStr = re.sub(r'=(\w+)','=<xss>',bodyStr)
newMsgBody = bodyStr
newMessage = self._helpers.buildHttpMessage(headers, newMsgBody)
print "Sending modified message:"
print "----------------------------------------------"
print self._helpers.bytesToString(newMessage)
print "----------------------------------------------\n\n"
currentRequest.setRequest(newMessage)
return
You need to print the response but you don't do anything in case messageIsRequest is false. When messageIsRequest is false it means that the currentRequest is a response and you can print out the response as you did for the request. I did it in Java like this:
def processHttpMessage(self, toolFlag, messageIsRequest, httpRequestResponse):
if messageIsRequest:
....
else
HTTPMessage = httpRequestResponse.getResponse()
print HTTPMessage
There is even a method that lets you bind request and response together when using a proxy. It can be found in IInterceptedProxyMessage:
/**
* This method retrieves a unique reference number for this
* request/response.
*
* #return An identifier that is unique to a single request/response pair.
* Extensions can use this to correlate details of requests and responses
* and perform processing on the response message accordingly.
*/
int getMessageReference();
I don't think it is supported for HTTPListeners.
I am writing the extensions in Java and tried to translate to Python for this anwser. I haven't tested this code and some bugs might be introduced due to translation.