Lambda behaves differently when invoked from Console (test) and through AWS Gateway REST API - python-3.x

Problem: Pandas behaves differently when
I call it from the "invoke" test (and via AWS Gateway which is pointed to the said Lambda)
I call it through a REST API via AWS Gateway
Lambda works fine with A) (it returns intended values) but does not work with B) (returns NaN values).
Details: I have a simple python script in a Lambda (python 3.8). With the code, I have uploaded pandas, numpy and a .csv file into the Lambda. I built a rest API with AWS Gateway that uses the Lambda as a resource.
My objective is to load the .csv into a Pandas dataframe when Lambda is requested, then filter the dataframe with the event parameters to find particular values, and return those values.
Sample code below.
Any help would be appreciated.
Code
Lambda:
def lambda_handler(event, context):
# extract filter values from request
vendor = event['Vendor']
# load the csv
data = pd.read_csv('vendor_data.csv')
print("Data shape", data.shape)
# correct types
data.quantity = (data.quantity).apply(lambda x: int(x))
data.loc[:,'Vendor'] = data['Vendor'].astype('str') # I also tried astype('string'), doesn't change
# filter the dataframe with Vendor == vendor
condition = (data.Vendor.str.contains(str(vendor))) #I also tried (data.Vendor == vendor)
data = data.loc[condition]
#verify it worked: it works with 'invoke' but not through rest API
print('First five rows', data[['Vendor']].head())
data = data['quantity'].apply(lambda x: int(x))
sum = data.sum()
return sum
Gateway Rest API
Request model template:
{ "$schema": "http://json-schema.org/draft-04/schema#",
"title": "Request",
"type": "object",
"properties": {
"Vendor": {
"type": "string"
},
},
"required": ["Vendor"]
}
Response model template:
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Response",
"type": "object",
"properties": {
"sum": {
"type": "number"
}
}
}
Example test (post to rest API)
PARAMS = {
"Vendor": "2202630"
}
PARAMS = json.dumps(PARAMS)
print(PARAMS)
r = requests.post(url = URL, data = PARAMS)
print(r.content)
results in:
{ "Vendor": "2202630"}
b'{"statusCode": 200, "body": NaN, "event": {"Vendor": "2202630"}, "data": "[]"}'
Example test (through AWS Gateway):
Request:
{
"Vendor": "2202630"
}
Response body:
{
"statusCode": 200,
"body": 599,
"event": {
"Vendor": "22026305"
},
"data": "[50 44 38 45 50 45 38 26 18 31 21 9 53 47 45 39]"
}

Related

How to mock Athena query results values with Moto3 for a specific table?

I am using pytest and moto3 to test some code similar to this:
response = athena_client.start_query_execution(
QueryString='SELECT * FROM xyz',
QueryExecutionContext={'Database': myDb},
ResultConfiguration={'OutputLocation': someLocation},
WorkGroup=myWG
)
execution_id = response['QueryExecutionId']
if response['QueryExecution']['Status']['State'] == 'SUCCEEDED':
response = athena_client.get_query_results(
QueryExecutionId=execution_id
)
results = response['ResultSet']['Rows']
...etc
In my test I need that the values from results = response['ResultSet']['Rows'] are controlled by the test. I am using some code like this:
backend = athena_backends[DEFAULT_ACCOUNT_ID]["us-east-1"]
rows = [{"Data": [{"VarCharValue": "xyz"}]}, {"Data": [{"VarCharValue": ...}, etc]}]
column_info = [
{
"CatalogName": "string",
"SchemaName": "string",
"TableName": "xyz",
"Name": "string",
"Label": "string",
"Type": "string",
"Precision": 123,
"Scale": 123,
"Nullable": "NOT_NULL",
"CaseSensitive": True,
}
]
results = QueryResults(rows=rows, column_info=column_info)
backend.query_results[NEEDED_QUERY_EXECUTION_ID] = results
but that is not working as I guess NEEDED_QUERY_EXECUTION_ID is not known before from the test. How can I control it?
UPDATE
Based on suggestion I tried to use:
results = QueryResults(rows=rows, column_info=column_info)
d = defaultdict(lambda: results.to_dict())
backend.query_results = d
to force a return of values, but it seems not working as from the moto3's models.AthenaBackend.get_query_results, I have this code:
results = (
self.query_results[exec_id]
if exec_id in self.query_results
else QueryResults(rows=[], column_info=[])
)
return results
which will fail as the if condition won't be satifsfied.
Extending the solution of the defaultdict, you could create a custom dictionary that contains all execution_ids, and always returns the same object:
class QueryDict(dict):
def __contains__(self, item):
return True
def __getitem__(self, item):
rows = [{"Data": [{"VarCharValue": "xyz"}]}, {"Data": [{"VarCharValue": "..."}]}]
column_info = [
{
"CatalogName": "string",
"SchemaName": "string",
"TableName": "xyz",
"Name": "string",
"Label": "string",
"Type": "string",
"Precision": 123,
"Scale": 123,
"Nullable": "NOT_NULL",
"CaseSensitive": True,
}
]
return QueryResults(rows=rows, column_info=column_info)
backend = athena_backends[DEFAULT_ACCOUNT_ID]["us-east-1"]
backend.query_results = QueryDict()
An alternative solution to using custom dictionaries would to be seed Moto.
Seeding Moto ensures that it will always generate the same 'random' identifiers, which means you always know what the value of NEEDED_QUERY_EXECUTION_ID is going to be.
backend = athena_backends[DEFAULT_ACCOUNT_ID]["us-east-1"]
rows = [{"Data": [{"VarCharValue": "xyz"}]}, {"Data": [{"VarCharValue": "..."}]}]
column_info = [...]
results = QueryResults(rows=rows, column_info=column_info)
backend.query_results["bdd640fb-0667-4ad1-9c80-317fa3b1799d"] = results
import requests
requests.post("http://motoapi.amazonaws.com/moto-api/seed?a=42")
# Test - the execution id will always be the same because we just seeded Moto
execution_id = athena_client.start_query_execution(...)
Documentation on seeding Moto can be found here: http://docs.getmoto.org/en/latest/docs/configuration/recorder/index.html#deterministic-identifiers
(It only talks about seeding Moto in the context of recording/replaying requests, but the functionality can be used on it's own.)

Is it possible for azure logic app to receive a file from incoming http request?

I wanted to know if its possible to perform a file upload request to azure logic app's HTTP listener?
I am not looking for built-in HTTP trigger which makes an HTTP call to the specified URL OR built-in HTTP action which makes an HTTP call to the specified URL
One of the workarounds is through postman. Here is my logic app for your reference
postman request :-
Here is the output :-
Yes, it's possible for an Azure Logic App to receive files via an HTTP POST request. Here is the request body JSON schema to use in the Logic App:
{
"properties": {
"formdata": {
"items": {
"properties": {
"key": {
"type": "string"
},
"type": {
"type": "string"
},
"value": {
"type": "string"
}
},
"required": [
"key",
"value",
"type"
],
"type": "object"
},
"type": "array"
},
"mode": {
"type": "string"
}
},
"type": "object"
}
The Python script below will send a request to the Logic App, including a dictionary of parameters and a separate dictionary associating each filename with its contents.
import requests
import pathlib
attachments = ["path/to/first_file.txt", "path/to/second_file.txt"] # Insert file paths
logic_app_url = "paste_logic_app_url_here" # Insert URL in quote marks
file_dict = {}
for filepath in attachments:
file_dict[pathlib.Path(filepath).name] = open(filepath, 'rb')
payload = {"first_key": "first_val"} # Extra fields to include in your request
response = requests.post(logic_app_url, headers=None, data=payload,
files=file_dict)
I've run the request above, and it works. The request is received and processed by the Logic App. However, I haven't yet figured out how to parse the individual attachments in the Azure Logic App GUI. I think this may require a For Each loop as explained in Microsoft docs. I hope this helps!

How to iterate through each property element in API response in Katalon studio?

I'm writing test script in katalon studio to verify response body of the API.My response body is of format:
{
"status": "Success",
"correlationCode": "1234-5678",
"type": {
"id": 51247,
"name": "Student",
},
"data": {
"name": "Sara Nieves",
"gender": "Female",
"dob": "1995-08-06",
"libraryCard": {
"id": "11178",
"type": "Seniors"
},
"qualifications": [
{
"id": "45650986546",
"name": "Graduate Certificate in Environmental Engineering Management"
}
]
}
}
I want to verify that none of the elements return 'null' value. Since, the elements returned for the API response are not static(meaning name, gender etc might not get returned every time) therefore, i can't use something like "data.name" to verify if it has null value. So, i want a generic way to loop through each and every attribute returned and check if its value is returned as null or not.
Any help will be much appreciated. Thanks!
You have the error message:
groovy.lang.MissingMethodException: No signature of method: WSVerification1569811424284$_run_closure1.doCall() is applicable for argument types: (com.kms.katalon.core.testobject.ResponseObject) values: [200 1 KB] 22572.groovy:21)
I assume your response object type: com.kms.katalon.core.testobject.ResponseObject
The code to parse response as json and validate it:
import groovy.json.JsonSlurper
/**
* the recursive method to validate that json object does not have null values
* #param obj - the parsed json object (sequence of maps and lists)
* #param path - a variable to know where the error occurred in json data.
*/
void assertNoNullValue(Object obj, String path='ROOT'){
//the main assertion
assert obj!=null : "value in json could not be null: $path"
if(obj instanceof Map){
//iterate each key-value in map and validate the value recursively
obj.each{k,v-> assertNoNullValue(v,path+".$k") }
} else if(obj instanceof List){
//iterate each value in list and validate the value recursively
obj.eachWithIndex{v,i-> assertNoNullValue(v,path+"[$i]") }
}
}
def response = ...
assert response.isJsonContentType()
def responseText = response.getResponseText()
//parse body
def data = new JsonSlurper().parseText(responseText)
assertNoNullValue(data)
This solution is not as precise as the one suggested by #dagget, but it is a quick check:
def response = '''
{
"status": "Success",
"correlationCode": "1234-5678",
"type": {
"id": 51247,
"name": "Student",
},
"data": {
"name": "Sara Nieves",
"gender": "femmina",
"dob": "1995-08-06",
"libraryCard": {
"id": "11178",
"type": "Seniors"
},
"qualifications": [
{
"id": "45650986546",
"name": "Graduate Certificate in Environmental Engineering Management"
}
]
}
}
'''
assert !response.contains("null")

How to fix "error": "'parameter_name'" when using IBM Cloud Function's REST API?

I have an action in IBM Cloud Functions that only receives one parameter: "frame". I'm using Postman to test the REST API endpoint provided with the action. However, when I provide the "frame" parameter it returns the following:
"response": {
"result": {
"error": "'frame'"
},
"status": "application error",
"success": false
}
I've experienced this problem when I invoke this action in the IBM Cloud Functions' console. I resolve it by erasing a space in the input modal and adding it again, then it works like a charm in the console. However, I can't do the same thing with an HTTP request.
The way I'm currently doing the HTTP request is like this:
POST https://us-south.functions.cloud.ibm.com/api/v1/namespaces/{namespace}/actions/{action_name}?blocking=true&frame={value}
The action should return the result I'm expecting but it doesn't do that right now. Please help me, any answers would be great!
EDIT:
This is the action's code:
import requests, base64, json, cv2
from PIL import Image
from six import BytesIO
def json_to_dict(json_str):
return json.loads(json.dumps(json_str))
def frame_to_bytes(frame):
frame_im = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
pil_im = Image.fromarray(frame_im)
stream = BytesIO()
pil_im.save(stream, format="JPEG")
stream.seek(0)
img_for_post = stream.read()
img_base64 = base64.b64encode(img_for_post)
return img_base64
def main(dict):
cap = cv2.VideoCapture(dict['frame'])
if not cap.isOpened():
return { "error": "Unable to open video source" }
ret, frame = cap.read()
if ret is False:
return { "error": "Unable to read video source" }
# openALPR API part
OPENALPR_SECRET_KEY = {my_secret_key}
url = "https://api.openalpr.com/v2/recognize_bytes?recognize_vehicle=1&country=us&secret_key=%s" % (
OPENALPR_SECRET_KEY)
r = requests.post(url, data=frame_to_bytes(frame))
resp = json_to_dict(r.json())
print(resp)
if not resp['results']:
return { "error": "Plate number not recognized" }
plates = []
for plate in resp['results']:
if plate['confidence'] < 75:
pass
else:
print(plate['plate'])
plates.append(plate['plate'])
return { "plates": plates }
This is the activation response (the status returned was 502 Bad Gateway according to Postman):
{
"activationId": "5a83396b9f53447483396b9f53e47452",
"annotations": [
{
"key": "path",
"value": "{namespace}/{name}"
},
{
"key": "waitTime",
"value": 5531
},
{
"key": "kind",
"value": "python:3.7"
},
{
"key": "timeout",
"value": false
},
{
"key": "limits",
"value": {
"concurrency": 1,
"logs": 10,
"memory": 1024,
"timeout": 60000
}
},
{
"key": "initTime",
"value": 3226
}
],
"duration": 3596,
"end": 1560669652454,
"logs": [],
"name": "{name}",
"namespace": "{namesapce}",
"publish": false,
"response": {
"result": {
"error": "'frame'"
},
"status": "application error",
"success": false
},
"start": 1560669648858,
"subject": "{my_email}",
"version": "0.0.7"
}
EDIT 2:
I've also tried to enable it as a web action to see if it changes anything. However, it's no use. When I use this HTTP request:
https://us-south.functions.cloud.ibm.com/api/v1/web/{namespace}/default/{action_name}?frame={value}
I get:
{
"code": "e1c36666f4db1884c48f028ef58243fc",
"error": "Response is not valid 'message/http'."
}
which is understandable since what my functions returns is json. However, when I use this HTTP request:
https://us-south.functions.cloud.ibm.com/api/v1/web/{namespace}/default/{action_name}.json?frame={value}
I get:
{
"code": "010fc0efaa29f96b47f92735ff763f50",
"error": "Response is not valid 'application/json'."
}
I really don't know what to do here
After googling a bit I found something that works for me right now although it might not work for everyone. Apache has a python "client" example for using an action's REST API which uses the requests library.
Thing is that in order to use it you need to provide your API KEY, which I don't know how to get by any other means than getting it directly from the IBM Cloud CLI. Since I'm trying to access the function from a web server, I would need to save the key in an environment variable or save it in a text file and access it from there or install the CLI on the server, login with my credentials and call ibmcloud wsk property get --auth.
Also, this method didn't work with the web action endpoint when I tried it.

Error on testing AWS Lambda handler function: Data format for event and context parameters

I have the following code from a blog which gets the bitcoin price for today. I could access this Lambda function from the AWS Lex console and test the bot to get the price for today.
"""
Lexbot Lambda handler.
"""
from urllib.request import Request, urlopen
import json
def get_bitcoin_price(date):
print('get_bitcoin_price, date = ' + str(date))
request = Request('https://rest.coinapi.io/v1/ohlcv/BITSTAMP_SPOT_BTC_USD/latest?period_id=1DAY&limit=1&time_start={}'.format(date))
request.add_header('X-CoinAPI-Key', 'E4107FA4-A508-448A-XXX')
response = json.loads(urlopen(request).read())
return response[0]['price_close']
def lambda_handler(event, context):
print('received request: ' + str(event))
date_input = event['currentIntent']['slots']['Date']
btc_price = get_bitcoin_price(date_input)
response = {
"dialogAction": {
"type": "Close",
"fulfillmentState": "Fulfilled",
"message": {
"contentType": "SSML",
"content": "Bitcoin's price was {price} dollars".format(price=btc_price)
},
}
}
print('result = ' + str(response))
return response
But when I test the function from the AWS Lex console, I get the following error:
Response:
{
"errorMessage": "'currentIntent'",
"errorType": "KeyError",
"stackTrace": [
[
"/var/task/lambda_function.py",
18,
"lambda_handler",
"date_input = event['currentIntent']['slots']['Date']"
]
]
}
Request ID:
"2488187a-2b76-47ba-b884-b8aae7e7a25d"
Function Logs:
START RequestId: 2488187a-2b76-47ba-b884-b8aae7e7a25d Version: $LATEST
received request: {'Date': 'Feb 22'}
'currentIntent': KeyError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 18, in lambda_handler
date_input = event['currentIntent']['slots']['Date']
KeyError: 'currentIntent'
How do I test the function in the AWS Lambda console? 'lambda_handler' function, what format would be the 'event' and 'context' be? Also, what would be the 'context' here?
What should I pass as 'event' and 'context' in my case?
Your code is failing because the event object is filled in with {'Date': 'Feb 22'} but your code expects much more than this. Therefore, it fails when you try to parse this JSON by trying to access currentIntent:
date_input = event['currentIntent']['slots']['Date']
You cannot pass any context to your Lambda when testing from the console as it is automatically populated by AWS. Also, the context is only used in very specific occasions, so I would not worry about it for now.
You can, however, pass the event as argument and there are many ways to do it. The simplest way to do it manually is to go to AWS's Lambda Console, click on Test and, if you haven't configured any Test Event yet, the following screen will pop up
Now, on the dropdown, you can select your event and AWS will fill it in for you, like this:
You can now customise the event the way you want it.
Once you save it and click on Test, the event object will be populated with the provided JSON.
Another option is to check Sample Events Published By Event Sources, so you can simply grab any JSON event you'd like and tailor it accordingly.
I have grabbed the Lex Sample Event for you, which looks like this:
{
"messageVersion": "1.0",
"invocationSource": "FulfillmentCodeHook or DialogCodeHook",
"userId": "user-id specified in the POST request to Amazon Lex.",
"sessionAttributes": {
"key1": "value1",
"key2": "value2",
},
"bot": {
"name": "bot-name",
"alias": "bot-alias",
"version": "bot-version"
},
"outputDialogMode": "Text or Voice, based on ContentType request header in runtime API request",
"currentIntent": {
"name": "intent-name",
"slots": {
"slot-name": "value",
"slot-name": "value",
"slot-name": "value"
},
"confirmationStatus": "None, Confirmed, or Denied
(intent confirmation, if configured)"
}
}
Use that as your event and you'll be able to test it accordingly.

Resources