Error on deploying Python app to AWS Lambda - python-3.x

I have built a Python-Tornado app and am trying to deploy it to AWS Lambda using zappa. But, I am getting an error Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 502 response code.
My folder structure inside the root folder is :
├── amortization.py
├── config.py
├── dmi-amort-dev-1557138776.zip
├── main.py
├── requirements.txt
├── venv
│   ├── bin
│  
└── zappa_settings.json
zappa deploy dev gives me :
Calling deploy for stage dev..
Downloading and installing dependencies..
- pandas==0.24.2: Using locally cached manylinux wheel
- numpy==1.16.3: Using locally cached manylinux wheel
- sqlite==python3: Using precompiled lambda package
Packaging project as zip.
Uploading dmi-amort-dev-1557143681.zip (30.8MiB)..
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32.3M/32.3M [00:19<00:00, 1.94MB/s]
Scheduling..
Scheduled dmi-amort-dev-zappa-keep-warm-handler.keep_warm_callback with expression rate(4 minutes)!
Uploading dmi-amort-dev-template-1557143718.json (1.5KiB)..
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.56K/1.56K [00:00<00:00, 10.6KB/s]
Waiting for stack dmi-amort-dev to create (this can take a bit)..
75%|█████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 3/4 [00:09<00:04, 5.00s/res]
Deploying API Gateway..
Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 502 response code.
zappa tail gives me
Traceback (most recent call last):
File "/var/task/handler.py", line 602, in lambda_handler
return LambdaHandler.lambda_handler(event, context)
File "/var/task/handler.py", line 245, in lambda_handler
handler = cls()
File "/var/task/handler.py", line 142, in __init__
wsgi_app_function = getattr(self.app_module, self.settings.APP_FUNCTION)
AttributeError: module 'main' has no attribute 'app'
zappa_settings.json:
{
"dev": {
"app_function": "main.app",
"aws_region": "ap-south-1",
"profile_name": "default",
"project_name": "dmi-amort",
"runtime": "python3.6",
"s3_bucket": "zappa-mekp457ye",
"manage_roles": false,
"role_name": "lambda-role",
}
}
main.py:
import tornado.web
from tornado.ioloop import IOLoop
from tornado.web import MissingArgumentError
from config import get_arguments
from amortization import get_amort_schedule
class MainHandler(tornado.web.RequestHandler):
def prepare(self):
"""Checking if all the required parameters are present."""
if self.request.method != 'POST':
self.write_error(status_code=405, message="Method not allowed")
return
self.parameters = dict()
for key in get_arguments():
try:
self.parameters[key] = self.get_argument(key)
except MissingArgumentError:
self.write_error(status_code=400,
message="Missing Argument(s)")
return
# checking if 'label' is provided
if 'label' in self.request.arguments.keys():
self.parameters['label'] = self.get_argument('label')
# Set up response dictionary.
self.response = dict()
def get(self, *args, **kwargs):
self.write_error(status_code=405, message="Method not allowed")
def post(self, *args, **kwargs):
"""Executes the main logic part."""
self.response = get_amort_schedule(self.parameters)
self.write_json()
def set_default_headers(self):
"""Sets content-type as 'application/json' for response as JSON."""
self.set_header('Content-Type', 'application/json')
def write_error(self, status_code, **kwargs):
"""Invokes when error occurs in processing the request."""
if 'message' not in kwargs:
if status_code == 405:
kwargs['message'] = 'Invalid HTTP method.'
else:
kwargs['message'] = 'Unknown error.'
kwargs["error"] = True
self.set_status(status_code=status_code)
self.response = dict(kwargs)
self.write_json()
def write_json(self):
"""Responsible for writing the response."""
if "status" in self.response:
self.set_status(self.response.get("status"))
self.set_default_headers()
self.write(self.response)
self.finish()
def main():
app = tornado.web.Application([
(r'/', MainHandler),
], debug=True)
# server = HTTPServer(app)
# server.bind(8888)
# server.start(0)
app.listen()
# app.run(host='0.0.0.0')
IOLoop.current().start()
if __name__ == '__main__':
main()
What is the mistake here and how can I fix it?

It looks like the deployment is succeeding, but when Zappa checks to see if the code is working, the return code is 502, which suggests that the lambda function is failing to run in the lambda environment.
Taking a look at the logs, the critical line is:
AttributeError: module 'main' has no attribute 'app'
And this is true, if we look at your code, at no point do you expose an attribute called app in main.py.
I'm not experienced with Tornado, but I suspect that if you move the declaration of app out of the main() function and into the root scope, then the handler should succeed.
For example:
# rest of the file...
self.finish()
app = tornado.web.Application([
(r'/', MainHandler),
], debug=True)
app.listen()
def main():
IOLoop.current().start()
if __name__ == '__main__':
main()

Related

Error 500 when trying to send file from remote server (python code) to an SFTP server

I have a Flask API in a Docker container where I do the following :
from flask import Flask, request
import os
import json
import paramiko
import subprocess
app = Flask(__name__)
#app.route("/")
def hello():
return "Service up in K8S!"
#app.route("/get", methods=['GET'])
def get_ano():
print("Test liveness")
return "Pod is alive !"
#app.route("/run", methods=['POST'])
def run_dump_generation():
rules_str = request.headers.get('database')
print(rules_str)
postgres_bin = r"/usr/bin/"
dump_file = "database_dump.sql"
os.environ['PGPASSWORD'] = 'XXXXX'
print('Before dump generation')
with open(dump_file, "w") as f:
result = subprocess.call([
os.path.join(postgres_bin, "pg_dump"),
"-Fp",
"-d",
"XX",
"-U",
"XX",
"-h",
"XX",
"-p",
"XX"
],
stdout=f
)
print('After dump generation')
transport = paramiko.Transport(("X", X))
transport.connect(username="X", password="X")
sftp = transport.open_sftp_client()
remote_file = '/data/database_dump.sql'
sftp.put('database_dump.sql', remote_file)
print("SFTP object", sftp)
if __name__ == "__main__":
app.run(host='0.0.0.0', debug=True)
When I run the app in Kubernetes with Post request I have the error : "POST /run HTTP/1.1" 500
Here are the requirements.txt:
Flask==2.0.1
paramiko==3.0.0
The error comes from transport = paramiko.Transport(("X", X)). The same code works locally. I don't understand why I have this error when I am on Kubernetes. But in the logs no print are displaying, I guess it is because I have error 500. I guess it is not possible with this code to send file from this container to the SFTP server (has OpenSSH).
What can I do ?
---- UPDATE ----
I think I have found the problem. In a Flask pod (VM) I try to send file from this pod to SFTP server. So I have to modify the following code to "allow" this type of send. This an SFTP server with OpenSSH.
Here is the code where to modify :
transport = paramiko.Transport(("X", X))
transport.connect(username="X", password="X")
sftp = transport.open_sftp_client()
remote_file = '/data/database_dump.sql'
sftp.put('database_dump.sql', remote_file)
print("SFTP object", sftp)
SFTP server (with OpenSSH) is alpine and Flask code is in Alpine container too.
UPDATE BELOW, I tried the following :
ssh=paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
session = ssh.connect(hostname="X", port=X, username='X', password="X")
print(ssh)
But I have the following error :
File "c:/Users/X/dump_generator_api/t.py", line 32, in <module>
session = ssh.connect(hostname="X", port=X, username='X', password="X")
File "C:\Users\X\AppData\Local\Programs\Python\Python38-32\lib\site-packages\paramiko\client.py", line 449, in connect
self._auth(
File "C:\Users\X\AppData\Local\Programs\Python\Python38-32\lib\site-packages\paramiko\client.py", line 780, in _auth
raise saved_exception
File "C:\Users\X\AppData\Local\Programs\Python\Python38-32\lib\site-packages\paramiko\client.py", line 767, in _auth
self._transport.auth_password(username, password)
File "C:\Users\X\AppData\Local\Programs\Python\Python38-32\lib\site-packages\paramiko\transport.py", line 1567, in auth_password
return self.auth_handler.wait_for_response(my_event)
File "C:\Users\X\AppData\Local\Programs\Python\Python38-32\lib\site-packages\paramiko\auth_handler.py", line 259, in wait_for_response
raise e
paramiko.ssh_exception.AuthenticationException: Authentication failed.

azure function app python module import failure

I'm not able to execute my cloud function due to failures to import modules, module azure-identity for example, it's telling me module not found, I had my requirements.txt file:
msrestazure
azure-identity
azure.keyvault.secrets
azure-cli
and my code imports:
import logging
import json
import base64
import azure.functions as func
import requests
from azure.identity import AzureCliCredential
from azure.mgmt.resource import ResourceManagementClient
from azure.mgmt.network import NetworkManagementClient
from azure.mgmt.compute import ComputeManagementClient
import os
Result: Failure Exception: ModuleNotFoundError: No module named 'azure.identity'. Troubleshooting Guide: https://aka.ms/functions-modulenotfound Stack: File "/azure-functions-host/workers/python/3.6/LINUX/X64/azure_functions_worker/dispatcher.py", line 318, in _handle__function_load_request func_request.metadata.entry_point) File "/azure-functions-host/workers/python/3.6/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 42, in call raise extend_exception_message(e, message) File "/azure-functions-host/workers/python/3.6/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 40, in call return func( * args, ** kwargs) File "/azure-functions-host/workers/python/3.6/LINUX/X64/azure_functions_worker/loader.py", line 85, in load_function mod = importlib.import_module(fullmodname) File "/usr/local/lib/python3.6/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/home/site/wwwroot/dworkerfn/init.py", line 6, in from azure.identity import AzureCliCredential**
My files hierarchy:
├── dworkerfn
│   ├── function.json
│   ├── __init__.py
│   ├── launch-worker.sh
│   └── sample.dat
├── host.json
└── requirements.txt
This issue is not related to the module itself, I tried with different modules, requests for example and always the same issue.

Unable to render template with flask wsgi [duplicate]

This question already has answers here:
Flask raises TemplateNotFound error even though template file exists
(13 answers)
Closed 2 years ago.
I have deployed a flask application, using mod_wsgi with apache2
Now, checking if everything works as intended, it looks like my endpoints which render templates are provoking some 500 status code errors.
Here's an approximative tree of my project:
main_folder
requirements.txt
mainfile.wsgi
app_folder
controllers
views.py
models
repository
services
static
templates
terms
en
terms.html
fr
terms.html
uploads
__init__.py
config.cfg
webapp.py
I am calling an endpoint which is supposed to render the templates/terms/en/terms.html and I get the following in the logs:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/dist-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/var/www/main_folder/app_folder/controllers/views.py", line 33, in gts
return render_template('terms/' + lang + '/terms.html')
File "/usr/local/lib/python3.7/dist-packages/flask/templating.py", line 138, in render_template
ctx.app.jinja_env.get_or_select_template(template_name_or_list),
File "/usr/local/lib/python3.7/dist-packages/jinja2/environment.py", line 869, in get_or_select_template
return self.get_template(template_name_or_list, parent, globals)
File "/usr/local/lib/python3.7/dist-packages/jinja2/environment.py", line 830, in get_template
return self._load_template(name, self.make_globals(globals))
File "/usr/local/lib/python3.7/dist-packages/jinja2/environment.py", line 804, in _load_template
template = self.loader.load(self, name, globals)
File "/usr/local/lib/python3.7/dist-packages/jinja2/loaders.py", line 113, in load
source, filename, uptodate = self.get_source(environment, name)
File "/usr/local/lib/python3.7/dist-packages/flask/templating.py", line 60, in get_source
return self._get_source_fast(environment, template)
File "/usr/local/lib/python3.7/dist-packages/flask/templating.py", line 89, in _get_source_fast
raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: terms/en/terms.html
Here's an excerp of the code where I try to render the template in app_folder/controllers/views.py:
from flask import render_template, request, send_file
from app_folder.controllers import Controllers as controllers
def gts(lang='fr'):
return render_template('terms/' + lang + '/terms.html')
controllers().register(
'/cgu/<string:lang>',
'cgu_lang',
gts
)
I want to specify that endpoints with JSON body results are functionning, uploading ressources are working as well, and getting an uploaded ressource is working.
Only endpoints which should render templates are not working, which is getting me worried as well for endpoints which should send emails using an email template
Is there any fix I should apply when serving the application with wsgi?
Edit: FYI, this is the __init__.py in the controllers folder
import pkgutil
METHODS = set([
'GET',
'POST',
'PUT',
'PATCH',
'DELETE',
'COPY',
'HEAD',
'OPTIONS',
'LINK',
'UNLINK',
'PURGE',
'LOCK',
'UNLOCK',
'PROPFIND',
'VIEW'
])
class Controllers:
class __OnlyOne:
# ======================================================
# the code goes here
# =======================================================
def __init__(self):
self.val = None
self.rules = []
def __str__(self):
return 'self' + self.val
def register(self, rule, view_name, view_func):
r = {}
r["rule"] = rule
r["view_name"] = view_name
r["view_func"] = view_func
self.rules.append(r)
def register_methods(self, rule, view_name, view_func, methods=METHODS):
r = {}
r["rule"] = rule
r["view_name"] = view_name
r["view_func"] = view_func
r["methods"] = methods
self.rules.append(r)
def grab(self, app):
for r in self.rules:
if "methods" not in r.keys():
app.add_url_rule(r["rule"], r["view_name"], r["view_func"])
else:
app.add_url_rule(r["rule"], r["view_name"],
r["view_func"], methods=r["methods"])
# =================================================================
# the code goes up there
# ==================================================================
instance = None
def __new__(cls): # __new__ always a classmethod
if not Controllers.instance:
Controllers.instance = Controllers.__OnlyOne()
return Controllers.instance
def __getattr__(self, name):
return getattr(self.instance, name)
def __setattr__(self, name):
return setattr(self.instance, name)
# import all the modules in folder
__all__ = []
for loader, module_name, is_pkg in pkgutil.walk_packages(__path__):
__all__.append(module_name)
_module = loader.find_module(module_name).load_module(module_name)
globals()[module_name] = _module
I symlinked my app_folder/templates into my main_folder and it's now working

The run time error by import python module

I am trying to make correctly import the function "mmenu" from another module
I got a run time error:
Traceback (most recent call last):
File "path.../venv/src/routes.py", line 4, in <module>
from venv.src.main_menu import mmenu
ModuleNotFoundError: No module named 'venv.src'
after Java this Python is a bit of a mystery to me :)
I have 2 files in the same "src" directory
main_menu.py
def mmenu():
menu = [{"name": "HOME", "url": "home"},
{"name": "fooo", "url": "foo"},
{"name": "bar", "url": "bar"},
{"name": "CONTACT", "url": "contact"}]
return menu
routes.py
from flask import Flask, render_template, request, flash, session, redirect, abort
from jinja2 import Template
from flask.helpers import url_for
from venv.src.main_menu import mmenu
app = Flask(__name__)
menu = mmenu
#app.route("/")
#app.route("/index")
def index():
# return "index"
print("loaded" + url_for('index'))
return render_template('index.html', title="Index page", menu=menu)
# ...
if __name__ == "__main__":
app.run(debug=True)
import was generated by IDE like:
from venv.src.parts.main_menu import mmenu
Your IDE (which IDE?) is likely misconfigured if it generates imports like that.
The venv itself, nor any src directory within them, shouldn't be within import paths.
You never mention you have a parts/ package, but from the import I'll assume you do, and your structure is something like
venv/
src/ # source root
routes.py
parts/ # parts package
__init__.py # (empty) init for parts package
main_menu.py
With this structure, main_menu is importable as parts.main_menu from routes.py assuming you start the program with python routes.py.
If you have __init__.py files in venv/ and/or src/, get rid of them; you don't want those directories to be assumed to be Python packages.

Flask Deployment to AWS getting permission errno 13

I have working app on the localhost but when I deployed to the AWS it shows several errors.It can receive my sent image but it cant save it. I tried change permissions with chmod,still its not working. This is first time i am deploying flask app,so I dont know much about i My code is as follow:
***modules***
ImageFile.LOAD_TRUNCATED_IMAGES = True
app = flask.Flask(__name__)
#app.route('/', methods = ['GET','POST'])
def hello():
return 'hey there!!!'
#app.route('/imageupload', methods = ['GET','POST'])
def handle_request():
try:
imagefile = flask.request.files['image']
filename = werkzeug.utils.secure_filename(imagefile.filename)
print("\nReceived image file name:" + imagefile.filename)
imagefile.save(filename)
print("hey")
for image in glob.glob('./frames/*.*'):
img_filter(imagefile,image)
json = flask.request.values['Id']
return {"status": "true","message": "Uploaded Successfully", "Id": json }
except:
return {"status" : "false"}
if __name__ == '__main__' :
app.run()
And these are the errors :
Received image file name:1.jpg
ERROR:image_flask:Exception on /imageupload [POST]
Traceback (most recent call last):
File "/home/ubuntu/pic-edit/env/lib/python3.6/site-packages/flask/app.py", line 2446, in ws$
response = self.full_dispatch_request()
File "/home/ubuntu/pic-edit/env/lib/python3.6/site-packages/flask/app.py", line 1951, in fu$
rv = self.handle_user_exception(e)
File "/home/ubuntu/pic-edit/env/lib/python3.6/site-packages/flask/app.py", line 1820, in ha$
reraise(exc_type, exc_value, tb)
File "/home/ubuntu/pic-edit/env/lib/python3.6/site-packages/flask/_compat.py", line 39, in $
raise value
File "/home/ubuntu/pic-edit/env/lib/python3.6/site-packages/flask/app.py", line 1949, in fu$
rv = self.dispatch_request()
File "/home/ubuntu/pic-edit/env/lib/python3.6/site-packages/flask/app.py", line 1935, in di$
return self.view_functions[rule.endpoint](**req.view_args)
File "/var/www/html/pic-edit/image_flask.py", line 26, in handle_request
imagefile.save(f'/home/ubuntu/pic-edit/{filename}')
File "/home/ubuntu/pic-edit/env/lib/python3.6/site-packages/werkzeug/datastructures.py", li$
dst = open(dst, "wb")
PermissionError: [Errno 13] Permission denied: '/home/ubuntu/pic-edit/1.jpg'
you don't have the right permission. Try to open first the folder with chmod -R 700 /home/ubuntu/pic-edit/
Please note open the permissions like this can be dangerous. Rather I would assign the permission to the "Flask" - Running user
Further for investigations under which user are you running the flask app? How do you start the app?
Also you might change the handle func to:
def handle_request():
try:
f = request.files['file']
f.save(secure_filename(f.filename))
json = flask.request.values['Id']
return {"status": "true","message": "Uploaded Successfully", "Id": json }
note you nead to import from flask import request
I got it where I went wrong, that is because of permissions to the folder. I already tried "chmod 755" to it but it did not worked. After that I tried
chmod a+rwx
And that work fluently.

Resources