I have a AWS Lambda function based on python 3.7 and trying to use a module dicttoxml via AWS layers. My Python code is as below:
import json
import dicttoxml
def lambda_handler(event, context):
xml = dicttoxml.dicttoxml({"name": "Foo"})
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}
At my local machine, it works perfectly fine but Lambda gives error as below:
{
"errorMessage": "module 'dicttoxml' has no attribute 'dicttoxml'",
"errorType": "AttributeError",
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 4, in lambda_handler\n xml = dicttoxml.dicttoxml({\"name\": \"Ankur\"})\n"
]
}
The directory structure of dicttoxml layer is as below:
dicttoxml.zip > python > dicttoxml > dicttoxml.py
I feel puzzled, what is wrong here?
I created the custom layer with dicttoxml can confirm that it works.
The technique used includes docker tool described in the recent AWS blog:
How do I create a Lambda layer using a simulated Lambda environment with Docker?
Thus for this question, I verified it as follows:
Create empty folder, e.g. mylayer.
Go to the folder and create requirements.txt file with the content of
echo dicttoxml > ./requirements.txt
Run the following docker command:
docker run -v "$PWD":/var/task "lambci/lambda:build-python3.7" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.7/site-packages/; exit"
Create layer as zip:
zip -9 -r mylayer.zip python
Create lambda layer based on mylayer.zip in the AWS Console. Don't forget to specify Compatible runtimes to python3.7.
Test the layer in lambda using the following lambda function:
import dicttoxml
def lambda_handler(event, context):
print(dir(dicttoxml))
The function executes correctly:
['LOG', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '__version__', 'collections', 'convert', 'convert_bool', 'convert_dict', 'convert_kv', 'convert_list', 'convert_none', 'default_item_func', 'dicttoxml', 'escape_xml', 'get_unique_id', 'get_xml_type', 'ids', 'key_is_valid_xml', 'logging', 'long', 'make_attrstring', 'make_id', 'make_valid_xml_name', 'numbers', 'parseString', 'randint', 'set_debug', 'unicode', 'unicode_literals', 'unicode_me', 'version', 'wrap_cdata']
Related
I am trying to write an ansible playbook to crawl a website and then store its contents into a static file under aws s3 bucket. Here is the crawler code :
"""
Handling pages with the Next button
"""
import sys
from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup
url = "https://xyz.co.uk/"
file_name = "web_content.txt"
while True:
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
raw_html = soup.prettify()
file = open(file_name, 'wb')
print('Collecting the website contents')
file.write(raw_html.encode())
file.close()
print('Saved to %s' % file_name)
#print(type(raw_html))
# Finding next page
next_page_element = soup.select_one('li.next > a')
if next_page_element:
next_page_url = next_page_element.get('href')
url = urljoin(url, next_page_url)
else:
break
This is my ansible-playbook:
---
- name: create s3 bucket and upload static website content into it
hosts: localhost
connection: local
tasks:
- name: create a s3 bucket
amazon.aws.aws_s3:
bucket: testbucket393647914679149
region: ap-south-1
mode: create
- name: create a folder in the bucket
amazon.aws.aws_s3:
bucket: testbucket393647914679149
object: /my/directory/path
mode: create
- name: Upgrade pip
pip:
name: pip
version: 21.1.3
- name: install virtualenv via pip
pip:
requirements: /root/ansible/requirements.txt
virtualenv: /root/ansible/myvenv
virtualenv_python: python3.6
environment:
PATH: "{{ ansible_env.PATH }}:{{ ansible_user_dir }}/.local/bin"
- name: Run script to crawl the website
script: /root/ansible/beautiful_crawl.py
- name: copy file into bucket folder
amazon.aws.aws_s3:
bucket: testbucket393647914679149
object: /my/directory/path/web_content.text
src: web_content.text
mode: put
Problem is when I run this, it runs fine upto the task name: install virtualenv via pip and then throws following error while executing the task name: Run script to crawl the website:
fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 2, "stderr": "/root/.ansible/tmp/ansible-tmp-1625137700.8854306-13026-9798 3643645466/beautiful_crawl.py: line 1: import: command not found\n/root/.ansible /tmp/ansible-tmp-1625137700.8854306-13026-97983643645466/beautiful_crawl.py: lin e 2: from: command not found\n/root/.ansible/tmp/ansible-tmp-1625137700.8854306- 13026-97983643645466/beautiful_crawl.py: line 3: import: command not found\n/roo t/.ansible/tmp/ansible-tmp-1625137700.8854306-13026-97983643645466/beautiful_cra wl.py: line 4: from: command not found\n/root/.ansible/tmp/ansible-tmp-162513770 0.8854306-13026-97983643645466/beautiful_crawl.py: line 6: url: command not foun d\n/root/.ansible/tmp/ansible-tmp-1625137700.8854306-13026-97983643645466/beauti ful_crawl.py: line 7: file_name: command not found\n/root/.ansible/tmp/ansible-t mp-1625137700.8854306-13026-97983643645466/beautiful_crawl.py: line 10: syntax e rror near unexpected token ('\n/root/.ansible/tmp/ansible-tmp-1625137700.885430 6-13026-97983643645466/beautiful_crawl.py: line 10: response = requests.get (url)'\n", "stderr_lines": ["/root/.ansible/tmp/ansible-tmp-1625137700.8854306-1 3026-97983643645466/beautiful_crawl.py: line 1: import: command not found", "/ro ot/.ansible/tmp/ansible-tmp-1625137700.8854306-13026-97983643645466/beautiful_cr awl.py: line 2: from: command not found", "/root/.ansible/tmp/ansible-tmp-162513 7700.8854306-13026-97983643645466/beautiful_crawl.py: line 3: import: command no t found", "/root/.ansible/tmp/ansible-tmp-1625137700.8854306-13026-9798364364546 6/beautiful_crawl.py: line 4: from: command not found", "/root/.ansible/tmp/ansi ble-tmp-1625137700.8854306-13026-97983643645466/beautiful_crawl.py: line 6: url: command not found", "/root/.ansible/tmp/ansible-tmp-1625137700.8854306-13026-97 983643645466/beautiful_crawl.py: line 7: file_name: command not found", "/root/. ansible/tmp/ansible-tmp-1625137700.8854306-13026-97983643645466/beautiful_crawl. py: line 10: syntax error near unexpected token ('", "/root/.ansible/tmp/ansibl e-tmp-1625137700.8854306-13026-97983643645466/beautiful_crawl.py: line 10: response = requests.get(url)'"], "stdout": "", "stdout_lines": []}
What am I doing wrong here?
You have multiple problems.
Check the documentation.
No. 1: The script modules will run bash scripts by default, not python scripts. If you want to run a python script, you need to add a shebang like #!/usr/bin/env python3 as the first line of the script or use the executable parameter.
No 2: You create a venv, so I assume you want to run the script in that venv. You can't do that out of the box with the script module, so you would need to work around that.
This should work for you (you don't need the shebang, as you tell the script module to run it with python in the venv using the executable parameter):
- name: Run script to crawl the website
script: /root/ansible/beautiful_crawl.py
executable: /root/ansible/myvenv/bin/python
I am trying to make correctly import the function "mmenu" from another module
I got a run time error:
Traceback (most recent call last):
File "path.../venv/src/routes.py", line 4, in <module>
from venv.src.main_menu import mmenu
ModuleNotFoundError: No module named 'venv.src'
after Java this Python is a bit of a mystery to me :)
I have 2 files in the same "src" directory
main_menu.py
def mmenu():
menu = [{"name": "HOME", "url": "home"},
{"name": "fooo", "url": "foo"},
{"name": "bar", "url": "bar"},
{"name": "CONTACT", "url": "contact"}]
return menu
routes.py
from flask import Flask, render_template, request, flash, session, redirect, abort
from jinja2 import Template
from flask.helpers import url_for
from venv.src.main_menu import mmenu
app = Flask(__name__)
menu = mmenu
#app.route("/")
#app.route("/index")
def index():
# return "index"
print("loaded" + url_for('index'))
return render_template('index.html', title="Index page", menu=menu)
# ...
if __name__ == "__main__":
app.run(debug=True)
import was generated by IDE like:
from venv.src.parts.main_menu import mmenu
Your IDE (which IDE?) is likely misconfigured if it generates imports like that.
The venv itself, nor any src directory within them, shouldn't be within import paths.
You never mention you have a parts/ package, but from the import I'll assume you do, and your structure is something like
venv/
src/ # source root
routes.py
parts/ # parts package
__init__.py # (empty) init for parts package
main_menu.py
With this structure, main_menu is importable as parts.main_menu from routes.py assuming you start the program with python routes.py.
If you have __init__.py files in venv/ and/or src/, get rid of them; you don't want those directories to be assumed to be Python packages.
I am trying to write a small program using the AzureML Python SDK (v1.0.85) to register an Environment in AMLS and use that definition to construct a local Conda environment when experiments are being run (for a pre-trained model). The code works fine for simple scenarios where all dependencies are loaded from Conda/ public PyPI, but when I introduce a private dependency (e.g. a utils library) I am getting a InternalServerError with the message "Error getting recipe specifications".
The code I am using to register the environment is (after having authenticated to Azure and connected to our workspace):
environment_name = config['environment']['name']
py_version = "3.7"
conda_packages = ["pip"]
pip_packages = ["azureml-defaults"]
private_packages = ["./env-wheels/utils-0.0.3-py3-none-any.whl"]
print(f"Creating environment with name {environment_name}")
environment = Environment(name=environment_name)
conda_deps = CondaDependencies()
print(f"Adding Python version: {py_version}")
conda_deps.set_python_version(py_version)
for conda_pkg in conda_packages:
print(f"Adding Conda denpendency: {conda_pkg}")
conda_deps.add_conda_package(conda_pkg)
for pip_pkg in pip_packages:
print(f"Adding Pip dependency: {pip_pkg}")
conda_deps.add_pip_package(pip_pkg)
for private_pkg in private_packages:
print(f"Uploading private wheel from {private_pkg}")
private_pkg_url = Environment.add_private_pip_wheel(workspace=ws, file_path=Path(private_pkg).absolute(), exist_ok=True)
print(f"Adding private Pip dependency: {private_pkg_url}")
conda_deps.add_pip_package(private_pkg_url)
environment.python.conda_dependencies = conda_deps
environment.register(workspace=ws)
And the code I am using to create the local Conda environment is:
amls_environment = Environment.get(ws, name=environment_name, version=environment_version)
print(f"Building environment...")
amls_environment.build_local(workspace=ws)
The exact error message being returned when build_local(...) is called is:
Traceback (most recent call last):
File "C:\Anaconda\envs\AMLSExperiment\lib\site-packages\azureml\core\environment.py", line 814, in build_local
raise error
File "C:\Anaconda\envs\AMLSExperiment\lib\site-packages\azureml\core\environment.py", line 807, in build_local
recipe = environment_client._get_recipe_for_build(name=self.name, version=self.version, **payload)
File "C:\Anaconda\envs\AMLSExperiment\lib\site-packages\azureml\_restclient\environment_client.py", line 171, in _get_recipe_for_build
raise Exception(message)
Exception: Error getting recipe specifications. Code: 500
: {
"error": {
"code": "ServiceError",
"message": "InternalServerError",
"detailsUri": null,
"target": null,
"details": [],
"innerError": null,
"debugInfo": null
},
"correlation": {
"operation": "15043e1469e85a4c96a3c18c45a2af67",
"request": "19231be75a2b8192"
},
"environment": "westeurope",
"location": "westeurope",
"time": "2020-02-28T09:38:47.8900715+00:00"
}
Process finished with exit code 1
Has anyone seen this error before or able to provide some guidance around what the issue may be?
The issue was with out firewall blocking the required requests between AMLS and the storage container (I presume to get the environment definitions/ private wheels).
We resolved this by updating the firewall with appropriate ALLOW rules for the AMLS service to contact and read from the attached storage container.
Assuming that you'd like to run in the script on a remote compute, then my suggestion would be to pass the environment you just "got". to a RunConfiguration, then pass that to an ScriptRunConfig, Estimator, or a PythonScriptStep
from azureml.core import ScriptRunConfig
from azureml.core.runconfig import DEFAULT_CPU_IMAGE
src = ScriptRunConfig(source_directory=project_folder, script='train.py')
# Set compute target to the one created in previous step
src.run_config.target = cpu_cluster.name
# Set environment
amls_environment = Environment.get(ws, name=environment_name, version=environment_version)
src.run_config.environment = amls_environment
run = experiment.submit(config=src)
run
Check out the rest of the notebook here.
If you're looking for a local run this notebook might help.
I have built a Python-Tornado app and am trying to deploy it to AWS Lambda using zappa. But, I am getting an error Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 502 response code.
My folder structure inside the root folder is :
├── amortization.py
├── config.py
├── dmi-amort-dev-1557138776.zip
├── main.py
├── requirements.txt
├── venv
│ ├── bin
│
└── zappa_settings.json
zappa deploy dev gives me :
Calling deploy for stage dev..
Downloading and installing dependencies..
- pandas==0.24.2: Using locally cached manylinux wheel
- numpy==1.16.3: Using locally cached manylinux wheel
- sqlite==python3: Using precompiled lambda package
Packaging project as zip.
Uploading dmi-amort-dev-1557143681.zip (30.8MiB)..
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32.3M/32.3M [00:19<00:00, 1.94MB/s]
Scheduling..
Scheduled dmi-amort-dev-zappa-keep-warm-handler.keep_warm_callback with expression rate(4 minutes)!
Uploading dmi-amort-dev-template-1557143718.json (1.5KiB)..
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.56K/1.56K [00:00<00:00, 10.6KB/s]
Waiting for stack dmi-amort-dev to create (this can take a bit)..
75%|█████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 3/4 [00:09<00:04, 5.00s/res]
Deploying API Gateway..
Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 502 response code.
zappa tail gives me
Traceback (most recent call last):
File "/var/task/handler.py", line 602, in lambda_handler
return LambdaHandler.lambda_handler(event, context)
File "/var/task/handler.py", line 245, in lambda_handler
handler = cls()
File "/var/task/handler.py", line 142, in __init__
wsgi_app_function = getattr(self.app_module, self.settings.APP_FUNCTION)
AttributeError: module 'main' has no attribute 'app'
zappa_settings.json:
{
"dev": {
"app_function": "main.app",
"aws_region": "ap-south-1",
"profile_name": "default",
"project_name": "dmi-amort",
"runtime": "python3.6",
"s3_bucket": "zappa-mekp457ye",
"manage_roles": false,
"role_name": "lambda-role",
}
}
main.py:
import tornado.web
from tornado.ioloop import IOLoop
from tornado.web import MissingArgumentError
from config import get_arguments
from amortization import get_amort_schedule
class MainHandler(tornado.web.RequestHandler):
def prepare(self):
"""Checking if all the required parameters are present."""
if self.request.method != 'POST':
self.write_error(status_code=405, message="Method not allowed")
return
self.parameters = dict()
for key in get_arguments():
try:
self.parameters[key] = self.get_argument(key)
except MissingArgumentError:
self.write_error(status_code=400,
message="Missing Argument(s)")
return
# checking if 'label' is provided
if 'label' in self.request.arguments.keys():
self.parameters['label'] = self.get_argument('label')
# Set up response dictionary.
self.response = dict()
def get(self, *args, **kwargs):
self.write_error(status_code=405, message="Method not allowed")
def post(self, *args, **kwargs):
"""Executes the main logic part."""
self.response = get_amort_schedule(self.parameters)
self.write_json()
def set_default_headers(self):
"""Sets content-type as 'application/json' for response as JSON."""
self.set_header('Content-Type', 'application/json')
def write_error(self, status_code, **kwargs):
"""Invokes when error occurs in processing the request."""
if 'message' not in kwargs:
if status_code == 405:
kwargs['message'] = 'Invalid HTTP method.'
else:
kwargs['message'] = 'Unknown error.'
kwargs["error"] = True
self.set_status(status_code=status_code)
self.response = dict(kwargs)
self.write_json()
def write_json(self):
"""Responsible for writing the response."""
if "status" in self.response:
self.set_status(self.response.get("status"))
self.set_default_headers()
self.write(self.response)
self.finish()
def main():
app = tornado.web.Application([
(r'/', MainHandler),
], debug=True)
# server = HTTPServer(app)
# server.bind(8888)
# server.start(0)
app.listen()
# app.run(host='0.0.0.0')
IOLoop.current().start()
if __name__ == '__main__':
main()
What is the mistake here and how can I fix it?
It looks like the deployment is succeeding, but when Zappa checks to see if the code is working, the return code is 502, which suggests that the lambda function is failing to run in the lambda environment.
Taking a look at the logs, the critical line is:
AttributeError: module 'main' has no attribute 'app'
And this is true, if we look at your code, at no point do you expose an attribute called app in main.py.
I'm not experienced with Tornado, but I suspect that if you move the declaration of app out of the main() function and into the root scope, then the handler should succeed.
For example:
# rest of the file...
self.finish()
app = tornado.web.Application([
(r'/', MainHandler),
], debug=True)
app.listen()
def main():
IOLoop.current().start()
if __name__ == '__main__':
main()
I set up a very simple project to try out Haystack using a Whoosh engine, mostly following the example in the documentation. I installed everything using pip and no version numbers, so I should have the latest release versions.
I'm getting this error and I have no idea what I'm supposed to do now, I cannot find anything similar though I've scoured Google. Please help!
The project folder structure is very simple, with one app called cat.
project
|-cat
| |-migrations
| |-admin.py, apps.py, models.py, search_indexes.py
|-templates
| |-search
| |-indexes
| | |-cat
| | |-cat_text.txt
| |-search.html
|-manage.py, settings.py, urls.py
The error I'm getting is:
Environment:
Request Method: GET
Request URL: http://localhost:8000/search/?q=felix&models=cat.cat
Django Version: 1.9.4
Python Version: 2.7.10
Installed Applications:
['django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'haystack',
'cat',]
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware']
Traceback:
File "/Users/lebouuski/projects/django/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
149. response = self.process_exception_by_middleware(e, request)
File "/Users/lebouuski/projects/django/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
147. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/Users/lebouuski/projects/django/lib/python2.7/site-packages/haystack/views.py" in __call__
51. self.results = self.get_results()
File "/Users/lebouuski/projects/django/lib/python2.7/site-packages/haystack/views.py" in get_results
91. return self.form.search()
File "/Users/lebouuski/projects/django/lib/python2.7/site-packages/haystack/forms.py" in search
116. return sqs.models(*self.get_models())
File "/Users/lebouuski/projects/django/lib/python2.7/site-packages/haystack/forms.py" in get_models
110. search_models.append(models.get_model(*model.split('.')))
Exception Type: AttributeError at /search/
Exception Value: 'module' object has no attribute 'get_model'
models.py:
class Cat(models.Model):
name = models.CharField(max_length=255)
birth_date = models.DateField(default=datetime.date.today)
bio = models.TextField(blank=True)
created = models.DateTimeField(default=datetime.datetime.now)
updated = models.DateTimeField(default=datetime.datetime.now)
def __unicode__(self):
return self.name
#models.permalink
def get_absolute_url(self):
return ('cat_detail', [], {'id': self.id})
search_indexes.py
class CatIndex(indexes.BasicSearchIndex, indexes.Indexable):
def get_model(self):
return Cat
For anyone else struggling with this, this is all due to a little change in Django 1.9, in which the get_model() method is moved to the django.apps.apps module, and is no longer available from django.db
The issue is resolved now in Haystack community, so updating to the newest version, from their GitHub repository (and not PyPI) must solve it.
The following should solve it:
pip install git+https://github.com/django-haystack/django-haystack.git
Or you can simply downgrade to Django 1.8 as Sam suggested.
You can find more over this issue here
Solution: downgrade Django from 1.9 to 1.8
sudo pip install Django==1.8
Motivations here: link