I am using the docker-py client to create containers on a need basis. So for this, I am using a generator to come up with a port number and trying to use httpd image on a particular port of the host from the generator. But, the client gives out ("invalid port specification: "port number here"") for any number, for any port number that I am trying to use.
Below is sample code that I am trying:
import docker
client = docker.from_env()
container= client.containers.run(image="httpd", ports={'80/tcp': 43545}, detach=True)
To note: The number 43545 does not have any significance here.
Docker details:
Client - 19.03.6
API Version - 1.40
Engine: 19.03.6
Error:
File "/project/api/.venv/lib/python3.7/site-packages/docker/models/containers.py", line 803, in run
detach=detach, **kwargs)
File "/project/api/.venv/lib/python3.7/site-packages/docker/models/containers.py", line 861, in create
resp = self.client.api.create_container(**create_kwargs)
File "/project/api/.venv/lib/python3.7/site-packages/docker/api/container.py", line 429, in create_container
return self.create_container_from_config(config, name)
File "/projectapi/.venv/lib/python3.7/site-packages/docker/api/container.py", line 440, in create_container_from_config
return self._result(res, True)
File "/projectpi/.venv/lib/python3.7/site-packages/docker/api/client.py", line 267, in _result
self._raise_for_status(response)
File "/projectapi/.venv/lib/python3.7/site-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/projectyter/api/.venv/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 400 Client Error: Bad Request ("invalid port specification: "43545"")
Related
I am building a simple API endpoint using AWS Lambda function and API Gateway. I am orchestrating those resources using SAM. I can deploy my function to the AWS cloud and access the endpoint. It's working as expected.
Here is my lambda function.
exports.lambdaHandler = async (event) => ({
statusCode: 200,
body: {
message: `Hello World!`
}
})
Here is my SAM CloudFormation template
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: HTTP API
Globals:
Function:
Timeout: 5
Handler: app.lambdaHandler
Runtime: nodejs16.x
Resources:
LambdaFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: src/
Events:
Api:
Type: Api
Properties:
Path: /
Method: GET
Outputs:
WebEndpoint:
Description: "API Gateway endpoint URL for Prod stage"
Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/"
But I am getting the error when I run the endpoint locally in the Docker container.
I run the following command to run the API locally.
sam build --use-container && sam local start-api
When I run that, the docker image was build successfully and there is no error in the console. In the console, it gives me a local endpoint, http://127.0.0.1:3000/.
When I visit the endpoint in the browser, I am getting this response.
{"message":"Internal server error"}
I also seeing the following errors in the console.
Invoking app.lambdaHandler (nodejs16.x)
Image was not found.
Removing rapid images for repo public.ecr.aws/sam/emulation-nodejs16.x
Building image........................
Failed to build Docker Image
NoneType: None
Exception on / [GET]
Traceback (most recent call last):
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/flask/app.py", line 1518, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/samcli/local/apigw/local_apigw_service.py", line 361, in _request_handler
self.lambda_runner.invoke(route.function_name, event, stdout=stdout_stream_writer, stderr=self.stderr)
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/samcli/commands/local/lib/local_lambda.py", line 137, in invoke
self.local_runtime.invoke(
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/samcli/lib/telemetry/metric.py", line 315, in wrapped_func
return_value = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/samcli/local/lambdafn/runtime.py", line 177, in invoke
container = self.create(function_config, debug_context, container_host, container_host_interface)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/samcli/local/lambdafn/runtime.py", line 73, in create
container = LambdaContainer(
^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/samcli/local/docker/lambda_container.py", line 93, in __init__
image = LambdaContainer._get_image(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/samcli/local/docker/lambda_container.py", line 236, in _get_image
return lambda_image.build(runtime, packagetype, image, layers, architecture, function_name=function_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/samcli/local/docker/lambda_image.py", line 164, in build
self._build_image(
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/samcli/local/docker/lambda_image.py", line 279, in _build_image
raise ImageBuildException("Error building docker image: {}".format(log["error"]))
samcli.commands.local.cli_common.user_exceptions.ImageBuildException: Error building docker image: The command '/bin/sh -c mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie' returned a non-zero code: 1
2023-01-21 09:29:27 127.0.0.1 - - [21/Jan/2023 09:29:27] "GET / HTTP/1.1" 502 -
2023-01-21 09:29:27 127.0.0.1 - - [21/Jan/2023 09:29:27] "GET /favicon.ico HTTP/1.1" 403 -
What is wrong with my code and how can I fix it?
We had a similar problem only on MacOS so if you are using MacOS this might help you (there is homebrew in your error response so i guess you are)
In our case aws-sam-cli was installed via brew.
What we did was deleting aws-sam-cli and installed it via Command Line - All users using arm64.pkg
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html
After that we got
=== ERROR: Error: Protocol error (Target.setDiscoverTargets): Target closed.
but this might be related to our project only.
When I try to connect to SQS using Celery, I'm not able to. The worker crashes with the following message:
[2022-10-28 14:05:39,237: CRITICAL/MainProcess] Unrecoverable error: ClientError('An error occurred (InvalidClientTokenId) when calling the GetQueueAttributes operation: The security token included in the request is invalid.')
Traceback (most recent call last):
File "/home/aditya/.local/lib/python3.10/site-packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
File "/home/aditya/.local/lib/python3.10/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/home/aditya/.local/lib/python3.10/site-packages/celery/bootsteps.py", line 365, in start
return self.obj.start()
File "/home/aditya/.local/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 332, in start
blueprint.start(self)
File "/home/aditya/.local/lib/python3.10/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/home/aditya/.local/lib/python3.10/site-packages/celery/worker/consumer/tasks.py", line 38, in start
c.task_consumer = c.app.amqp.TaskConsumer(
File "/home/aditya/.local/lib/python3.10/site-packages/celery/app/amqp.py", line 274, in TaskConsumer
return self.Consumer(
File "/home/aditya/.local/lib/python3.10/site-packages/kombu/messaging.py", line 387, in __init__
self.revive(self.channel)
File "/home/aditya/.local/lib/python3.10/site-packages/kombu/messaging.py", line 409, in revive
self.declare()
File "/home/aditya/.local/lib/python3.10/site-packages/kombu/messaging.py", line 422, in declare
queue.declare()
File "/home/aditya/.local/lib/python3.10/site-packages/kombu/entity.py", line 606, in declare
self._create_queue(nowait=nowait, channel=channel)
File "/home/aditya/.local/lib/python3.10/site-packages/kombu/entity.py", line 615, in _create_queue
self.queue_declare(nowait=nowait, passive=False, channel=channel)
File "/home/aditya/.local/lib/python3.10/site-packages/kombu/entity.py", line 643, in queue_declare
ret = channel.queue_declare(
File "/home/aditya/.local/lib/python3.10/site-packages/kombu/transport/virtual/base.py", line 523, in queue_declare
return queue_declare_ok_t(queue, self._size(queue), 0)
File "/home/aditya/.local/lib/python3.10/site-packages/kombu/transport/SQS.py", line 633, in _size
resp = c.get_queue_attributes(
File "/home/aditya/.local/lib/python3.10/site-packages/botocore/client.py", line 514, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/aditya/.local/lib/python3.10/site-packages/botocore/client.py", line 938, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the GetQueueAttributes operation: The security token included in the request is invalid.
The credentials are correct, and my Cloud Admin insists that my credentials have these permissions.
If it helps, these are in my settings.py :
from urllib.parse import quote
CELERY_BROKER_URL = 'sqs://{access_key}:{secret_key}#'.format(
access_key=quote(env.str("AWS_ACCESS_KEY_ID"), safe=''),
secret_key=quote(env.str("AWS_SECRET_ACCESS_KEY"), safe=''),
)
CELERY_BROKER_TRANSPORT_OPTIONS = {
"region": "us-east-2",
"polling_interval": 60,
'sdb_persistence': False,
"predefined_queues":{
'myq': {
'url': 'https://sqs.us-east-2.amazonaws.com/164782647287/myq',
'access_key_id': env.str("AWS_ACCESS_KEY_ID"),
'secret_access_key': env.str("AWS_SECRET_ACCESS_KEY"),
},
}
}
CELERY_TASK_DEFAULT_QUEUE = 'myq'
I'm using the latest version of Celery and associated dependencies:
celery[sqs]==5.2.7
Overview:
Selenium scraper works perfectly in headless mode. Spawning a virtual display shows no errors via XVFB:
from xvfbwrapper import Xvfb
vdisplay = Xvfb()
vdisplay.start()
vdisplay.stop()
But when I try to run them together, it errors out with:
[ERROR] SessionNotCreatedException: Message: session not created
from disconnected: Unable to receive message from renderer (Session info: chrome=96.0.4664.0)
Traceback:
Traceback (most recent call last):
File "/var/task/slack_main.py", line 34, in handler
scrape_price(asin_list)
File "/var/task/slack_main.py", line 58, in scrape_price
driver = webdriver.Chrome("/opt/chromedriver",options=options)
File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py", line 70, in __init__
super(WebDriver, self).__init__(DesiredCapabilities.CHROME['browserName'], "goog",
File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/chromium/webdriver.py", line 92, in __init__
RemoteWebDriver.__init__(
File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 275, in __init__
self.start_session(capabilities, browser_profile)
File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 365, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 430, in execute
self.error_handler.check_response(response)
File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
raise exception_class(message, screen, stacktrace)
Configuration:
Below is my complete Selenium and XVFB configuration:
from selenium import webdriver
from selenium_stealth import stealth
from xvfbwrapper import Xvfb
vdisplay = Xvfb()
vdisplay.start()
options = webdriver.ChromeOptions()
prefs = {"browser.downloads.dir": "//tmp//", "download.default_directory": "//tmp//", "directory_upgrade": True}
options.add_experimental_option("prefs", prefs)
options.binary_location = '/opt/chrome/chrome'
#options.add_argument('--headless') #toggled on and off when running with or without XVFB
options.add_argument('--no-sandbox')
options.add_argument("--disable-gpu")
options.add_argument("--window-size=1280x1696")
options.add_argument("--single-process")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--disable-dev-tools")
options.add_argument("--no-zygote")
options.set_capability('unhandledPromptBehavior', 'ignore')
options.add_argument("download.default_directory=/tmp")
driver = webdriver.Chrome("/opt/chromedriver",options=options)
stealth(driver,
languages=["en-US", "en"],
vendor="Google Inc.",
platform="Win32",
fix_hairline=True,
)
vdisplay.stop()
driver.close()
Why is it not connecting to the display? My guess is it has something to do with the '--headless' toggle?
Versions and Tools:
Selenium version 3.141.0
xvfbwrapper version 0.2.9
Docker is used to compile and push to AWS Lambda, base image used (no changes are made in docker file with or without XVFB)
Edit:
Found a pull request for XVFB configuration in the Github Repo of my base image. Even used the exact same code from the pull request and I still recieve the exact same error. Maybe this has something to do with AWS offboard?
I'm working with an OpenAPI 3.0.1 yaml and it's unable to get the API webserver started due to the below error. I tried almost everything that is under my knowledge but I'm very new at OpenAPI and the documentation was followed as it is. Any thoughts on what could be wrong here?
This is the error on loading up the server:
Failed to add operation for GET /v2/catalog
Traceback (most recent call last):
File "C:\Programs\Python\Python38\lib\site-packages\connexion\apis\abstract.py", line 209, in add_paths
self.add_operation(path, method)
File "C:\Programs\Python\Python38\lib\site-packages\connexion\apis\abstract.py", line 162, in add_operation
operation = make_operation(
File "C:\Programs\Python\Python38\lib\site-packages\connexion\operations\__init__.py", line 8, in make_operation
return spec.operation_cls.from_spec(spec, *args, **kwargs)
File "C:\Programs\Python\Python38\lib\site-packages\connexion\operations\openapi.py", line 128, in from_spec
return cls(
File "C:\Programs\Python\Python38\lib\site-packages\connexion\operations\openapi.py", line 75, in __init__
super(OpenAPIOperation, self).__init__(
File "C:\Programs\Python\Python38\lib\site-packages\connexion\operations\abstract.py", line 96, in __init__
self._resolution = resolver.resolve(self)
File "C:\Programs\Python\Python38\lib\site-packages\connexion\resolver.py", line 40, in resolve
return Resolution(self.resolve_function_from_operation_id(operation_id), operation_id)
File "C:\Programs\Python\Python38\lib\site-packages\connexion\resolver.py", line 64, in resolve_function_from_operation_id
raise ResolverError(msg, sys.exc_info())
connexion.exceptions.ResolverError: <ResolverError: Cannot resolve operationId "catalog.get"! Import error was "No module named 'catalog'">
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Programs\Python\Python38\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Programs\Python\Python38\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "D:\API\swagger_server\__main__.py", line 25, in <module>
main()
File "D:\API\swagger_server\__main__.py", line 18, in main
app.add_api('D:\API\swagger_server\swagger\swagger.yaml', arguments={'title': 'GPI API Broker'}, pythonic_params=True)
File "C:\Programs\Python\Python38\lib\site-packages\connexion\apps\flask_app.py", line 57, in add_api
api = super(FlaskApp, self).add_api(specification, **kwargs)
File "C:\Programs\Python\Python38\lib\site-packages\connexion\apps\abstract.py", line 141, in add_api
api = self.api_cls(specification,
File "C:\Programs\Python\Python38\lib\site-packages\connexion\apis\abstract.py", line 111, in __init__
self.add_paths()
File "C:\Programs\Python\Python38\lib\site-packages\connexion\apis\abstract.py", line 216, in add_paths
self._handle_add_operation_error(path, method, err.exc_info)
File "C:\Programs\Python\Python38\lib\site-packages\connexion\apis\abstract.py", line 231, in _handle_add_operation_error
raise value.with_traceback(traceback)
File "C:\Programs\Python\Python38\lib\site-packages\connexion\resolver.py", line 61, in resolve_function_from_operation_id
return self.function_resolver(operation_id)
File "C:\Programs\Python\Python38\lib\site-packages\connexion\utils.py", line 110, in get_function_from_name
module = importlib.import_module(module_name)
File "C:\Programs\Python\Python38\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'catalog'
The startup command is basically: python -m swagger_server
Finally, this is part of my YAML where the operationId is mentioned:
openapi: 3.0.1
info:
title: Open Service Broker API
description: The Open Service Broker API defines an HTTP(S) interface between Platforms
and Service Brokers.
contact:
name: Open Service Broker API
url: https://www.openservicebrokerapi.org/
email: open-service-broker-api#googlegroups.com
license:
name: Apache 2.0
url: http://www.apache.org/licenses/LICENSE-2.0.html
version: master - might contain changes that are not yet released
externalDocs:
description: The offical Open Service Broker API specification
url: https://github.com/openservicebrokerapi/servicebroker/
servers:
- url: http://localhost:80/
- url: https://localhost:80/
security:
- basicAuth: []
paths:
/v2/catalog:
get:
tags:
- Catalog
summary: get the catalog of services that the service broker offers
operationId: 'catalog.get'
parameters:
- name: X-Broker-API-Version
in: header
...
Thank you all in advance!
The operationId must be relative to where your app is running.
swagger_server
|-- app.py
|-- __init__.py
|-- OpenAPI
| |-- openapi.yml
|-- models
| |-- catalog.py
Given the above folder structure, you start the app in the directory swagger_server. Consequently, the path of app.py is in your sys.path. Relative to this path you need to specify the operationId in your openapi.yml:
paths:
/v2/catalog:
get:
tags:
- Catalog
summary: get the catalog of services that the service broker offers
operationId: 'models.catalog.get'
The file catalog.py must contain a function def get(name). Here you can define your service/handler for the endpoint. It must include the parameter name because you have specified the parameter in your YAML file.
I am trying to perform jwt.encode on a small request message with a private key that I am retrieving from DynamoDB. My work station is Mac OS.
The code works in my local environment but failed when I tried to execute it in AWS lambda python3.7 environment. Any suggestions on how to solve this issue?
{
"errorMessage": "Invalid PKCS8 header",
"errorType": "JWSError",
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 228, in lambda_handler\n response = send_request(url, body=body)\n",
" File \"/var/task/lambda_function.py\", line 67, in send_request\n token = _gen_jwt(private_key)\n",
" File \"/var/task/lambda_function.py\", line 52, in _gen_jwt\n token = jwt.encode(payload, private_key, algorithm='RS256', headers=headers)\n",
" File \"/var/task/jose/jwt.py\", line 61, in encode\n return jws.sign(claims, key, headers=headers, algorithm=algorithm)\n",
" File \"/var/task/jose/jws.py\", line 47, in sign\n signed_output = _sign_header_and_claims(encoded_header, encoded_payload, algorithm, key)\n",
" File \"/var/task/jose/jws.py\", line 168, in _sign_header_and_claims\n raise JWSError(e)\n"
]
}
This issue was resolved after I rebuild my lambda package within a Linux environment and redeploy to AWS Lambda.
I created a docker image from python3.7.4 and deploy the package to lambda from within the container.
This solution also solves the issue:
ImportError: /var/task/cryptography/hazmat/bindings/_constant_time.so: invalid ELF header
According to https://github.com/pyca/cryptography/issues/3051