How to resolve 'Invalid PKCS8 header error' in AWS lambda - python-3.x

I am trying to perform jwt.encode on a small request message with a private key that I am retrieving from DynamoDB. My work station is Mac OS.
The code works in my local environment but failed when I tried to execute it in AWS lambda python3.7 environment. Any suggestions on how to solve this issue?
{
"errorMessage": "Invalid PKCS8 header",
"errorType": "JWSError",
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 228, in lambda_handler\n response = send_request(url, body=body)\n",
" File \"/var/task/lambda_function.py\", line 67, in send_request\n token = _gen_jwt(private_key)\n",
" File \"/var/task/lambda_function.py\", line 52, in _gen_jwt\n token = jwt.encode(payload, private_key, algorithm='RS256', headers=headers)\n",
" File \"/var/task/jose/jwt.py\", line 61, in encode\n return jws.sign(claims, key, headers=headers, algorithm=algorithm)\n",
" File \"/var/task/jose/jws.py\", line 47, in sign\n signed_output = _sign_header_and_claims(encoded_header, encoded_payload, algorithm, key)\n",
" File \"/var/task/jose/jws.py\", line 168, in _sign_header_and_claims\n raise JWSError(e)\n"
]
}

This issue was resolved after I rebuild my lambda package within a Linux environment and redeploy to AWS Lambda.
I created a docker image from python3.7.4 and deploy the package to lambda from within the container.
This solution also solves the issue:
ImportError: /var/task/cryptography/hazmat/bindings/_constant_time.so: invalid ELF header
According to https://github.com/pyca/cryptography/issues/3051

Related

AWS SAM API endpoint using a Lambda is throwing error when it runs locally with "sam start local-api"

I am building a simple API endpoint using AWS Lambda function and API Gateway. I am orchestrating those resources using SAM. I can deploy my function to the AWS cloud and access the endpoint. It's working as expected.
Here is my lambda function.
exports.lambdaHandler = async (event) => ({
statusCode: 200,
body: {
message: `Hello World!`
}
})
Here is my SAM CloudFormation template
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: HTTP API
Globals:
Function:
Timeout: 5
Handler: app.lambdaHandler
Runtime: nodejs16.x
Resources:
LambdaFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: src/
Events:
Api:
Type: Api
Properties:
Path: /
Method: GET
Outputs:
WebEndpoint:
Description: "API Gateway endpoint URL for Prod stage"
Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/"
But I am getting the error when I run the endpoint locally in the Docker container.
I run the following command to run the API locally.
sam build --use-container && sam local start-api
When I run that, the docker image was build successfully and there is no error in the console. In the console, it gives me a local endpoint, http://127.0.0.1:3000/.
When I visit the endpoint in the browser, I am getting this response.
{"message":"Internal server error"}
I also seeing the following errors in the console.
Invoking app.lambdaHandler (nodejs16.x)
Image was not found.
Removing rapid images for repo public.ecr.aws/sam/emulation-nodejs16.x
Building image........................
Failed to build Docker Image
NoneType: None
Exception on / [GET]
Traceback (most recent call last):
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/flask/app.py", line 1518, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/samcli/local/apigw/local_apigw_service.py", line 361, in _request_handler
self.lambda_runner.invoke(route.function_name, event, stdout=stdout_stream_writer, stderr=self.stderr)
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/samcli/commands/local/lib/local_lambda.py", line 137, in invoke
self.local_runtime.invoke(
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/samcli/lib/telemetry/metric.py", line 315, in wrapped_func
return_value = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/samcli/local/lambdafn/runtime.py", line 177, in invoke
container = self.create(function_config, debug_context, container_host, container_host_interface)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/samcli/local/lambdafn/runtime.py", line 73, in create
container = LambdaContainer(
^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/samcli/local/docker/lambda_container.py", line 93, in __init__
image = LambdaContainer._get_image(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/samcli/local/docker/lambda_container.py", line 236, in _get_image
return lambda_image.build(runtime, packagetype, image, layers, architecture, function_name=function_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/samcli/local/docker/lambda_image.py", line 164, in build
self._build_image(
File "/opt/homebrew/Cellar/aws-sam-cli/1.70.0/libexec/lib/python3.11/site-packages/samcli/local/docker/lambda_image.py", line 279, in _build_image
raise ImageBuildException("Error building docker image: {}".format(log["error"]))
samcli.commands.local.cli_common.user_exceptions.ImageBuildException: Error building docker image: The command '/bin/sh -c mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie' returned a non-zero code: 1
2023-01-21 09:29:27 127.0.0.1 - - [21/Jan/2023 09:29:27] "GET / HTTP/1.1" 502 -
2023-01-21 09:29:27 127.0.0.1 - - [21/Jan/2023 09:29:27] "GET /favicon.ico HTTP/1.1" 403 -
What is wrong with my code and how can I fix it?
We had a similar problem only on MacOS so if you are using MacOS this might help you (there is homebrew in your error response so i guess you are)
In our case aws-sam-cli was installed via brew.
What we did was deleting aws-sam-cli and installed it via Command Line - All users using arm64.pkg
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html
After that we got
=== ERROR: Error: Protocol error (Target.setDiscoverTargets): Target closed.
but this might be related to our project only.

InvalidClientTokenId error when trying to access SQS from Celery

When I try to connect to SQS using Celery, I'm not able to. The worker crashes with the following message:
[2022-10-28 14:05:39,237: CRITICAL/MainProcess] Unrecoverable error: ClientError('An error occurred (InvalidClientTokenId) when calling the GetQueueAttributes operation: The security token included in the request is invalid.')
Traceback (most recent call last):
File "/home/aditya/.local/lib/python3.10/site-packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
File "/home/aditya/.local/lib/python3.10/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/home/aditya/.local/lib/python3.10/site-packages/celery/bootsteps.py", line 365, in start
return self.obj.start()
File "/home/aditya/.local/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 332, in start
blueprint.start(self)
File "/home/aditya/.local/lib/python3.10/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/home/aditya/.local/lib/python3.10/site-packages/celery/worker/consumer/tasks.py", line 38, in start
c.task_consumer = c.app.amqp.TaskConsumer(
File "/home/aditya/.local/lib/python3.10/site-packages/celery/app/amqp.py", line 274, in TaskConsumer
return self.Consumer(
File "/home/aditya/.local/lib/python3.10/site-packages/kombu/messaging.py", line 387, in __init__
self.revive(self.channel)
File "/home/aditya/.local/lib/python3.10/site-packages/kombu/messaging.py", line 409, in revive
self.declare()
File "/home/aditya/.local/lib/python3.10/site-packages/kombu/messaging.py", line 422, in declare
queue.declare()
File "/home/aditya/.local/lib/python3.10/site-packages/kombu/entity.py", line 606, in declare
self._create_queue(nowait=nowait, channel=channel)
File "/home/aditya/.local/lib/python3.10/site-packages/kombu/entity.py", line 615, in _create_queue
self.queue_declare(nowait=nowait, passive=False, channel=channel)
File "/home/aditya/.local/lib/python3.10/site-packages/kombu/entity.py", line 643, in queue_declare
ret = channel.queue_declare(
File "/home/aditya/.local/lib/python3.10/site-packages/kombu/transport/virtual/base.py", line 523, in queue_declare
return queue_declare_ok_t(queue, self._size(queue), 0)
File "/home/aditya/.local/lib/python3.10/site-packages/kombu/transport/SQS.py", line 633, in _size
resp = c.get_queue_attributes(
File "/home/aditya/.local/lib/python3.10/site-packages/botocore/client.py", line 514, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/aditya/.local/lib/python3.10/site-packages/botocore/client.py", line 938, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the GetQueueAttributes operation: The security token included in the request is invalid.
The credentials are correct, and my Cloud Admin insists that my credentials have these permissions.
If it helps, these are in my settings.py :
from urllib.parse import quote
CELERY_BROKER_URL = 'sqs://{access_key}:{secret_key}#'.format(
access_key=quote(env.str("AWS_ACCESS_KEY_ID"), safe=''),
secret_key=quote(env.str("AWS_SECRET_ACCESS_KEY"), safe=''),
)
CELERY_BROKER_TRANSPORT_OPTIONS = {
"region": "us-east-2",
"polling_interval": 60,
'sdb_persistence': False,
"predefined_queues":{
'myq': {
'url': 'https://sqs.us-east-2.amazonaws.com/164782647287/myq',
'access_key_id': env.str("AWS_ACCESS_KEY_ID"),
'secret_access_key': env.str("AWS_SECRET_ACCESS_KEY"),
},
}
}
CELERY_TASK_DEFAULT_QUEUE = 'myq'
I'm using the latest version of Celery and associated dependencies:
celery[sqs]==5.2.7

(404) when calling the HeadObject operation: Not Found

I am trying to download a file from S3, I have already check, the file path is correct, the bucket do exist and my lambda function have fullAdmin Access to S3
def download_file():
s3_bucket_name = os.getenv("BUCKET_NAME")
file_key=os.getenv("FILE_NAME")
try:
s3_client.download_file(s3_bucket_name, file_key, 'user-data')
except Exception as e:
raise e
When my code is ran I get the following errors.
{
"errorMessage": "An error occurred (404) when calling the HeadObject operation: Not Found",
"errorType": "ClientError",
"requestId": "e2145aa0-8ab6-4448-a410-d05fa8b3795f",
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 106, in lambda_handler\n user_list = read_agent_list(h_group)\n",
" File \"/var/task/lambda_function.py\", line 100, in read_agent_list\n return map_agents_to_hgroup(userList,h_group)\n",
" File \"/var/task/lambda_function.py\", line 24, in map_agents_to_hgroup\n download_file()#download the file to update\n",
" File \"/var/task/lambda_function.py\", line 82, in download_file\n raise e\n",
" File \"/var/task/lambda_function.py\", line 80, in download_file\n s3_client.download_file(s3_bucket_name, file_key, 'user-data')\n",
" File \"/var/runtime/boto3/s3/inject.py\", line 171, in download_file\n return transfer.download_file(\n",
" File \"/var/runtime/boto3/s3/transfer.py\", line 307, in download_file\n future.result()\n",
" File \"/var/runtime/s3transfer/futures.py\", line 106, in result\n return self._coordinator.result()\n",
" File \"/var/runtime/s3transfer/futures.py\", line 265, in result\n raise self._exception\n",
" File \"/var/runtime/s3transfer/tasks.py\", line 255, in _main\n self._submit(transfer_future=transfer_future, **kwargs)\n",
" File \"/var/runtime/s3transfer/download.py\", line 340, in _submit\n response = client.head_object(\n",
" File \"/var/runtime/botocore/client.py\", line 386, in _api_call\n return self._make_api_call(operation_name, kwargs)\n",
" File \"/var/runtime/botocore/client.py\", line 705, in _make_api_call\n raise error_class(parsed_response, operation_name)\n"
]
}

unable to loacte credentials while make a connection with DB using boto3

I want to make connection to my Database and want to retrieve data. I am currently using AWS amazon linux2 instance. I used boto3 to connect.
def db_conn():
secret_id = 'XXXXXXXXXXXXXXXX'
try:
client = boto3.client('secretsmanager',region_name="ap-southeast-2")
get_secret_value_response = client.get_secret_value(SecretId=secret_id)
except Exception as e:
raise e
else:
if 'SecretString' in get_secret_value_response:
Secret_Json = json.loads(get_secret_value_response['SecretString'])
if Secret_Json is None:
print("secret string is null")
exit()
driver = 'postgresql+psycopg2://'
db_user = Secret_Json['username']
db_pw = Secret_Json['password']
db_address_port_db = Secret_Json['host'] + \
':' + \
str(Secret_Json['port']) + \
'/' + \
Secret_Json['dbInstanceIdentifier']
application.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
application.config['SQLALCHEMY_DATABASE_URI'] = driver + db_user + ':' + db_pw + '#' + db_address_port_db
db = SQLAlchemy(application)
return db
I face an error saying no credentials found
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 622, in _make_api_call
operation_model, request_dict, request_context)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 641, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 132, in _send_request
request = self.create_request(request_dict, operation_model)
File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 116, in create_request
operation_name=operation_model.name)
File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line 90, in handler
return self.sign(operation_name, request)
File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line 160, in sign
auth.add_auth(request)
File "/usr/local/lib/python3.7/site-packages/botocore/auth.py", line 357, in add_auth
raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
Please help me what to do to solve this?
You have to assign an IAM role to your instance with required permissions:
How do I assign an existing IAM role to an EC2 instance?
IAM roles for Amazon EC2
Boto3 will use the permissions from the role to get access to your resources, such as Secrets Manager.
The role could include, for example, an inline policy to read from Secrets Manager:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "secretsmanager:GetSecretValue",
"Resource": "<arn-of-your-sercert>"
}
]
}
If you use KMS for encrypting your secret, KMS permissions may also be required.

Docker-py client gives "invalid client port specification"

I am using the docker-py client to create containers on a need basis. So for this, I am using a generator to come up with a port number and trying to use httpd image on a particular port of the host from the generator. But, the client gives out ("invalid port specification: "port number here"") for any number, for any port number that I am trying to use.
Below is sample code that I am trying:
import docker
client = docker.from_env()
container= client.containers.run(image="httpd", ports={'80/tcp': 43545}, detach=True)
To note: The number 43545 does not have any significance here.
Docker details:
Client - 19.03.6
API Version - 1.40
Engine: 19.03.6
Error:
File "/project/api/.venv/lib/python3.7/site-packages/docker/models/containers.py", line 803, in run
detach=detach, **kwargs)
File "/project/api/.venv/lib/python3.7/site-packages/docker/models/containers.py", line 861, in create
resp = self.client.api.create_container(**create_kwargs)
File "/project/api/.venv/lib/python3.7/site-packages/docker/api/container.py", line 429, in create_container
return self.create_container_from_config(config, name)
File "/projectapi/.venv/lib/python3.7/site-packages/docker/api/container.py", line 440, in create_container_from_config
return self._result(res, True)
File "/projectpi/.venv/lib/python3.7/site-packages/docker/api/client.py", line 267, in _result
self._raise_for_status(response)
File "/projectapi/.venv/lib/python3.7/site-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/projectyter/api/.venv/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 400 Client Error: Bad Request ("invalid port specification: "43545"")

Resources