Currently it looks like boto3 has an upper limit of 12 hours as per here. Is there a way you can increase the S3 session duration for more than 12 hours?
After 12 hours any call to show data on spark dataframe lands into this exception: Error Code: 403 Forbidden; Request ID: tx0000055c5a217ed148183-0063170ae4-189e9643-de-bs01; S3 Extended Request ID: null), S3 Extended Request ID
sdf.show(3, False)
22/09/06 08:55:00 WARN TaskSetManager: Lost task 3.0 in stage 464.0
(TID 77378) (100.66.45.57 executor 128): java.io.IOException:
com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service:
Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID:
tx0000055c5a217ed148183-0063170ae4-189e9643-de-bs01; S3 Extended
Request ID: null), S3 Extended Request ID: null
at com.ibm.stocator.fs.cos.COSAPIClient.getFileStatus(COSAPIClient.java:540)
at com.ibm.stocator.fs.ObjectStoreFileSystem.getFileStatus(ObjectStoreFileSystem.java:513)
at org.apache.parquet.hadoop.util.HadoopInputFile.fromPath(HadoopInputFile.java:39)
at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:448)
at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.footerFileMetaData$lzycompute$1(ParquetFileFormat.scala:271)
Related
i want to download artifact from remote server i used for mlflow tracking with below code :
from mlflow.tracking import MlflowClient
import pandas as pd
import mlflow
uri="http://some_url"
ex_id = '1'
mlflow.set_tracking_uri(uri)
experiments = mlflow.list_experiments()
runs = mlflow.search_runs(ex_id)
runs=runs[runs['tags.dataset'] == 'my_dataset']
client = MlflowClient()
client.download_artifacts("c21d2c4e75c047608003235f213e5bb5", "myarts/arts.csv", dst_path="./local_arts/")
but after a minute it give me below error :
The following failures occurred while downloading one or more artifacts from http://some_url/api/2.0/mlflow-artifacts/artifacts/1/c21d2c4e75c047608003235f213e5bb5/artifacts: {'myarts/': 'MlflowException("API request to http://some_url/api/2.0/mlflow-artifacts/artifacts/1/c21d2c4e75c047608003235f213e5bb5/artifacts/myarts/ failed with exception HTTPConnectionPool(host='some_url', port=80): Max retries exceeded with url: /api/2.0/mlflow-artifacts/artifacts/1/c21d2c4e75c047608003235f213e5bb5/artifacts/myarts/ (Caused by ResponseError('too many 500 error responses'))")'}
what is wrong with it ?
i upgraded mlflow but nothing changed
I am calling deployed sagemaker Endpoint from lambda for load testing and lambda is getting called from api gateway integration when I am making call using gate POST rest api(with API Proxy Integration Enabled) from client side. But always rest api with lambda returning 200 even when invoke_endpoint doesn't return anything due to traffic load on SageMaker Endpoint.
#logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
#tracer.capture_lambda_handler
def lambda_handler(event: dict, context: LambdaContext) -> dict:
# return app.resolve(event, context)
try:
print("Received event: " + json.dumps(event, indent=2))
print(type(event), '--> ', event)
payload = event['data']
print(payload)
response = runtime_client.invoke_endpoint(EndpointName=ENDPOINT_NAME,
ContentType='application/json',
Body=json.dumps(payload),
Accept='Accept')
print(response)
print(response.keys())
# if "errorMessage" in str(respone.text):
# return
if "errorMessage" in response.keys() and "502 Bad Gateway" in str(response):
return {'statusCode': 502,'body': response}
elif "errorMessage" in response.keys():
return {'statusCode': 504,'body': response}
result = json.loads(response['Body'].read().decode())
return result
except Exception as e:
print(e)
exception_handler(e)
I enabled API Integration proxy while creating api method in api gateway and using aws-lambda-powertools to resolve the error but still client is getting 200 response always event my SageMaker Endpoint is not responding in the code due to load.
Error example with 200 code is below:
{"errorMessage":"2023-01-16T15:23:25.132Z c1bf04bf-12f7-46ab-b9a3-ffb1865eed26 Task timed out after 3.01 seconds"}
status_code: 200
Can anyone help me out here. Thanks in advance.
PS: I am very new to lambda.
Increase your lambda timeout limit.
I have a basic lambda function which accepts a JSON, and transforms to XML and uploads the same to s3 bucket. I am running the test through the lambda console and the xml is getting generated and uploaded to s3 as well. Issue happens when I run the test through API Gateway, and there I get the error:
"message": "Internal server error" and Status: 502. The error also points out where is the code issue:
{"errorMessage": "'header'", "errorType": "KeyError", "stackTrace": [" File \"/var/task/sample.py\", line 21, in printMessage\n b1.text = event['header']['echoToken']\n"]}
The same code is working when running through lambda console. Below is the code snippet with the line that throws the error:
def printMessage(event, context):
print(event)
#event = json.loads(event['header'])
root = ET.Element("Reservation")
m1 = ET.Element("header")
root.append(m1)
b1 = ET.SubElement(m1, "echoToken")
b1.text = event['header']['echoToken'] #this line throws error
I got a similar question here:
AWS Lambda-API gateway "message": "Internal server error" (502 Bad Gateway). Below is the print of event object.
{'header': {'echoToken': '907f44fc-6b51-4237-8018-8a840fd87f04', 'timestamp': '2018-03-07 20:59:575Z'}, 'reservation': {'hotel': {'uuid': '3_c5f3c903-c43d-4967-88d1-79ae81d00fcb', 'code': 'TASK1', 'offset': '+06:00'}, 'reservationId': 12345, 'confirmationNumbers': [{'confirmationNumber': '12345', 'source': 'ENCORA', 'guest': 'Arturo Vargas'}, {'confirmationNumber': '67890', 'source': 'NEARSOFT', 'guest': 'Carlos Hernández'}], 'lastUpdateTimestamp': '2018-03-07 20:59:541Z', 'lastUpdateOperatorId': 'task.user'}}
As per the solution in the link, if I try to do : header = json.loads(event['header'])
I get the below error:
"errorMessage": "the JSON object must be str, bytes or bytearray, not dict",
This is a task that I need to complete, and I have tried few things, but is always failing when tested through API Gateway and POSTMAN. So, my question is: I don't want to change my complete code for XML transformation, and is there any way that I load the complete event object in python, so that it works both through lambda console and API gateway?
Thanks in Advance!!
The problem is similar to the one discussed here
Below code will fix the line causing the issue.
eventDict = json.loads(event)
b1Text = json.dumps(eventDict['header']['echoToken'])
print(b1Text)
I am trying to deploy my first function on AWS Lambda that should store results on my S3 instance following this tutorial. To be sure I have the permission to write files to the S3 bucket I've created I added the code below the commented one into the Function code section.
# import json
# def lambda_handler(event, context):
# # TODO implement
# return {
# 'statusCode': 200,
# 'body': json.dumps('Hello from fucking Lambda!')
# }
import boto3
def lambda_handler(event, context):
s3 = boto3.resource("s3") # pointer to AWS S3 service
bucket_name = "my-own-bucket" # bucket for saving our data
file_name = "HelloWorld.txt" # dummy file
body = event['key1'] + ' ' + event['key2'] + event['key3']
# "Hello World!" from event input
s3.Bucket(bucket_name).put_object(Key=file_name, Body=body)
# write object to bucket
return f"Succeed! Find your new {file_name} file in {bucket_name} bucket ;)"
I used the template test event:
My code automatically saved, so I opened the S3 bucket again and verified that a HelloWorld.txt file existed. But it didn't. And the execution results were:
Response:
{
"statusCode": 200,
"body": "\"Hello from Lambda!\""
}
Request ID:
"93c1c61c-73cf-4308-9a88-0e8771612b8e"
Function logs:
START RequestId: 93c1c61c-73cf-4308-9a88-0e8771612b8e Version: $LATEST
END RequestId: 93c1c61c-73cf-4308-9a88-0e8771612b8e
REPORT RequestId: 93c1c61c-73cf-4308-9a88-0e8771612b8e Duration: 0.32 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 43 MB
Like if the commented code had still been taken into account.
It could be due to following reasons:
You are invoking old lambda version, rather then your latest version.
You are not deploying your changes. In the new UI, there is no 'Save' button. Instead there is orange Deploy button.
I'am trying to run a simple code of Spark that copies the content of an RDD into an elastic search document. Both spark and elastic search are installed on my local machine.
import org.elasticsearch.spark.sql._
import org.apache.spark.sql.SparkSession
object ES {
case class Person(ID: Int, name: String, age: Int, numFriends:
Int);
def mapper(line: String): Person = {
val fields = line.split(',')
val person: Person = Person(fields(0).toInt, fields(1),
fields(2).toInt, fields(3).toInt)
return person}
def main(args: Array[String]): Unit = {
val spark: SparkSession =
SparkSession
.builder().master("local[*]")
.appName("SparkEs")
.config("es.index.auto.create", "true")
.config("es.nodes","localhost:9200")
.getOrCreate()
import spark.implicits._
val lines = spark.sparkContext.textFile("/home/herch/fakefriends.csv")
val people = lines.map(mapper).toDF()
people.saveToEs("spark/people")
}
}
I'am Getting this error. After multiples retries
INFO HttpMethodDirector: I/O exception (java.net.ConnectException)
caught when processing request:Connection timed out (Connection timed
out)
INFO HttpMethodDirector: Retrying request
INFO DAGScheduler: ResultStage 0 (runJob at EsSparkSQL.scala:97)
failed in 525.902 s due to Job aborted due to stage failure: Task 1
in stage 0.0 failed 1 times, most recent failure: Lost task 1.0 in
stage 0.0 (TID 1, localhost, executor driver):
org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException:
Connection error (check network and/or proxy settings)- all nodes
failed; tried [[192.168.0.22:9200]]
It seems to be a connection problem but i cannot identify its cause. Elastic search is running on my local machine on localhost:9200 and i'am able to query it via the terminal.
Since you are running both locally, you need to set es.nodes.wan.only to true (default false) in your SparkConf. I ran into the same exact problem and that fixed it.
See: https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html
As seen on the elasticsearch / spark connector documentation page, you need to separate the host and port arguments inside the configuration :
val options13 = Map("path" -> "spark/index",
"pushdown" -> "true",
"es.nodes" -> "someNode", "es.port" -> "9200")
See how es.nodes only contains the host name, and es.port contains the HTTP port.