I have few json files in s3, now want to delete some objects from that json file by opening the file for one time and delete those objects at a time and also chaeck if the objects are present in the json file or not. and lastly want to update the file in the same location in s3.
img_list is the list of objects to be deleted.
img_list = ['0000002.jpg','0000003.jpg','0000006.jpg']
edit_jsons_path = 'path where the json is located in s3'
# Contents like this:
# {
# "abc": {
# "0000001.jpg": {
# },
# "0000002.jpg": {
# },
# "0000003.jpg": {
# },
# "0000004.jpg": {
# }
# },
# "xyz": {
# "xx": 467,
# "yy": 4
# }
# }
s3 = boto3.client('s3')
def json_edit(img_list,edit_jsons_path):
response = s3.get_object(Bucket = bucket, Key = edit_jsons_path)
json_content = response["Body"].read().decode("utf-8")
data = json.loads(json_content)
for obj in img_list:
del data['abc'][obj]
s3.put_object(Bucket = bucket, Key = edit_jsons_path, Body = json.dumps(data))
Expected output:
{
"abc": {
"0000001.jpg": {
},
"0000004.jpg": {
}
},
"xyz": {
"xx": 467,
"yy": 4
}
}
S3 is object/blob storage. There isn't a way to "edit" an S3 file like a traditional file on your local file system. You will need to read the file, make the edits you desire, and push the update over the top of the existing file.
I have aerospike 6.0.0 installed on server and being used as cache. Also I am using Node.js client 5.2.0 to connect to the database.The aerospike nodes are configured to use TLS. The config of the namespace I am using is:
namespace application_cache {
memory-size 5G
replication-factor 2
default-ttl 1h
nsup-period 1h
high-water-memory-pct 80
stop-writes-pct 90
storage-engine memory
}
The config I am passing while connecting to the Db is:
"config": {
"captureStackTraces":true,
"connTimeoutMs": 10000,
"maxConnsPerNode": 1000,
"maxCommandsInQueue" : 300,
"maxCommandsInProcess" : 100,
"user": "test",
"password": "test",
"policies": {
"read": {
"maxRetries": 3,
"totalTimeout": 0,
"socketTimeout": 0
},
"write": {
"maxRetries": 3,
"totalTimeout": 0,
"socketTimeout": 0
},
"info": {
"timeout": 10000
}
},
"tls": {
"enable": true,
"cafile": "./certs/ca.crt",
"keyfile": "./certs/server.key",
"certfile": "./certs/server.crt"
},
"hosts": [
{
"addr": "-----.com",
"port": 4333,
"tlsname": "----"
},
{
"addr": "---.com",
"port": 4333,
"tlsname": "-----"
},
{
"addr": "----.com",
"port": 4333,
"tlsname": "-----"
}
]
}
This is the function I am using to write to a set in this namespace:
async function updateApp(appKey, bins) {
let result = {};
try {
const writePolicy = new Aerospike.WritePolicy({
maxRetries: 3,
socketTimeout: 0,
totalTimeout: 0,
exists: Aerospike.policy.exists.CREATE_OR_UPDATE,
key: Aerospike.policy.key.SEND
});
const key = new Aerospike.Key(namespace, set, appKey);
let meta = {
ttl: 3600
};
let bins = {};
result = await asc.put(key, bins, meta, writePolicy);
} catch (err) {
console.error("Aerospike Error:", JSON.stringify({params: {namespace, set, appKey}, err: {code: err.code, message: err.message}}));
return err;
}
return result;
}
It works most of the time but once in a while I get this error:
Aerospike Error: {"params":{"namespace":"application_cache","set":"apps","packageName":"com.test.application"},"err":{"code":-6,"message":"TLS write failed: -2 34D53CA5C4E0734F 10.57.49.180:4333"}}
Every record in this set has about 12-15 bins and the size of the record varies between 300Kb to 1.5Mb. The bigger the size of the record the higher chance of this error showing up. Has anyone faced this issue before and what could be causing this?
I answered this on the community forum but figured I'd add it here as well.
Looking at the server error codes a code: 6 is AS_ERR_BIN_EXISTS. Your write policy is setup with exists: Aerospike.policy.exists.CREATE_OR_UPDATE but the options for exists are:
Name
Description
IGNORE
Write the record, regardless of existence. (I.e. create or update.)
CREATE
Create a record, ONLY if it doesn't exist.
UPDATE
Update a record, ONLY if it exists.
REPLACE
Completely replace a record, ONLY if it exists.
CREATE_OR_REPLACE
Completely replace a record if it exists, otherwise create it.
My guess is that it's somehow defaulting to a CREATE only causing it to throw that error. Try updating your write policy to use exists: Aerospike.policy.exists.IGNORE and see if you still get that error.
I'm trying to figure out how to insert an image into CouchDB using the node-CouchDB library found here: https://www.npmjs.com/package/node-couchdb
Here's what I've done:
fs.readFile('download.jpeg', (err, data) => {
binary_data = new Buffer(data, 'binary');
couch.insertAttachment("node_db", doc_number, "download.jpeg", binary_data, rev_number).then(({data, headers, status}) => {
}, err => {
console.log("ERROR"+ err.code);
});
});
The result is that CouchDB stores this in the document format like such:
{
"_id": "2741d6f37d61d6bbdf63df3be5000504",
"_rev": "22-bfdbe6db35c7d9873a2cc8a38afb2833",
"_attachments": {
"attachment": {
"content_type": "application/json",
"revpos": 22,
"digest": "md5-on0A+d7045WPI6FyS1ut4g==",
"length": 22482,
"stub": true
}
}
}
//This is what the data looks like in CouchDB using the View Attachment Function through the interface:
{"type":"Buffer","data":[255,216,255,224,0,16,74,70,73,70,0,1,1,0,0,1,0,1,0,0,255,219,0,132,0,9,6,7,18,18,18,21,18,19,19,22,21,21,23,23,23,24,21,21,21,23,23,21,21,24,21,21,21,23,22,22,21,21,22,24,29,40,32,24,26,37,29,21,21,33,49,33,37,41,43,46,46,46,23,31,51,56,51,45,55,40,45,46,43,1,10,10,10,14,13,14,26,16,16,26,45,37,29,37,45,45,45,45,45,45,45,241,...]
I then tried changing the Content-Type attribute to "image/jpeg" in the header of the request resulting in:
{
"_id": "2741d6f37d61d6bbdf63df3be5000504",
"_rev": "23-cf8c2076b43082fdfe605cad68ef2355",
"_attachments": {
"attachment": {
"content_type": "image/jpeg",
"revpos": 23,
"digest": "md5-SaekQP37DCCeGX2M8UVeGQ==",
"length": 22482,
"stub": true
}
}
}
However, this still results in an image that isn't viewable from the CouchDB interface (clicking View Attachments). The image, in this case, is only size 6,904 bytes but it's being stored with a length of ~22k (inflating the size in CouchDB) so I'm assuming I'm not passing the correct representation (encoding) of the image to CouchDB.
You can encode your image data as a base64 string and save it, although i would not recommend it at all. What I would do is to upload the file to a object storage like AWS S3 or it's open source alternative MinIO, and then save in the DB just a reference to the file (e.g. an Image URL).
P.S.: I'm sorry about the lack of links and references in my answer, I'm writing it on my phone. I can edit it and include references as soon as I'm home.
Azure Blob storage supports a "static web" feature through a forced container name, "$web".
When building a functions app, if you want to save what your function generated inside a blob document, you need to specify a path, but a path with the container name "$web" is considered as invalid.
I'm using Azure functions runtime ~2 using Javascript as language.
The function.json file is shown below.
{
"bindings": [
{
"name": "myTimer",
"type": "timerTrigger",
"direction": "in",
"schedule": "0 0 5 * * *"
},
{
"type": "blob",
"name": "$return",
"path": "$web/index.html",
"connection": "AzureWebJobsStorage",
"direction": "out"
}
]
}
I'd expect the output to generate and save the file to the blob path $web/index.html, but I obtain the following error.
Validation failed for property 'BlobPath', value '$web/index.html'. The field BlobPath is invalid.
Instead of using the standard Azure functions output pipe, I used the Azure node SDK.
I promisified the upload method (which uses callbacks in the V2 SDK).
function upload(svc, container, blob, content) {
return new Promise((res, rej) => svc.createBlockBlobFromText(container, blob, content, (err, resp, _) => {
if (err) {
rej(err);
}
res(resp);
}));
}
I then used it to upload my generated file.
const blobSvc = createBlobService();
await upload(blobSvc, process.env.OUTPUT_CONTAINER, process.env.OUTPUT_BLOB, render(readFileSync('D:/home/site/wwwroot/TimerTrigger1/template.html', 'utf-8'), { user, photos }));
I am trying to create a static website on AWS S3 via the AWS node SDK. I am at the step where I am putting the bucket website. I am calling putBucketWebsite(params = {}, callback) with the following parameters:
{
"Bucket": "xxx.example.com",
"WebsiteConfiguration": {
"IndexDocument": {
"Suffix": "index.html"
},
"RoutingRules": []
}
}
but I am getting the following error:
MalformedXML: The XML you provided was not well-formed or did not validate against our published schema
What am I doing wrong?
When I getBucketWebsite from a site that works, I get:
{
"IndexDocument": {
"Suffix": "index.html"
},
"RoutingRules": []
}
Try removing RoutingRules from your request. As per documentation it requires some properties to be present.