How can I delete folder on s3 with node.js? - node.js

Question

Yes, I know. There is no folder concept on s3 storage. but I really want to delete a specific folder from s3 with node.js. I tried two solutions, but both didn't work.
My code is below:
Solution 1:
Deleting folder directly.
var key='level/folder1/folder2/';
var strReturn;
var params = {Bucket: MyBucket};
var s3 = new AWS.S3(params);
s3.client.listObjects({
Bucket: MyBucket,
Key: key
}, function (err, data) {
if(err){
strReturn="{\"status\":\"1\"}";
}else{
strReturn=+"{\"status\":\"0\"}";
}
res.send(returnJson);
console.log('error:'+err+' data:'+JSON.stringify(data));
});
Actually, I have a lot of files under folder2. I can delete single file from folder2 if I define key like this:
var key='level/folder1/folder2/file1.txt', but it didn't work when I deleted a folder(key='level/folder1/folder2/').
Solution 2:
I tried to set expiration to an object when I uploaded this file or folder to s3. code is below:
s3.client.putObject({
Bucket: Camera_Bucket,
Key: key,
ACL:'public-read',
Expires: 60
}
But it didn't either. After finishing uploading, I checked the properties of that file. it showed there was nothing value for expiry date:
Expiry Date:none
Expiration Rule:N/A
How can I delete folder on s3 with node.js?

Answer 1

Here is an implementation in ES7 with an async function and using listObjectsV2 (the revised List Objects API):
async function emptyS3Directory(bucket, dir) {
const listParams = {
Bucket: bucket,
Prefix: dir
};
const listedObjects = await s3.listObjectsV2(listParams).promise();
if (listedObjects.Contents.length === 0) return;
const deleteParams = {
Bucket: bucket,
Delete: { Objects: [] }
};
listedObjects.Contents.forEach(({ Key }) => {
deleteParams.Delete.Objects.push({ Key });
});
await s3.deleteObjects(deleteParams).promise();
if (listedObjects.IsTruncated) await emptyS3Directory(bucket, dir);
}
To call it:
await emptyS3Directory(process.env.S3_BUCKET, 'images/')

Answer 2

You can use aws-sdk module for deleting folder. Because you can only delete a folder when it is empty, you should first delete the files in it. I'm doing it like this :
function emptyBucket(bucketName,callback){
var params = {
Bucket: bucketName,
Prefix: 'folder/'
};
s3.listObjects(params, function(err, data) {
if (err) return callback(err);
if (data.Contents.length == 0) callback();
params = {Bucket: bucketName};
params.Delete = {Objects:[]};
data.Contents.forEach(function(content) {
params.Delete.Objects.push({Key: content.Key});
});
s3.deleteObjects(params, function(err, data) {
if (err) return callback(err);
if (data.IsTruncated) {
emptyBucket(bucketName, callback);
} else {
callback();
}
});
});
}

Answer 3

A much simpler way is to fetch all objects (keys) at that path & delete them. In each call fetch 1000 keys & s3 deleteObjects can delete 1000 keys in each request too. Do that recursively to achieve the goal
Written in typescript
/**
* delete a folder recursively
* #param bucket
* #param path - without end /
*/
deleteFolder(bucket: string, path: string) {
return new Promise((resolve, reject) => {
// get all keys and delete objects
const getAndDelete = (ct: string = null) => {
this.s3
.listObjectsV2({
Bucket: bucket,
MaxKeys: 1000,
ContinuationToken: ct,
Prefix: path + "/",
Delimiter: "",
})
.promise()
.then(async (data) => {
// params for delete operation
let params = {
Bucket: bucket,
Delete: { Objects: [] },
};
// add keys to Delete Object
data.Contents.forEach((content) => {
params.Delete.Objects.push({ Key: content.Key });
});
// delete all keys
await this.s3.deleteObjects(params).promise();
// check if ct is present
if (data.NextContinuationToken) getAndDelete(data.NextContinuationToken);
else resolve(true);
})
.catch((err) => reject(err));
};
// init call
getAndDelete();
});
}
According doc at https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html:
A response can contain CommonPrefixes only if you specify a delimiter.
CommonPrefixes contains all (if there are any) keys between Prefix and the next occurrence of the string specified by the delimiter.
Omitting Delimiter parameter will make ListObject return all keys starting by the Prefix parameter.

Answer 4

According to accepted answer I created promise returned function, so you can chain it.
function emptyBucket(bucketName){
let currentData;
let params = {
Bucket: bucketName,
Prefix: 'folder/'
};
return S3.listObjects(params).promise().then(data => {
if (data.Contents.length === 0) {
throw new Error('List of objects empty.');
}
currentData = data;
params = {Bucket: bucketName};
params.Delete = {Objects:[]};
currentData.Contents.forEach(content => {
params.Delete.Objects.push({Key: content.Key});
});
return S3.deleteObjects(params).promise();
}).then(() => {
if (currentData.Contents.length === 1000) {
emptyBucket(bucketName, callback);
} else {
return true;
}
});
}

Answer 5

The accepted answer throws an error when used in typescript. I made it work by modifying the code in the following way. I'm very new to Typescript but at least it is working now.
async function emptyS3Directory(prefix: string) {
const listParams = {
Bucket: "bucketName",
Prefix: prefix, // ex. path/to/folder
};
const listedObjects = await s3.listObjectsV2(listParams).promise();
if (listedObjects.Contents.length === 0) return;
const deleteParams = {
Bucket: bucketName,
Delete: { Objects: [] as any },
};
listedObjects.Contents.forEach((content: any) => {
deleteParams.Delete.Objects.push({ Key: content.Key });
});
await s3.deleteObjects(deleteParams).promise();
if (listedObjects.IsTruncated) await emptyS3Directory(prefix);
}

Answer 6

Better solution with #aws-sdk/client-s3 module:
private async _deleteFolder(key: string, bucketName: string): Promise<void> {
const DeletePromises: Promise<DeleteObjectCommandOutput>[] = [];
const { Contents } = await this.client.send(
new ListObjectsCommand({
Bucket: bucketName,
Prefix: key,
}),
);
if (!Contents) return;
Contents.forEach(({ Key }) => {
DeletePromises.push(
this.client.send(
new DeleteObjectCommand({
Bucket: bucketName,
Key,
}),
),
);
});
await Promise.all(DeletePromises);
}
ListObjectsCommand returns the keys of files in the folder, even with subfolders

Answer 7

listObjectsV2 list files only with current dir Prefix not with subfolder Prefix. If you want to delete folder with subfolders recursively this is the source code: https://github.com/tagspaces/tagspaces-common/blob/develop/packages/common-aws/io-objectstore.js#L1060
deleteDirectoryPromise = async (path: string): Promise<Object> => {
const prefixes = await this.getDirectoryPrefixes(path);
if (prefixes.length > 0) {
const deleteParams = {
Bucket: this.config.bucketName,
Delete: { Objects: prefixes }
};
return this.objectStore.deleteObjects(deleteParams).promise();
}
return this.objectStore
.deleteObject({
Bucket: this.config.bucketName,
Key: path
})
.promise();
};
/**
* get recursively all aws directory prefixes
* #param path
*/
getDirectoryPrefixes = async (path: string): Promise<any[]> => {
const prefixes = [];
const promises = [];
const listParams = {
Bucket: this.config.bucketName,
Prefix: path,
Delimiter: '/'
};
const listedObjects = await this.objectStore
.listObjectsV2(listParams)
.promise();
if (
listedObjects.Contents.length > 0 ||
listedObjects.CommonPrefixes.length > 0
) {
listedObjects.Contents.forEach(({ Key }) => {
prefixes.push({ Key });
});
listedObjects.CommonPrefixes.forEach(({ Prefix }) => {
prefixes.push({ Key: Prefix });
promises.push(this.getDirectoryPrefixes(Prefix));
});
// if (listedObjects.IsTruncated) await this.deleteDirectoryPromise(path);
}
const subPrefixes = await Promise.all(promises);
subPrefixes.map(arrPrefixes => {
arrPrefixes.map(prefix => {
prefixes.push(prefix);
});
});
return prefixes;
};

Answer 8

You can try this:
import { s3DeleteDir } from '#zvs001/s3-utils'
import { S3 } from 'aws-sdk'
const s3Client = new S3()
await s3DeleteDir(s3Client, {
Bucket: 'my-bucket',
Prefix: `folder/`,
})

Answer 9

I like the list objects and then delete approach, which is what the aws cmd line does behind the scenes btw. But I didn't want to await the list (few seconds) before deleting them. So I use this 1 step (background) process, I found it slightly faster. You can await the child process if you really want to confirm deletion, but I found that took around 10 seconds, so I don't bother I just fire and forget and check logs instead. The entire API call with other stuff now takes 1.5s which is fine for my situation.
var CHILD = require("child_process").exec;
function removeImagesAndTheFolder(folder_name_str, callback){
var cmd_str = "aws s3 rm s3://"
+ IMAGE_BUCKET_STR
+ "/" + folder_name_str
+ "/ --recursive";
if(process.env.NODE_ENV === "development"){
//When not on an EC2 with a role I use my profile
cmd_str += " " + "--profile " + LOCAL_CONFIG.PROFILE_STR;
}
// In my situation I return early for the user. You could make them wait tho'.
callback(null, {"msg_str": "Check later that these images were actually removed."});
//do not return yet still stuff to do
CHILD(cmd_str, function(error, stdout, stderr){
if(error || stderr){
console.log("Problem removing this folder with a child process:" + stderr);
}else{
console.log("Child process completed, here are the results", stdout);
}
});
}

Answer 10

I suggest you to do it in 2 steps, so you can "follow" whats happen (with a progressBar etc...):
Get all keys to remove
Remove keys
Of course , the #1 is a recursive function, such as:
https://gist.github.com/ebuildy/7ac807fd017452dfaf3b9c9b10ff3b52#file-my-s3-client-ts
import { ListObjectsV2Command, S3Client, S3ClientConfig } from "#aws-sdk/client-s3"
/**
* Get all keys recurively
* #param Prefix
* #returns
*/
public async listObjectsRecursive(Prefix: string, ContinuationToken?: string): Promise<
any[]
> {
// Get objects for current prefix
const listObjects = await this.client.send(
new ListObjectsV2Command({
Delimiter: "/",
Bucket: this.bucket.name,
Prefix,
ContinuationToken
})
);
let deepFiles, nextFiles
// Recurive call to get sub prefixes
if (listObjects.CommonPrefixes) {
const deepFilesPromises = listObjects.CommonPrefixes.flatMap(({Prefix}) => {
return this.listObjectsRecursive(Prefix)
})
deepFiles = (await Promise.all(deepFilesPromises)).flatMap(t => t)
}
// If we must paginate
if (listObjects.IsTruncated) {
nextFiles = await this.listObjectsRecursive(Prefix, listObjects.NextContinuationToken)
}
return [
...(listObjects.Contents || []),
...(deepFiles || []),
...(nextFiles || [])
]
}
Then, delete all objects:
public async deleteKeys(keys: string[]): Promise<any[]> {
function spliceIntoChunks(arr: any[], chunkSize: number) {
const res = [];
while (arr.length > 0) {
const chunk = arr.splice(0, chunkSize);
res.push(chunk);
}
return res;
}
const allKeysToRemovePromises = keys.map(k => this.listObjectsRecursive(k))
const allKeysToRemove = (await Promise.all(allKeysToRemovePromises)).flatMap(k => k)
const allKeysToRemoveGroups = spliceIntoChunks(allKeysToRemove, 3)
const deletePromises = allKeysToRemoveGroups.map(group => {
return this.client.send(
new DeleteObjectsCommand({
Bucket: this.bucket.name,
Delete: {
Objects: group.map(({Key}) => {
return {
Key
}
})
}
})
)
})
const results = await Promise.all(deletePromises)
return results.flatMap(({$metadata, Deleted}) => {
return Deleted.map(({Key}) => {
return {
status: $metadata.httpStatusCode,
key: Key
}
})
})
}

Answer 11

According to Emi's answer I made a npm package so you don'
t need to write the code yourself. Also the code is written in typescript.
See https://github.com/bingtimren/s3-commons/blob/master/src/lib/deleteRecursive.ts

Answer 12

You can delete an empty folder the same way you delete a file. In order to delete a non-empty folder on AWS S3, you'll need to empty it first by deleting all files and folders inside. Once the folder is empty, you can delete it as a regular file. The same applies to the bucket deletion. We've implemented it in this app called Commandeer so you can do it from a GUI.