I'm trying to archive pdf files from remote websites to Google Cloud Storage using a google function triggered by a firebase write.
The code below works. However, this function copies the remote file to the bucket root.
I'd like to copy the pdf to the pth of the bucket: library-xxxx.appspot.com/Orgs/${params.ukey}.
How to do this?
exports.copyFiles = functions.database.ref('Orgs/{orgkey}/resources/{restypekey}/{ukey}/linkDesc/en').onWrite(event => {
const snapshot = event.data;
const params = event.params;
const filetocopy = snapshot.val();
if (validFileType(filetocopy)) {
const pth = 'Orgs/' + params.orgkey;
const bucket = gcs.bucket('library-xxxx.appspot.com')
return bucket.upload(filetocopy)
.then(res => {
console.log('res',res);
}).catch(err => {
console.log('err', err);
});
}
});
Let me begin with a brief explanation of how GCS file system works: as explained in the documentation of Google Cloud Storage, GCS is a flat name space where the concept of directories does not exist. If you have an object like gs://my-bucket/folder/file.txt, this means that there is an object called folder/file.txt stored in the root directory of gs://my-bucket, i.e. the object name includes / characters. It is true that the GCS UI in the Console and the gsutil CLI tool make the illusion of having a hierarchical file structure, but this is only to provide more clarity for the user, even though those directories do not exist, and everything is stored in a "flat" name space.
That being said, as described in the reference for the storage.bucket.upload() method, you can specify an options parameter containing the destination field, where you can specify a string with the complete filename to use.
Just as an example (note the options paramter difference between both functions):
var bucket = storage.bucket('my-sample-bucket');
var options = {
destination: 'somewhere/here.txt'
};
bucket.upload('sample.txt', function(err, file) {
console.log("Created object gs://my-sample-bucket/sample.txt");
});
bucket.upload('sample.txt', options, function(err, file) {
console.log("Created object gs://my-sample-bucket/somewhere/here.txt");
});
So in your case you can build a string containing the complete name that you want to use (containing also the "directory" structure you have in mind).
filepath --> local machine file storage path
await bucket.upload(filepath, {
public: true,
gzip: true,
metadata: {
cacheControl: "public, max-age=31536000",
},
});
Related
This is likely a duh mistake but I can't figure this out.
I'm successfully uploading images to a bucket with a signed URL. When trying to delete the object from my Express backend, using the below code from Google's example, I get Not Found, yet the object is there with the correct name. Thoughts?
async function deleteFile(filename) {
console.log(filename); // correct file name as exists in bucket
try {
await storage
.bucket(bucketName) // correct bucket name and subfolder 'my-image-bucket/posts'
.file(filename)
.delete();
} catch (e) {
console.log('Error message = ', e.message); // Not Found
}
}
The only red flag I'm seeing is, "correct bucket name and subfolder 'my-image-bucket/posts'" next to .bucket(). You should only be passing the bucket name to .bucket() and then the full path to .file().
const bucketName = 'my-image-bucket';
const filename = 'posts/image.jpg';
await storage
.bucket(bucketName)
.file(filename)
.delete();
I am following a tutorial to resize images via Cloud Functions on upload and am experiencing two major issues which I can't figure out:
1) If a PNG is uploaded, it generates the correctly sized thumbnails, but the preview of them won't load in Firestorage (Loading spinner shows indefinitely). It only shows the image after I click on "Generate new access token" (none of the generated thumbnails have an access token initially).
2) If a JPEG or any other format is uploaded, the MIME type shows as "application/octet-stream". I'm not sure how to extract the extension correctly to put into the filename of the newly generated thumbnails?
export const generateThumbs = functions.storage
.object()
.onFinalize(async object => {
const bucket = gcs.bucket(object.bucket);
const filePath = object.name;
const fileName = filePath.split('/').pop();
const bucketDir = dirname(filePath);
const workingDir = join(tmpdir(), 'thumbs');
const tmpFilePath = join(workingDir, 'source.png');
if (fileName.includes('thumb#') || !object.contentType.includes('image')) {
console.log('exiting function');
return false;
}
// 1. Ensure thumbnail dir exists
await fs.ensureDir(workingDir);
// 2. Download Source File
await bucket.file(filePath).download({
destination: tmpFilePath
});
// 3. Resize the images and define an array of upload promises
const sizes = [64, 128, 256];
const uploadPromises = sizes.map(async size => {
const thumbName = `thumb#${size}_${fileName}`;
const thumbPath = join(workingDir, thumbName);
// Resize source image
await sharp(tmpFilePath)
.resize(size, size)
.toFile(thumbPath);
// Upload to GCS
return bucket.upload(thumbPath, {
destination: join(bucketDir, thumbName)
});
});
// 4. Run the upload operations
await Promise.all(uploadPromises);
// 5. Cleanup remove the tmp/thumbs from the filesystem
return fs.remove(workingDir);
});
Would greatly appreciate any feedback!
I just had the same problem, for unknown reason Firebase's Resize Images on purposely remove the download token from the resized image
to disable deleting Download Access Tokens
goto https://console.cloud.google.com
select Cloud Functions from the left
select ext-storage-resize-images-generateResizedImage
Click EDIT
from Inline Editor goto file FUNCTIONS/LIB/INDEX.JS
Add // before this line (delete metadata.metadata.firebaseStorageDownloadTokens;)
Comment the same line from this file too FUNCTIONS/SRC/INDEX.TS
Press DEPLOY and wait until it finish
note: both original and resized will have the same Token.
I just started using the extension myself. I noticed that I can't access the image preview from the firebase console until I click on "create access token"
I guess that you have to create this token programatically before the image is available.
I hope it helps
November 2020
In connection to #Somebody answer, I can't seem to find ext-storage-resize-images-generateResizedImage in GCP Cloud Functions
The better way to do it, is to reuse the original file's firebaseStorageDownloadTokens
this is how I did mine
functions
.storage
.object()
.onFinalize((object) => {
// some image optimization code here
// get the original file access token
const downloadtoken = object.metadata?.firebaseStorageDownloadTokens;
return bucket.upload(tempLocalFile, {
destination: file,
metadata: {
metadata: {
optimized: true, // other custom flags
firebaseStorageDownloadTokens: downloadtoken, // access token
}
});
});
I am using firebase cloud function storage for the first time.
I succeeded to
perform changes on the default folder by using :
exports.onFileUpload = functions.storage.bucket().object().onFinalize(data => {
const bucket = data.bucket;
const filePath = data.name;
const destBucket = admin.storage().bucket(bucket);
const file = destBucket.file(filePath);
but now I want the function to be triggered from a folder inside the storage
folder like this
How can I do that?
There is currently no way to configure trigger conditions for certain file paths, similar to what you can do with database triggers.
i.e. you can not set a cloud storage trigger for 'User_Pictures/{path}'
What you have to do is to inspect the object attributes once the function is triggered and handle it accordingly there.
Either you create a trigger function for each case you want to handle and stop the function if it's not the path you're looking for.
functions.storage.object().onFinalize((object) => {
if (!object.name.startsWith('User_Pictures/')) {
console.log(`File ${object.name} is not a user picture. Ignoring it.`);
return null;
}
// ...
})
Or you do a master handling function that dispatches the processing to different functions
functions.storage.object().onFinalize((object) => {
if (object.name.startsWith('User_Pictures/')) {
return handleUserPictures(object);
} else if (object.name.startsWith('MainCategoryPics/')) {
return handleMainCategoryPictures(object);
}
})
I have a collection of URLs that may or may not belong to a particular bucket. These are not public.
I'm using the nodejs aws-sdk to get them.
However, the getObject function needs params Bucket and Key separately, which are already in my URL.
Is there any way I can use the URL?
I tried extracting the key by splitting URL with / and getting bucket by splitting with .. But the problem is the bucket name can also have . and I'm not sure if key name can have / as well.
The amazon-s3-uri library can parse out the Amazon S3 URI:
const AmazonS3URI = require('amazon-s3-uri')
try {
const uri = 'https://bucket.s3-aws-region.amazonaws.com/key'
const { region, bucket, key } = AmazonS3URI(uri)
} catch((err) => {
console.warn(`${uri} is not a valid S3 uri`) // should not happen because `uri` is valid in that example
})
use this module parse-s3-url to set the parameter for the getObject.
bucket.getObject( parseS3Url('https://s3.amazonaws.com/mybucket/mykey'), (err:any, data:any) =>{
if (err) {
// alert("Failed to retrieve an object: " + error);
} else {
console.log("Loaded " + data.ContentLength + " bytes");
// do something with data.Body
}
});
To avoid installing a package.
const objectUrl = 'https://s3.us-east-2.amazonaws.com/my-s3-bucket/some-prefix/file.json'
const { host, pathname } = new URL(objectUrl);
const [, region] = /s3.(.*).amazon/.exec(host)
const [, bucket, key] = pathname.split('/')
I If you are using NodeJS GCS client library and want to list directories in your bucket, how do you do that?
First add the dependency for the NodeJS GCS client library into your package.json file by running:
npm -i #google-cloud/storage --save
Then add this into your code to list all files:
const storage = require('#google-cloud/storage');
...
const projectId = '<<<<<your-project-id-here>>>>>';
const gcs = storage({
projectId: projectId
});
let bucketName = '<<<<<your-bucket-name-here>>>>>';
let bucket = gcs.bucket(bucketName);
bucket.getFiles({}, (err, files,apires) => {console.log(err,files,apires)});
This will return all files with full path into files.
To list only directories you must workaround a quirk in the client library that requires you to use no auto pagination and then returns an extra argument to the CB. To do so change the code to this:
let cb=(err, files,next,apires) => {
console.log(err,files,apires);
if(!!next)
{
bucket.getFiles(next,cb);
}
}
bucket.getFiles({delimiter:'/', autoPaginate:false}, cb);
This will return a list of directories under the root path with trailing / in apires.prefixes.
To list only directories under foo/ directory use this code:
let cb=(err, files,next,apires) => {
console.log(err,files,apires);
if(!!next)
{
bucket.getFiles(next,cb);
}
}
bucket.getFiles({prefix:'foo/', delimiter:'/', autoPaginate:false}, cb);