Node.js Google Cloud Storage Get Multiple Files' Metadata - node.js

I have several files on Google Cloud Storage that are named as 0.jpg, 1.jpg, 2.jpg, etc. I want to get the metadata for each file without setting file names separately. Then, want to send these metadata information to React application. In React application, when one clicks on an image, the popup displays the metadata information for this clicked image.
For only one file, I used the following code:
const express = require("express");
const cors = require("cors");
// Imports the Google Cloud client library
const { Storage } = require("#google-cloud/storage");
const bucketName = "bitirme_1";
const filename = "detected/0.jpg";
const storage = new Storage();
const app = express();
app.get("/api/metadata", cors(), async (req, res, next) => {
try {
// Gets the metadata for the file
const [metadata] = await storage
.bucket(bucketName)
.file(filename)
.getMetadata();
const metadatas = [
{id: 0, name: `Date: ${metadata.updated.substring(0,10)}, Time: ${metadata.updated.substring(11,19)}`},
{id: 1, name: metadata.contentType}
];
res.json(metadatas);
} catch (e) {
next(e);
}
});
const port = 5000;
app.listen(port, () => console.log(`Server started on port ${port}`));
I first set the bucket name. Then, set filename array as (89 is the number of files):
const filename = Array(89).fill(1).map((_, i) => ('detected/' + i + '.jpg'));
These files are in detected folder. When I try this, it gives me this error:
Error: No such object: bitirme_1/detected/0.jpg, detected/1.jpg, detected/2.jpg, detected/3.jpg, detected/4.jpg,detected/5.jpg,detected/6.jpg, ....
How can I solve the getting multiple files' metadata issue?
Also, I want to get the number of files in a bucket (or, in the detected folder). I searched the API but cannot found anything. I do not want to enter the total number of files as 89, want to get it from the API.

I found the solution for finding the number of files in a bucket or a folder in a bucket. This is the solution:
const [files] = await storage.bucket(bucketName).getFiles();
const fileStrings = files.map(file => file.name);
const fileSliced = fileStrings.map(el => el.slice(9, 11));
for (i = 0; i < fileSliced.length; i++) {
if (fileSliced[i].includes('.')) {
fileSliced[i] = fileSliced[i].slice(0, 1);
}
}
const fileNumbers = fileSliced.map(function(item) {
return parseInt(item, 10);
});
const numOfFiles = Math.max(...fileNumbers) + 1;
console.log(numOfFiles);
First, I got the all files with file names in a string array. In my case, file names are detected/0.jpg, detected/1.jpg, detected/2.jpg, etc. I just only want to the number part of the file name; hence, I sliced the string array starting from 9th index up to 11th index(not included). As a result, I got only the numbers except one digit numbers.
To handle one digit case, which have '.' at the end of the sliced name, I also removed '.' from these one digit file names.
As a result, I got ['0', '1', '2', '3', ...]. Next, I convert this string array to number array using parseInt function. Finally, to get the number of files, I got the maximum of the array and add 1 to this number.
I have an image detail component that includes location of sender ip, download option and exiting the popup page. This detail popup page opens at /#i where i is the name of image file, such as 1.jpg, 2.jpg. So, for example when I click the first image, the popup page opens at /#1. In this popup page, I want to get metadata information for the opened image. But, I could not find a solution for this.

Related

Node.js Express Temporary File Serving

I'm trying to do a reverse image search using googlethis on an image the user uploads. It supports reverse image searching, but only with a Google-reachable image URL. Currently, I upload the image to file.io, which deletes it after it gets downloaded.
This is the current application flow:
User POSTs file -> Server uploads file to file.io -> Google downloads the file -> Server does things with the reverse image search
However, I want to skip the middleman and have Google download files directly from the server:
User POSTs file -> Server serves file at unique URL -> Google downloads the file -> Server deletes the file -> Server does things with the reverse image search
I've looked at Serving Temporary Files with NodeJs but it just shows how to serve a file at a static endpoint. If I added a route to /unique-url, the route would stay there forever (a very slow memory leak! Probably! I'm not really sure!)
The only way I can think of is to save each file with a UUID and add a parameter: /download?id=1234567890, which would probably work, but if possible, I want to do things in memory.
So:
How do I do this using normal files?
How do I do this in-memory?
Currently working (pseudo) code:
app.post('/', (req, res) => {
const imagePath = saveImageTemporarily(req)
const tempUrl = uploadToFileIo(imagePath)
const reverseImageResults = reverseGoogleSearch(tempUrl)
deleteFile(imagePath)
doThingsWithResults(reverseImageResults).then((result) => { res.send(result) })
}
The other answer is a good one if you are able to use Redis -- it offers lots of helpful features like setting a time-to-live on entries so they're disposed of automatically. But if you can't use Redis...
The basic idea here is that you want to expose a (temporary) URL like example.com/image/123456 from which Google can download an image. You want to store the image in memory until after Google accesses it. So it sounds like there are two (related) parts to this question:
Store the file in memory temporarily
Rather than saving it to a file, why not create a Buffer holding the image data. Once you're done with it, release your reference to the buffer and the Node garbage collector will dispose of it.
let image = Buffer.from(myImageData);
// do something with the image
image = null; // the garbage collector will dispose of it now
Serve the file when Google asks for it
This is a straightforward route which determines which image to serve based on a route parameter. The query parameter you mention will work, and there's nothing wrong with that. Or you could do it as a route parameter:
app.get('/image/:id', (req, res) => {
const id = req.params.id;
res.status(200).send(/* send the image data here */);
});
Putting it all together
It might look something like this:
// store image buffers here
const imageStore = {};
app.post('/image', (req, res) => {
// get your image data here; there are a number of ways to do this,
// so I leave it up to you
const imageData = req.body;
// and generate the ID however you want
const imageId = generateUuid();
// save the image in your store
imageStore[imageId] = imageData;
// return the image ID to the client
res.status(200).send(imageId);
});
app.get('/image/:id', (req, res) => {
const imageId = req.params.id;
// I don't know off the top of my head how to correctly send an image
// like this, so I'll leave it to you to figure out. You'll also need to
// set the appropriate headers so Google recognizes that it's an image
res.status(200).send(imageStore[imageid]);
// done sending? delete it!
delete imageStore[imageId];
});
I would use REDIS for the in-memory DB, and on the server, I would transform the image to base64 to store it in Redis.
In Redis, you can also set TTL on the images.
Check my code below
import {
nanoid
} from 'nanoid'
function base64_encode(file) {
// read binary data
var bitmap = fs.readFileSync(file);
// convert binary data to base64 encoded string
return new Buffer(bitmap).toString('base64');
}
app.post('/', async(req, res) => {
const client = redisClient;
const imagePath = saveImageTemporarily(req)
//const tempUrl = uploadToFileIo(imagePath)
var base64str = base64_encode(imagePath);
const id = nanoid()
await client.set(id, JSON.stringify({
id,
image: base64str
}));
const reverseImageResults = reverseGoogleSearch(JSON.parse(await client.get(id)).image)
await client.del(id);
doThingsWithResults(reverseImageResults).then((result) => {
res.send(result)
})
}

Node Express Fast CSV download to client

I've set up a small node js BE app, built with express and fastCsv module on top of it. The desired outcome would be to be able to download a csv file to the client side, without storing it anywhere inside the server, since the data is generated depending on user criteria.
So far I've been able to get somewhere it it, Im using streams, since that csv file could be pretty large depending on the user selection. Im pretty sure something is missing inside the code bellow:
const fs = require('fs');
const fastCsv = require('fast-csv');
.....
(inside api request)
.....
router.get('/', async(req, res) => {
const gatheredData ...
const filename = 'sometest.csv'
res.writeHead(200, {
'Content-Type': 'text/csv',
'Content-Disposition': 'attachment; filename=' + filename
})
const csvDataStream = fastCsv.write(data, {headers: true}).pipe(res)
})
The above code 'works' in some way as it does deliver back the response, but not the actual file, but the contents of the csv file, which I can view in the preview tab as a response. To sum up, Im trying to stream in that data, into a csv and push it to download file to client, and not store it on the server. Any tips or pointers are very much appreciated.
Here's what worked for me after created a CSV file on the server using the fast-csv package. You need to specify the full, absolute directory path where the output CSV file was created:
const csv = require("fast-csv");
const csvDir = "abs/path/to/csv/dir";
const filename = "my-data.csv";
const csvOutput = `${csvDir}/${filename}`;
console.log(`csvOutput: ${csvOutput}`); // full path
/*
CREATE YOUR csvOutput FILE USING 'fast-csv' HERE
*/
res.type("text/csv");
res.header("Content-Disposition", `attachment; filename="${filename}"`);
res.header("Content-Type", "text/csv");
res.sendFile(filename, { root: csvDir });
You need to make sure to change the response content-type and headers to "text/csv", and try enclosing the filename=... part in double-quotes, like in the above example.

Firebase Storage-How to delete file from storage with node.js?

I want to delete a folder in firebase storage with node js because this is a firebase function.
For example :
storageRef.child(child1).child(child2).delete();
something like this, but firebase documentation doesn't tell anything.
One more question:
When initialize storage documentation node js requires my admin json, but realtime database doesn't want this wonder why?
Have a look at the Node.js client API Reference for Google Cloud Storage and in particular at the delete() method for a File.
You can do it like this using Node.js:
const firebase = require('firebase-admin');
async function deleteImageFromFirebase(imageName) {
await firebase.storage().bucket().file("folderName/"+imageName).delete();
}
And like this client side:
// Create a reference to the file to delete
var desertRef = storageRef.child('images/desert.jpg');
// Delete the file
desertRef.delete().then(function() {
// File deleted successfully
}).catch(function(error) {
// Uh-oh, an error occurred!
});
View this info on the Firebase website:
how to delete files Firebase-storage
This might be late but at least on Web (so basically what you need), there is new API to delete the whole folder.
I tested deleting a folder with 2 pictures inside and it works. I then tried a folder-A with contents: folder-B + picture-A. Folder-B also has a picture-B inside; it still deleted folder-A with all of its contents.
Solution:
const bucket = admin.storage().bucket();
return bucket.deleteFiles({
prefix: `posts/${postId}`
);
I couldn't find this on the official documentation (perhaps is really new API) but really cool article where I found the solution:
Automatically delete your Firebase Storage Files from Firestore with Cloud Functions for Firebase
import { storage } from "./firebaseClient";
import { bucket } from "./firebaseServer";
//Let's assume this is the URL of the image we want to delete
const downloadUrl = "https://storage.googleapis.com/storage/v1/b/<projectID>.appspot.com/o/<location>?"
//firebase delete function
const deleteImages = async ({ downloadUrl }) => {
const httpsRef = storage.refFromURL(downloadUrl).fullPath;
return await bucket
.file(httpsRef)
.delete()
.then(() => "success")
.catch(() => "error")
}
//call the deleteImages inside async function
const deleteStatus = await deleteImages({ downloadUrl: oldImage });
console.log(deleteStatus) //=> "success"

Automatic file upload to a Discord channel

So I'm trying to write a Discord selfbot (e.g. a bot that uses your token to write stuff from your own Discord account), using a Discord.js npm module and so far I'm doing this only for purpose of writing stuff every set interval of time.
I've successfully managed to get it to spam random words (using random-words npm module) and also the contents of a certain text file by lines. The final stage is getting it to upload random files within a certain folder to that channel, every set interval of time. How do I do that?
My code so far (token values and server/channel ids are hidden, naturally :P)
const discord = require("discord.js");
const TOKEN = "";
const bot = new discord.Client();
var fs = require("fs")
var fileContent = fs.readFileSync("text.txt", "utf8");
fileContent=fileContent.split("\n");
var ind=0
bot.on("ready",()=>{
console.log("Ready!");
var server = bot.guilds.find("id","")
var chan = new discord.TextChannel(server,{"id":""});
bot.setInterval(()=>
{
chan.send(fileContent[ind%fileContent.length]).then(msg=>{
console.log(msg.content); ind++;
});
},5000);
})
bot.login(TOKEN);
The current text file content sending feature doesn't have to stay. In fact, ideally I'd only like the bot to have an automatic file upload feature.
You can use glob library to find the list of all files in a specified folder, and then
From
Discord.js - stable release - textChannel#send()
you send files by
var glob = require("glob")
glob("*", {cwd: __dirname + "/folder"}, function (er, files) {
// files is an array of filenames.
// er is an error object or null.
channel.send({
files: files.map(filePath => ({
attachment: filePath,
name: "whateveryouwant"
}))[Math.floor(Math.random() * files.length)]
})
.then(console.log)
.catch(console.error);
})

Delete a file from firebase storage using download url with Cloud Functions

I have a collection of profiles in my Firestore db and a field named "profilePicture" with a downloadUrl as the value.
Im using cloud functions and been trying for a long time to figure out how to delete the profilePicture when the profile is deleted.
I know how to create a trigger when the profile is deleted and get the profile picture downloadUrl, but how do I delete the file from storage with only the downloadUrl?
The firebase storage documentation provides a method refFromURL(url) that can be used on a Storage instance. It states the url argument can be:
A URL in the form:
1) a gs:// URL, for example gs://bucket/files/image.png
2) a download URL taken from object metadata.
Based (2) above, it seems like an HTTP URL should also work. However it probably is better practise to store a path string, as the tokens on the HTTP URLs can get rotated by Firebase.
In Angular I use this to delete file from Cloud Firestore by downloadURL
constructor(private storage: AngularFireStorage) {}
onDeleteAttachment(downloadURL: string) {
this.storage.storage.refFromURL(downloadURL).delete();
}
My understanding is that the node SDK for Cloud Storage can't convert HTTP URLs into file paths within a storage bucket. Instead, you should be storing the file path along with the download URL in document. This will make it possible for to build a File object that can be used to delete the image when it's time to do so.
for admin.storage.Storage is no build in method for get reference from url for storage
but you can extract file path from URL ,by remove baseUrl and do some code replace on URL
im create method for this task to accept url from storage project and return path
function getPathStorageFromUrl(url:String){
const baseUrl = "https://firebasestorage.googleapis.com/v0/b/project-80505.appspot.com/o/";
let imagePath:string = url.replace(baseUrl,"");
const indexOfEndPath = imagePath.indexOf("?");
imagePath = imagePath.substring(0,indexOfEndPath);
imagePath = imagePath.replace("%2F","/");
return imagePath;
}
NOTE : You must replace baseUrl for every project, you can find it by open any image in you storage , and copy it from URL in browser from start to end of last slash '/'
Ex :
Some image link on my storage :
https://firebasestorage.googleapis.com/v0/b/project-80505.appspot.com/o/RequestsScreenshot%2F-M6CA-2bG2aP_WwOF-dR__1i5056O335?alt=media&token=d000fab7
the base URL will be
https://firebasestorage.googleapis.com/v0/b/project-80505.appspot.com/o/
now after get path call file to delete it from storage
const storage = admin.storage();
const imagePath:string = getPathStorageFromUrl(obj.imageUrl);
storage.bucket().file(imagePath).delete().catch((err) => console.error(err));
NOTE : There is no documentation explaining the format of the URL,
which implies that the Firebase team might feel the need to change it
some day , mean maybe will not work in the future if format is change.
confing.js
import firebase from 'firebase/app'
import "firebase/firestore";
import "firebase/storage";
const firebaseConfig = {
apiKey: "XXXX",
authDomain: "XXXXX.firebaseapp.com",
databaseURL: "https://XXXX-app-web.firebaseio.com",
projectId: "XXXX",
storageBucket: "XXXX-app-web.appspot.com",
messagingSenderId: "XXXXXX",
appId: "1:XXX:web:XXXX",
measurementId: "G-XXXX"
};
firebase.initializeApp(firebaseConfig);
export const firestore = firebase.firestore();
export const storageRef = firebase.storage();
export default firebase;
Button.js
import React from 'react';
import {firestore,storageRef} from './Config';
function removeFile(id,downloadUrl) {
const storageRefa = storageRef.refFromURL(downloadUrl);
storageRefa.delete().then(() => {
firestore.collection("All_Files").doc(id).delete().then((response) => {
console.log('delete response', response)
}).catch((error) => {
console.log('delete error', error)
})
}).catch((error) => {
console.log('delete error', error)
});
}
export default function MediaCard(props) {
return (
<>
<Button
onClick={() =>{
removeFile(props.ID,props.downloadUrl)
}}
variant="contained"
color="secondary"
>
Delete
</Button>
</>
);
}
Mahmoud's answer need a little edit .. it works tho .. He is doing the replacements wrongly and might not work if you have nested directories or spaced filenames in your storage
getPathStorageFromUrl(url:String){
const baseUrl = "https://firebasestorage.googleapis.com/v0/b/project-80505.appspot.com/o/";
let imagePath:string = url.replace(baseUrl,"");
const indexOfEndPath = imagePath.indexOf("?");
imagePath = imagePath.substring(0,indexOfEndPath);
imagePath = imagePath.replace(/%2F/g,"/");
imagePath = imagePath.replace(/%20/g," ");
return imagePath;
}

Resources