as I patiently wait for Firebase storage to be added to the emulators, I was wondering if there is a way I can avoid modifying live storage files and folders when running hosting / functions in the emulator?
For example I use the following code to delete all the files in a folder. Last night someone accidentally deleted all the documents in our emulator as part of a test and it deleted all the LIVE storage folders as we use an import of real documents into our emulator 🤦
async function deleteStorageFolder(path:string) {
const bucket = admin.storage().bucket();
return bucket.deleteFiles({
prefix: path
})
Is there any way I can tell firebase to avoid using the production storage APIs when emulators are running?
I have used the following condition in my function to prevent using firebase storage API when running in emulator:
if (process.env.FUNCTIONS_EMULATOR == "true") {
console.log(`Running in emulator, won't call firebase storage`)
} else {
// Code goes here to run storage APIs
}
Related
My application makes use of Firestore Function Triggers to perform background actions based on changes in the Firestore, e.g. user profile changes. For example, if they change their mobile number, a verification code is sent.
I have a trigger that should run when an onWrite() event happens on a specific collection. onWrite() runs when any of the following actions occur in Firebase on a specific collection:
onCreate()
onUpdate()
onDelete()
In my usecase, I need it to run for onCreate() and onUpdate(), thus I use onWrite()
For Firebase Triggers to work, a specific format is expected in addition to a document id/wildcard representing a document that was created/changed/deleted.
Constants:
const collections = {
...
conversations: "conversations",
...
}
Callable Function (updates firestore):
/**
* Add an conversation to conversations collection
* #type {HttpsFunction & Runnable<any>}
*/
exports.addConversations = functions.https.onCall(async (data, context) => {
// expects conversation & interested state
const {valid, errors} = validateRequiredBody(data, [
"conversation",
]);
if (!valid) {
return {
status: false,
message: "Missing or invalid parameters",
errors: errors,
jwtToken: "",
};
}
// get conversation item
const conversation = {
id: data["conversation"]["id"],
name: data["conversation"]["name"],
}
// create conversation with empty counter
// let writeResult = await collectionRefs.conversationsRef.doc(conversation.id).set({
let writeResult = await admin.firestore().collection(collections.conversations).doc(conversation.id).set({
id: conversation.id,
name: conversation.name,
count: 0
});
console.log(`[function-addConversations] New Conversation [${conversation.name}] added`);
return {
status: true,
message: ""
}
});
Firestore Trigger (not triggering):
/**
* On conversations updated/removed, update corresponding counter
* #type {CloudFunction<Change<QueryDocumentSnapshot>>}
*/
exports.onConversationProfileCollectionCreate = functions.firestore.document(`${collections.conversations}/{id}`)
.onWrite(async snapshot => {
console.log("Conversation Collection Changed");
// let conversation = collectionRefs.conversationsRef.doc(snapshot.id);
// await conversation.update({count: FieldValue.increment(1)});
});
In my mobile application, the user (calls) the addConversations() firebase function, this adds the new conversation to Firestore which is clearly visible, but the counter trigger (trigger function) doesn't run.
Emulator output:
...
{"verifications":{"app":"MISSING","auth":"MISSING"},"logging.googleapis.com/labels":{"firebase-log-type":"callable-request-verification"},"severity":"INFO","message":"Callable request verification passed"}
[function-addConversations] New Conversation [Test Conversation Topic] added
Profile updated
(print profile data)
...
What I SHOULD expect to see:
...
{"verifications":{"app":"MISSING","auth":"MISSING"},"logging.googleapis.com/labels":{"firebase-log-type":"callable-request-verification"},"severity":"INFO","message":"Callable request verification passed"}
[function-addConversations] New Conversation [Test Conversation Topic] added
Conversation Collection Changed // this is output by the trigger
Profile updated
(print profile data)
...
Did I do something wrong?
The issue was one closer to home.
I am developing using the firebase emulators and connecting to them using the emulators addition in Flutter's firebase packages built in emulator feature.
Firebase emulator setup e.g. Firebase Functions can be found here
TL;DR:
When starting your Firebase emulator, you should see something similar to:
C:\Users\User\MyApp\awesome-app-server>firebase emulators:start --only functions
i emulators: Starting emulators: functions
! functions: You are running the functions emulator in debug mode (port=9229). This means that functions will execute in sequence rather than in parallel.
! functions: The following emulators are not running, calls to these services from the Functions emulator will affect production: auth, firestore, database, hosting, pubsub, storage
! Your requested "node" version "12" doesn''t match your global version "14"
i ui: Emulator UI logging to ui-debug.log
i functions: Watching "C:\Users\User\MyApp\awesome-app-server\functions" for Cloud Functions...
> Debugger listening on ws://localhost:9229/03dc1d62-f2a3-418e-a343-bb0b357f7329
> Debugger listening on ws://localhost:9229/03dc1d62-f2a3-418e-a343-bb0b357f7329
> For help, see: https://nodejs.org/en/docs/inspector
! functions: The Cloud Firestore emulator is not running, so calls to Firestore will affect production.
! functions: The Realtime Database emulator is not running, so calls to Realtime Database will affect production.
...
BUT then you see this - THIS is very important!
i functions[us-central1-api-user-onConversationProfileCollectionCreate ]: function ignored because the database emulator does not exist or is not running.
This, since Firestore & (Realtime) Database use triggers which are found in functions, functions expects to find a local firestore/database emulator.
Since no firestore/database emulators were running locally
! functions: The Cloud Firestore emulator is not running, so calls to Firestore will affect production.
! functions: The Realtime Database emulator is not running, so calls to Realtime Database will affect production.
and these functions don't automagically attach to production Firestore/Database(s) (that would be potentially devestating), these triggers didn't run when I expected them to while emulating locally.
Solution:
Emulate firestore & database locally (see this to import your firestore data to a local emulator) with firebase emulators:start --only functions,...,firestore,database
Upload functions to work with Firestore/Database(s) (please do this with care)
More details:
Below I provide details what lead me to the problem, and how I figured out the issue:
What I was doing:
firebase emulators:start --inspect-functions --only functions,auth
[local] Firebase Functions I was developing and testing the backend for my mobile app using Firebase Functions for various interactivity.
Since Firebase Auth is handled locally through the use of custom tokens on my firebase functions app, I used local auth for testing
I had prepopulated my Firestore with data, thus I intended to use Firestore data while emulating locally which had worked as expected, BUT the firebase firestore & database triggers won't work due to emulating them locally.
I knew some of my triggers DID infact trigger correctly, thus it must be a configuration error of some kind.
Extended Solution (import firestore data to local emulator)
firebase emulators:start --inspect-functions --only functions,auth,firestore,database
notice the last 2, firestore & database - this should be added for triggers to work locally
I noticed when reviewing the logs at startup, the text indicating some functions won't run. This made me realize I was making a crucial mistake - the cause.
Import Data from production firestore
The reason for using production (during development) firestore is to not recreate all the data for each test. The solution is to export from firestore and import you data to the emulators.
See this for details - I wasn't able to export from the terminal, I thus went to Google Console to export to a storage bucket, and download it via the console from there.
To start the emulator (and allow debugging of firebase functions), I use:
firebase emulators:start --inspect-functions --import ./functions/{path/to/data} --only functions,auth,firestore,database
details:
firebase emulators:start - start emulator
--inspect-functions allow debugging functions via websockets - e.g. Webstorm NodeJs debugger on port 9229
--import ./functions/{path/to/data} - import data to firestore emulator using project root as ./
--only functions,auth,firestore,database specify the modules to emulate
You are using the wrong value. snapshot works only for onDelete and onCreate method
exports.useWildcard = functions.firestore
.document('users/{userId}')
.onWrite((change, context) => {
// If we set `/users/marie` to {name: "Marie"} then
context.params.userId == "marie"
// ... and ...
change.after.data() == {name: "Marie"}
});
for more info read here feedbackCloud Firestore triggers
I am trying to use this tool in order to test my NodeJS application :
https://cloud.google.com/sdk/gcloud/reference/beta/emulators/firestore
when I try to add something to the firestore database, I get this error :
Error: Unable to detect a Project Id in the current environment.
It is normal I think because locally there is no GCP Project.
I guess that I must configure firestore :
const Firestore = require('#google-cloud/firestore');
const db = new Firestore({
projectId: 'YOUR_PROJECT_ID',
keyFilename: '/path/to/keyfile.json',
});
When using the Google Cloud Firestore emulator, how may I create a "virtual" project Id ?
Also, is the keyfile mandatory ?
The final goal is to use a local emulator for firestore if a "FIRESTORE_EMULATOR_HOST" env var exist.
You'll need to setup and start the emulator in a separate terminal
gcloud beta emulators firestore start
Then for the project ID you can follow the documentation, and the github instructions:
firebase serve --only firestore
and then
require "google/cloud/firestore"
# Make Firestore use the emulator
ENV["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
firestore = Google::Cloud::Firestore.new project_id: "emulator-project-id"
# Get a document reference
nyc_ref = firestore.doc "cities/NYC"
nyc_ref.set({ name: "New York City" }) # Document created
You can also check this example here
You need to use express.js framework with node js to make requests to #google-cloud/firestore, and best practices will be to start from https://cloud.google.com/nodejs/getting-started?hl=fa
So I have a node js web app, this web app has a folder to store images uploaded by users from a mobile app. How I upload the image to the folder is by using the image's base64 string, and using fs.writeFile to save the image to the folder, like this:
fs.writeFile(__dirname + '/../images/complaintImg/complaintcase_' + data.cID + '.jpg', Buffer.from(data.complaintImage, 'base64'), function (err) {
if (err) {
console.log(err);
} else {
console.log("success");
}
});
The problem is, whenever the application is redeployed to google cloud, the images gets deleted. This is because the image folder of the local version of the application is empty - when the user uploads an image, i don't get a local copy of that image.
How do i prevent the images from getting deleted with every deployment? because the app is constantly updated (changes to js or html files), i can't have the images getting deleted with every deployment. How do i update a deployment to only deploy certain files? the gcloud app deploy command seems to deploy the entire project. or should i upload the images directly to google cloud storage?
please help, currently the mobile app isn't released to the public yet, so having the images deleted with every deployment is still not a big problem now, but it will be once it's released to the public. because the images they upload are very important. thank you in advance!
It appears that your __dirname directory you chose may be under /tmp or, if you use the flexible environment, some other directory local to your instance. If so the images will disappear whenever new instances are started (which always happens at new deployment, but it can happen in between deployments as well). This is expected, the instances are always started "from scratch".
You need to store the files that your app creates and you want to survive instance (re)starts on a persistent storage product, like Cloud Storage, see Using Cloud Storage (or Using Cloud Storage for flexible env). Note that you can't use the regular filesystem calls with Cloud Storage, you need to use the documented client library.
As stated in Dan Cornilescu's answer, for user uploaded files, you should store them in Cloud Storage for GAE Standard or for GAE Flexible.
Just as a reference, there is an alternative for those who are using Python 2.7, Java 8 or PHP 5, which is the BlobStore API
I get the error: "Failed to retrieve function source code" when I try and deploy a function.
This is all from the command line. I am using node 6.11.5 (but in the firebase-admin package.json file in the nodes folder it is says node 6.9.1 is used to download that). I am using firebase-admin#5.8.1 and firebase-functions#0.8.1.
This is the code in my index.js file that I am trying to deploy:
const functions = require('firebase-functions');
exports.helloWorld = functions.https.onRequest((request, response) => {
response.send("Hello from Firebase!");
});
I have also tried to deploy many different things.
Two interesting things:
- I used to be able to deploy any function without problem. This changed about a month ago and now every function I try gets this error. I can't remember making any change that would be related to this.
- Also I can deploy functions from my computer (with the exact same set up and firebase versions) to other projects in the same google account and different google accounts without any problem.
Thanks
I think you should check your billing settings in google cloud. I got the same problem and after updating billing information then redeploy the function, the error is gone.
I have a nodejs app running as a Worker Role in Azure Cloud Services where I have two blob storage containers that the app will read from when responding to certain user requests.
Here's my setup:
Using the azure-storage package to interface with my blob storage.
Two containers, each holding files of different types that the user may ask for at some point.
And I use the following code to stream the files to the HTTP response:
exports.getBlobToStream = function(containerName, fileName, res) {
var blobService = azure.createBlobService();
blobService.getBlobProperties(containerName, fileName, function(error, properties, status){
if(error || !status.isSuccessful)
{
res.header('Content-Type', "text/plain");
res.status(404).send("File " + fileName + " not found");
}
else
{
res.header('Content-Type', properties.contentType);
res.header('Content-Disposition', 'attachment; filename=' + fileName);
blobService.createReadStream(containerName, fileName).pipe(res);
}
});
};
One important
In the past I've had no issues reading from either container. In my research on the problemI've found an identical (but outdated) issue on the all-encompassing azure-sdk-for-node here https://github.com/Azure/azure-sdk-for-node/issues/434. The solution that fixed that problem also fixed mine, but I can't understand why. Particularly when I can read from the other container from within the same app and using the same code without any issues.
I can live with the solution but want to understand what's going on. Any thoughts or suggestions?
#winsome, thanks for raising this issue to us. If you set EMULATED, it means the code will call the local storage emulator other than the real storage account and expected to fail. Regarding the one container works under EMULATED, a guess is your local storage emulator also has a same container. Please check.