Why Firebase Firestore count reads operations while I am just adding new documents only? - node.js

I have created and an API endpoint with Firebase Functions usign node.js. This
API endpoint collect JSON data from client browser and I am saving that JSON data to Firebase Firestore databse using Firebase Functions.
While this works fine but when I see Firestore usage tab it's shows really high number of read operations even I have not created any read function till now.
My API is in Production and and current usage data is : Reads 9.7K, Writes 1K, Deletes 0.
I have already tried to check with Firebase Firestore Document and Pricing but never seems to find anything on this issue.
I am using Firestore add function to create document with an auto generated document id. ValidateSubscriberData() is a simple function to validate client req.body inputs which is JSON data.
app.post('/subscribe', (req, res) => {
let subscriber = {};
ValidateSubscriberData(req.body)
.then(data => {
subscriber = data;
//console.log(data);
subscriber.time = Date.now();
return subscriber;
})
.then(subscriber => {
//console.log(subscriber);
// noinspection JSCheckFunctionSignatures
return db.collection(subscriber.host).add(subscriber);
})
.then(document => {
console.log(document.id);
res.json({id: document.id, iid: subscriber.iid});
return 0;
})
.catch(error => {
console.log({SelfError: error});
res.json(error);
});
});
I don't know this is an issue with Firestore or I am doing something in a way that makes read operations internally but I want find a way so I can optimize my code.
English is not my first language and I am trying my best explain my issue.

I think Firestore is working perfectly fine and my code too. I assume Firebase is counting those reads which I made through Firebase Console.
To verify this I have clicked on Data tab on Firestore page and scroll down to make all document name/id visible. And after that I see 1K Reads added on my old stats. So its proven Firestore counting all reads even its from firebase console and it is obvious but my bad I have not thinking about this before.
I don't think this question has any relevance but may be people like me find it helpful before posting any silly question on this helpful platform.

Related

How to create a Flutter Stream using MongoDB (watch collection?) with Firebase Cloud Function

I've been trying out MongoDB as database for my Flutter project lately, since I want to migrate from pure Firebase database (some limitations in Firebase are an issue for my project, like the "in-array" limit of 10 for queries).
I already made some CRUD operations methods in some Firebase Cloud Functions, using MongoDB. I'm now able to save data and display it as a Future in a Flutter App (a simple ListView of Users in a FutureBuilder).
My question is : how would it be possible to create a StreamBuilder thanks to MongoDB and Firebase Cloud Functions ? I saw some stuff about watch collection and Stream change but nothing clear enough for me (usually I read a lot of examples or tutorial to understand).
Maybe some of you would have some clues or maybe tutorial that I can read/watch to learn a little bit more about that subject ?
For now, I have this as an example (NodeJS Cloud Function stored in Firebase), which obviously produces a Future in my Future app (not realtime) :
exports.getUsers = functions.https.onCall(async (data, context) => {
const uri = "mongodb+srv://....";
const client = new MongoClient(uri);
await client.connect();
var results = await client.db("myDB").collection("user").find({}).toArray();
await client.close();
return results;
});
What would you advice me to obtain a Stream instead of a Future, using maybe watch collection and Stream change from MongoDB, providing example if possible !
Thank you very much !
Cloud Functions are meant for short-lived operations, not for long-term listeners. It is not possible to create long-lived connections from Cloud Functions, neither to other services (such as you're trying to do to MongoDB here) nor from Cloud Functions back to the calling client.
Also see:
If I implement onSnapshot real-time listener to Firestore in Cloud Function will it cost more?
Can a Firestore query listener "listen" to a cloud function?
the documentation on EventArc, which is the platform that allows you build custom triggers. It'll be (a lot* more. involved though.

How do use transloadit addStream() function in the NodeJS SDK?

Trying out the transloadit api, the template works when I use the testing mode on the transloadit website, but when I try to use it in Node JS with the SDK I'm getting an error:
INVALID_FORM_DATA - https://api2.transloadit.com/assemblies - INVALID_FORM_DATA: The form contained bad data, which cannot be parsed.
The relevant code: (_asset.content) is a Buffer object
async function getThumbnailUrl(_assetkey: string, _asset: I.FormFile): Promise<string> {
let tOptions = {
waitForCompletion: true,
params: {
template_id: process.env.THUMB_TRANSLOADIT_TEMPLATE,
},
};
const stream = new Readable({
read() {
this.push(_asset.content);
this.push(null);
},
});
console.log(_asset.content);
util.transloadit.addStream(_assetkey, stream);
return new Promise((resolve, reject) => {
util.transloadit.createAssembly(tOptions, (err, status) => {
if (err) {
reject(err);
}
console.log(status);
//return status;
resolve(status);
});
});
}
I noticed that you also posted this question on the Transloadit forums - so in the case that anyone else runs into this problem you can find more information on this topic here.
Here's a work-around that the OP found that may be useful:
Just to provide some closure to this topic, I just tested my
workaround (upload to s3, then use import s3 robot to grab the file)
and got it to work with the nodejs sdk so i should be good using that.
I have a suspicion the error I was getting was not to do with the
transloadit api, but rather the form-data library for node js
(https://github.com/form-data/form-data 1) and that’s somehow not
inputting the form data in the way that the transloadit api is
expecting.
But as there aren’t alternatives to that library that I could find, I
wasn’t really able to test that hypothesis.
The Transloadit core team also gave this response regarding the issue:
It may try to set his streams to be Tus streams which would mean that
they’re not uploaded as multipart/form data.
In either case it seems like the error to his callback would be
originating from the error out of _remoteJson
These could be the problem areas
https://github.com/transloadit/node-sdk/blob/master/src/TransloaditClient.js#L146
https://github.com/transloadit/node-sdk/blob/master/src/TransloaditClient.js#L606
https://github.com/transloadit/node-sdk/blob/master/src/TransloaditClient.js#L642
It is also possible that the form-data library could be the source of
the error
To really test this further we’re going to need to try using the
library he was using, make sure the output of it is good, and then
debug the node-sdk to see where the logic failure is in it, or if the
logic failure is on the API side.

Cloud Firestore big data error - Deadline Exceeded [duplicate]

I would like to load collection that is ~30k records. I.e load it via.
const db = admin.firestore();
let documentsArray: Array<{}> = [];
db.collection(collection)
.get()
.then(snap => {
snap.forEach(doc => {
documentsArray.push(doc);
});
})
.catch(err => console.log(err));
This will always throw Deadline Exceeded error. I have searched for some sorts of mechanism that will allow me to paginate trough it but I find it unbelievable not to be able to query for not that big amount in one go.
I was thinking that it may be that due to my rather slow machine I was hitting the limit but then I deployed simple express app that would do the fetching to app engine and still had no luck.
Alternatively I could also export the collection with gcloud beta firestore export but it does not provide JSON data.
I'm not sure about firestore, but on datastore i was never able to fetch that much data in one shot, I'd always have fetch pages of about 1000 records at a time and build it up in memory before processing it. You said:
I have searched for some sorts of mechanism that will allow me to paginate trough
Perhaps you missed this page
https://cloud.google.com/firestore/docs/query-data/query-cursors
In the end the issue was that machine that was processing the 30k records from the Firestore was not powerful enough to get the data needed in time. Solved by using, GCE with n1-standard-4 GCE.

How do i reference Firebase's realtime database from within a cloud function?

In a Firebase Cloud function i am attempting to retrieve data from my Realtime Database and write an update to it as well. This is where i am having trouble.
The function:
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
...
api.post('/messages/add', (request: any, response: any) => {
const dbRef:any = functions.database.ref(`count/${request.body.chatRoomId}`);
dbRef.once('value').then((snapshot:any) => {
let messageCount:number = (snapshot.val() && snapshot.val().messages) || 0;
messageCount = Number(messageCount + 1);
dbRef.update({
messages: messageCount,
updated: admin.database().database.ServerValue.TIMESTAMP
});
});
});
When i call this function from the client, in the firebase cloud console logs, i am seeing the following error:
TypeError: dbRef.once is not a function
I have tried referencing the database in each of the following ways, all of which fail:
functions.database.ref(`count/${request.body.chatRoomId}`).once('value').then((snapshot)) =>
functions.database.ref(`count/${request.body.chatRoomId}`).once('value', (snapshot) =>
admin.database().ref(`count/${request.body.chatRoomId}`).once('value').then((snapshot)) =>
admin.database().ref(`count/${request.body.chatRoomId}`).once('value', (snapshot) =>
When attempting the reference using the Admin SDK via admin.database() i get a different error:
TypeError: Cannot read property 'ServerValue' of undefined
at dbRef.once.then
Lastly, i can confirm that the value for ${request.body.chatRoomId} is resolving correctly, i had tested this by printing its value to the console.
The answer here shows how to reference the realtime database from the 'event' response within a realtime database listener like onWrite() for example, however my function is not triggered by realtime database changes, but rather is an endpoint reachable from the client.
UPDATE
The answer proposed by Doug Stevenson below is correct but there was also an additional error due to attempting to write a timestamp incorrectly to a field that i had initially excluded which was causing my function to fail despite having tried his solution, thus I have updated the code to reflect this.
First, the answer is to reference the Realtime Database with admin.database()
Second, even though i had tried this, i was still seeing an error, but the error was due to attempting to set a timestamp incorrectly. Thanks to Frank van Puffelen for pointing this out.
This: admin.database().database.ServerValue.TIMESTAMP
Needs to be: admin.database.ServerValue.TIMESTAMP
And with that it works.
You're making the mistake of trying to use the functions SDK to query the database. This is not actually creating a database reference:
functions.database.ref(`count/${request.body.chatRoomId}`);
The functions SDK is just used for declaring and configuring Cloud Functions.
What you should be doing instead is using the Admin SDK to create a reference, then query it:
admin.database().ref(...).once(...)
You also need to initialize the Admin SDK exactly once before you use it:
admin.initializeApp();

Creating realtime project with Hapi, MongoDb, NodeJS & AngularJS

I'm creating my first web application using Hapi, MongoDb, NodeJS & Angular. I can already get and save some data from my app.
Now each time I need to refresh the data in the browser, I must restart the server. Because data is only fetched on server start, using the following function:
var getUsers = function(db, callback) {
db.collection("users", function(err, collection) {
if (err) return callback(err, "error getting collection");
collection.find({}).toArray(function(err, users) {
if (err) return callback(err, "error getting find()");
console.log("returning users list: ");
Common.setUsers(users);
//console.log(Common.getUsers());
});
});
};
The users can then be fetched via Common.getUsers()
Now what if I want to create a list that shows me - in realtime - which users register for my application? So without a need for server restart. How can I achieve this?
I've done a bit of searching and found RethinkDb, which has changefeeds. (I wish I found this earlier). MongoDb doesn't have changefeeds the way Rethink implements it.
Is there a way that I can create these changefeeds myself? Or is there another tool that can achieve this for me which works with MongoDb?
A little late but I created a playground application which shows how to build a realtime timeline with hapi and the nes plugin. It also utilizes RethinkDB and makes use of their changefeed feature:
Realtime timeline with hapi.js, nes and RethinkDB
Maybe this also helps you a little bit.

Resources