I need a utility function that can copy a single document in Firestore from one collection to another.
This excellent answer provides a way to copy a collection. But I can't work out how to modify it to copy a single document.
For instance I have this current structure:
collection1/abc123000000
The document abc123000000 has fields name and email with content Joe Bloggs and joe#bloggs.com respectively.
I wish to copy the xyz123000001 document from collection1 and all its fields and data to a new document in collection2:
collection2/xyz910110000
I would happily just run the command from the terminal to achieve this.
Ideally of course, it would be useful to have a function that looped through and copied all documents from one collection to the other dependent on the content of a field!
Many thanks in advance for any help.
[Original question title edited to assist in future searches as extra info added into the answer.]
Yo can do this by reading the collection, iterating on it and for each element of the collection 1 write it in collection 2.
Here is a quick example:
function copy(db){
db.collection('collection1').get()
.then((snapshot) => {
snapshot.forEach((doc) => {
// We are iterating on the documents of the collection
let data = doc.data();
console.log(doc.id, '=>', doc.data());
if(<PUT_CONDITIONS_HERE>){
//we have read the document till here
let setDoc = db.collection('collection2').doc(doc.id).set(data);
setDoc.then(res => {
console.log('Set: ', res);
});
}
});
})
.catch((err) => {
console.log('Error getting documents', err);
});
}
For more examples on how to read and write using the nodejs CLI you can go to the Github repository here
Also this can be done with a query from collection one to filter at that level, and iterate over less files. However this depends on the conditions you have to determine if it needs to be copied or not.
Many thanks to José Soní and to Lahiru Chandima on this post about copying collections for giving me the key bits of information which allowed me to solve this - outstandingly helpful!
I've found it really frustrating not having all the bits of the puzzle to solve this issue...so I am posting a heavily commented version below which I hope will be of use to anyone coming after. Apologies to anyone who already knows all this stuff...this answer is not for you ;-)
// Create a file with this code.
// In your Firestore DB, create the destination collection.
// const firebaseUrl refers to your databaseUrl which you can find in the project settings of your Firebase console.
// Save as filename.js within the directory where you have initialised Firebase.
// Ensure Node.js is installed and that node is available, try node --version
// Then run node filename.js from the terminal.
const firebaseAdmin = require('firebase-admin');
const serviceAccount = '../../firebase-service-account-key.json';
const firebaseUrl = 'https://my-app.firebaseio.com';
firebaseAdmin.initializeApp({
credential: firebaseAdmin.credential.cert(require(serviceAccount)),
databaseURL: firebaseUrl
});
const db = firebaseAdmin.firestore();
function copy(db){
db.collection('collectionName').get()
.then((snapshot) => {
snapshot.forEach((doc) => {
// We are iterating on the documents of the collection
let data = doc.data();
console.log(doc.id, '=>', doc.data());
if(doc.id == 'randomDocIdAssignedByFirestore'){
// We have read the document till here
//From here: we create a new document in the collection
// Change some of the data in the fields in the new document
let id = 'newMeaningfulDocId'; // Instead of allowing Firestore to create a random ID
data.title = 'Meaningful Title'; // Write new data into the field called title
data.description = 'Meaningful Description'; // Write new data into the field called description
/*
We are using a Firestore Reference type field in our DB to reference a parent doc.
If you just used data.parent = '/category/OS_EnDept'; you would end up writing a string field type.
However, you want to use a Reference Type AND you want the parent to be collectionName
do it like this:
data.parent = doc;
We, however, want to be able to write in different parent collection names - hence the next line.
*/
data.parent = db.collection('collectionName').doc('desiredParentDocId')
let setDoc = db.collection('collectionName').doc(id).set(data);
setDoc.then(res => {
console.log('Set: ', res);
});
}
});
})
.catch((err) => {
console.log('Error getting documents', err);
});
}
// Call the function when we run this file...
copy(db);
Related
I am trying to get all of the documents in one collection named 'repeated_tasks' in my Firebase Function.
I tried using the following code:
Is it possible to get all documents in a Firestore Cloud Function?, but this does not seem to work for me. I am trying to get the information, so that I can update every document in the collection, to set one field, to false. I have the following code:
exports.finishedUpdate = functions.pubsub.schedule('0 3 * * *').timeZone('Europe/Amsterdam').onRun((context) => {
// This is part of the above mentioned stack question
var citiesRef = database.collection('repeated_tasks');
const snapshot = citiesRef.get();
snapshot.forEach(doc => {
console.log(doc.id, '=>', doc.data());
});
// A way to update all of the documents in the repeated_tasks collection has to be found
// This part works, for only the two given document ids
var list = ['qfrxHTZAJZTJDQpA83fjsM03159438695', 'qfrxHTZAJZQTpM3pA83fjsM0315217389'];
for (var i = 0; i < list.length; i++) {
database.doc('repeated_tasks/' + list[i]).update({'finished': false});
}
return console.log("Done");
})
Help is much appreciated, as I can't seem to find any related information anywhere, except for the stack overflow page, which didn't help. I am using Node JS (Javascript) to set the functions.
By using the syntax of getting the information from Firestore, I was able to also update it in Firebase Functions and could update all, using the following code:
const reference = database.collection('repeated_tasks/');
const snapshot = await reference.where('finished', '==', true).get();
if (snapshot.empty) {
console.log('no matching documents');
return;
}
snapshot.forEach(doc => {
database.doc('repeated_tasks/' + doc.id).update({'finished': false});
});
I found this solution that helps to solve my problem:
function makeMenu(clientId){
let clientButton = [Markup.callbackButton('📗 '+clientId+' Information', 'info-'+clientId)]
return Markup.inlineKeyboard([clientButton])
}
bot.action(/^[client]+(-[a-z]+)?$/, ctx => {
console.log(ctx.match[1].split('-')[1] )
})
But this is a poor solution or a workaround because I need to pass a long list of parameters and there is a limitation in the telegram's api to pass strings up to 64 bytes.
One solution to the issue you're facing would be to put the large data inside a database and passing an Id (or a ref to that data) as the callback data and using the code you've posted.
An example code would be:
function makeMenu(clientId){
const id = storeDataToDB('info-'+clientId) // store the large data to DB in here
let clientButton = [Markup.callbackButton('📗 '+clientId+' Information', id)]
return Markup.inlineKeyboard([clientButton])
}
bot.action(/^[client]+(-[a-z]+)?$/, ctx => {
const data = getDataFromDB(ctx.match[1].split('-')[1]) // fetch the data from DB and continue..
console.log(data)
})
You could use firebase, mongodb, or any other DB.. (just make sure the ID adheres to the limit imposed by telegram)
I try to create a simple login form and fail to validate the password from the MongoDB.
First I create the .post route for the form validation and then I get the MongoDB data which I want to compare with the form.
Here is my code:
app.post('/users', (req, res) => {
const reqUser = req.body.params.name
const reqPW = req.body.params.password
// connect to mongoDB
const collection = client.db().collection("users")
collection.find({name: reqUser}).toArray(function (err, results) {
if (err) {
console.log(err)
res.send([])
return
}
else {
console.log('RESULT', results) // returns the object
console.log('RES PW', results.password) // returns undefined
// this does not work
Object.keys(results).forEach(function(key) {
console.log('key is: ', key); // returns 0
});
// validate user+pw
if (!reqUser || !reqPW/*|| reqPW !== password*/) {
return res.status(401).end()
}
// send result to frontend
res.send(results)
res.end
}
})
})
So, I get my object returned in results but I cannot get the data from the object.
I also tried to convert it to an array with Array.from() but that didn't work either.
Please note that I did not yet implement hashing and salting the passwords yet, as I thought I want a working validation first. Do I need to implement those first?
I just checked the doc:
The toArray() method returns an array that contains all the documents from a cursor. The method iterates completely the cursor, loading all the documents into RAM and exhausting the cursor.
So toArray() will return a array, not object, therefore your results will be an array containing all the items(object) you get from the db. If you console.log(results), it should print an array rather than object.
Assuming there won't be two users have the same name, the results you get will be just an array containing one object, so you can just do:
results[0].password // get the first object's password field
Not sure if this slove your question, but based on your code thats the problem i found in it.
Is it possible to query firestore documents by updateTime. The field that is available from the document snapshot as doc.updateTime and use it in a where query?
I am using the node.js sdk.
As far as I know there is no way to query on the metadata that Firestore automatically maintains. If you need to query the last update date, you will need to add a field with that value to the document's data.
I really need to query Firebase on document _updateTime, so I wrote a function that copies that hidden internal timestamp to a queryable field. This took some work to figure this out, so I am posting the complete solution. (Technically, this is "Cloud Firestore" rather then "Realtime Database".)
This is done using Firebase Functions, which itself took some tries to get working. This tutorial was helpful:
https://firebase.google.com/docs/functions/get-started
However, on Windows 10, the only command line that worked was the new Bash shell, available since about 2017. This was something of a runaround to install, but necessary. The GIT Bash shell, otherwise very useful, was not able to keep track of screen positions during Firebase project setup.
In my example code, I have left in all the 'console.log' statements, to show detail. Not obvious at first was where these logs go. They do not go to the command line, but to the Firebase console:
https://console.firebase.google.com/u/0/
under (yourproject) > Functions > Logs
For testing, I found it useful to, at first, deploy only one function (this is in the CLI):
firebase deploy --only functions:testFn
Below is my working function, heavily commented, and with some redundancy for illustration. Replace 'PlantSpp' with the name of your collection of documents:
// The Cloud Functions for Firebase SDK to create Cloud Functions and setup triggers.
const functions = require('firebase-functions');
// The Firebase Admin SDK to access the Firebase Realtime Database.
const admin = require('firebase-admin');
admin.initializeApp();
// Firestore maintains an interal _updateTime for every document, but this is
// not queryable. This function copies that to a visible field 'Updated'
exports.makeUpdateTimeVisible = functions.firestore
.document('PlantSpp/{sppId}')
.onWrite((sppDoc, context) => {
console.log("Event type: ", context.eventType);
// context.eventType = 'google.firestore.document.write', so cannot use
// to distinguish e.g. create from update
const docName = context.params.sppId // this is how to get the document name
console.log("Before: ", sppDoc.before); // if a create, a 'DocumentSnapshot',
// otherwise a 'QueryDocumentSnapshot'
// if a create, everything about sppDoc.before is undefined
if (typeof sppDoc.before._fieldsProto === "undefined"){
console.log('document "', docName, '" has been created');
// set flags here if desired
}
console.log("After: ", sppDoc.after); // if a delete, a 'DocumentSnapshot',
// otherwise a 'QueryDocumentSnapshot'
// if a delete, everything about sppDoc.after is undefined
if (typeof sppDoc.after._fieldsProto === "undefined"){
console.log('document "', docName, '" has been deleted');
// other fields could be fetched from sppDoc.before
return null; // no need to proceed
}
console.log(sppDoc.after.data()); // the user defined fields:values
// inside curly braces
console.log(sppDoc.after._fieldsProto); // similar to previous except with
// data types, e.g.
// data() has { Code: 'OLDO',...
// _fieldsProto has { Code: { stringValue: 'OLDO' },...
const timeJustUpdated = sppDoc.after._updateTime; // this is how to get the
// internal nonqueryable timestamp
console.log(timeJustUpdated);
// e.g. Timestamp { _seconds: 1581615533, _nanoseconds: 496655000 }
// later: Timestamp { _seconds: 1581617552, _nanoseconds: 566223000 }
// shows this is correctly updating
// see if the doc has the 'Updated' field yet
if (sppDoc.after._fieldsProto.hasOwnProperty('Updated')) {
console.log("doc has the field 'Updated' with the value",
sppDoc.after._fieldsProto.Updated);
console.log("sppDoc:", sppDoc);
const secondsInternal = timeJustUpdated._seconds;
console.log(secondsInternal, "seconds, internal timestamp");
const secondsExternal = sppDoc.after.data().Updated._seconds;
console.log(secondsExternal, "seconds, external timestamp");
// Careful here. If we just update the externally visible time to the
// internal time, we will go into an infinite loop because that update
// will call this function again, and by then the internal time will have
// advanced
// the following exit will not work:
if (secondsInternal === secondsExternal) return null; // will never exit
// instead, allow the external time to lag the internal by a little
const secondsLate = secondsInternal - secondsExternal;
if (secondsLate < 120) { // two minutes sufficient for this purpose
console.log("the field 'Updated' is", secondsLate,
"seconds late, good enough");
return null;
}
console.log("the field 'Updated' is", secondsLate,
"seconds late, updating");
// return a promise of a set operation to update the timestamp
return sppDoc.after.ref.set({
Updated: timeJustUpdated
}, {merge: true}); // 'merge' prevents overwriting whole doc
// this change will call this same function again
} else { // field 'Updated' does not exist in the document yet
// this illustrates how to add a field
console.log("doc does not have the field 'Updated', adding it now.");
// return a promise of a set operation to create the timestamp
return sppDoc.after.ref.set({
Updated: timeJustUpdated
}, {merge: true}); // 'merge' prevents overwriting the whole doc
// this change will call this same function again
}
});
True, there is no query for time created/modified, but when you fetch a document those fields exist in the payload. You have:
payload.doc['_document'].proto.createTime and payload.doc['_document'].proto.updateTime
Sure it's not good practice to rely on private fields, so will prolly need ongoing adjustments as Firestore changes its data model, but for now, for my uses, it gets me this otherwise un-query-able data.
Have a very basic understanding of the Typescript language, but would like to know, how can I copy multiple documents from one firestore database collection to another collection?
I know how to send the request from the app's code along with the relevant data (a string and firebase auth user ID), but unsure about the Typescript code to handle the request...
Thats a very broad question, but something like this can move data in moderate sizes from one collection to another:
import * as _ from 'lodash';
import {firestore} from 'firebase-admin';
export async function moveFromCollection(collectionPath1: string, collectionPath2: string): void {
try {
const collectionSnapshot1Ref = firestore.collection(collectionPath1);
const collectionSnapshot2Ref = firestore.collection(collectionPath2);
const collectionSnapshot1Snapshot = await collectionSnapshot1Ref.get();
// Here we get all the snapshots from collection 1. This is ok, if you only need
// to move moderate amounts of data (since all data will be stored in memory)
// Now lets use lodash chunk, to insert data in batches of 500
const chunkedArray = _.chunk(collectionSnapshot1Snapshot.docs, 500);
// chunkedArray is now an array of arrays, with max 500 in each
for (const chunk of chunkedArray) {
const batch = firestore.batch();
// Use the batch to insert many firestore docs
chunk.forEach(doc => {
// Now you might need some business logic to handle the new address,
// but maybe something like this is enough
const newDocRef = collectionSnapshot2Ref.doc(doc.id);
batch.set(newDocRef, doc.data(), {merge: false});
});
await batch.commit();
// Commit the batch
}
console.log('Done!');
} catch (error) {
console.log(`something went wrong: ${error.message}`);
}
}
But maybe you can tell more about the use case?