I am trying to develop a MS Teams bot that sends content to students module(unit) wise. I have created 3 classes:
methods.js = Contains all the methods for sending texts, attachments etc.
teamBot.js = Captures a specific keyword from the users and based on that executes a function.
test.js = Connects the bot with Airtable and sends the content accordingly
I am facing Cannot perform 'get' on a proxy that has been revoked error. I figured it might be because of the context. I am passing context as a parameter, which I feel might not be the correct way, how can I achieve the result, and retain the context between files.
teamsBot.js
const test = require("./test");
class TeamsBot extends TeamsActivityHandler {
constructor() {
super();
// record the likeCount
this.likeCountObj = { likeCount: 0 };
this.onMessage(async (context, next) => {
console.log("Running with Message Activity.");
let txt = context.activity.text;
// const removedMentionText = TurnContext.removeRecipientMention(context.activity);
// if (removedMentionText) {
// // Remove the line break
// txt = removedMentionText.toLowerCase().replace(/\n|\r/g, "").trim();
// }
// Trigger command by IM text
switch (txt) {
case "Begin": {
await test.sendModuleContent(context)
}
// By calling next() you ensure that the next BotHandler is run.
await next();
});
// Listen to MembersAdded event, view https://learn.microsoft.com/en-us/microsoftteams/platform/resources/bot-v3/bots-notifications for more events
this.onMembersAdded(async (context, next) => {
const membersAdded = context.activity.membersAdded;
for (let cnt = 0; cnt < membersAdded.length; cnt++) {
if (membersAdded[cnt].id) {
const card = cardTools.AdaptiveCards.declareWithoutData(rawWelcomeCard).render();
await context.sendActivity({ attachments: [CardFactory.adaptiveCard(card)] });
break;
}
}
await next();
});
}
test.js
const ms = require('./methods')
async function sendModuleContent(context) {
data = module_text //fetched from Airtable
await ms.sendText(context, data)
}
methods.js
const {TeamsActivityHandler, ActivityHandler, MessageFactory } = require('botbuilder');
async function sendText(context, text){
console.log("Sending text")
await context.sendActivity(text);
}
Refer this: TypeError: Cannot perform 'get' on a proxy that has been revoked
make the following changes to test.js
const {
TurnContext
} = require("botbuilder");
var conversationReferences = {};
var adapter;
async function sendModuleContent(context) {
data = module_text //fetched from Airtable
const currentUser = context.activity.from.id;
conversationReferences[currentUser] = TurnContext.getConversationReference(context.activity);
adapter = context.adapter;
await adapter.continueConversation(conversationReferences[currentUser], async turnContext => {
await turnContext.sendActivity(data);
});
}
I am trying to integrate PayPal server-side payment on a website with a new v2 since v1 is deprecated. Since I had to include quite a lot of code from their git, i will post here everything for a working example
In v1 i could do it really easily:
app.get('/create', function (req, res) {
//build PayPal payment request
let payReq = JSON.stringify({
'intent': 'sale',
'redirect_urls': {
'return_url': 'http://localhost:8081/process',
'cancel_url': 'http://localhost:8081/cancel'
},
'payer': {
'payment_method': 'paypal'
},
'transactions': [{
'amount': {
'total': '7.47',
'currency': 'USD'
},
'description': 'This is the payment transaction description.'
}]
});
paypal.payment.create(payReq, function (error, payment) {
if (error) {
console.error(error);
} else {
//capture HATEOAS links
let links = {};
payment.links.forEach(function (linkObj) {
links[linkObj.rel] = {
'href': linkObj.href,
'method': linkObj.method
};
})
//if redirect url present, redirect user
if (links.hasOwnProperty('approval_url')) {
res.redirect(links['approval_url'].href);
} else {
console.error('no redirect URI present');
}
}
});
});
app.get('/process', function (req, res) {
let paymentId = req.query.paymentId;
let payerId = {'payer_id': req.query.PayerID};
paypal.payment.execute(paymentId, payerId, function (error, payment) {
if (error) {
console.error(error);
} else {
if (payment.state == 'approved') {
const payerCountry = payment.payer.payer_info.country_code;
const total = payment.transactions[0].amount.total;
const currency = payment.transactions[0].amount.currency;
savePayment(payerCountry, total, currency);
res.send('payment completed successfully ' + cnt++);
} else {
res.send('payment not successful');
}
}
});
});
The create endpoint basically creates the order, the paypal API returns hateos links and the controller says the browser should redirect to that link. Once redirected , if user approves payment on paypal site he is redirected to on of
'redirect_urls': {
'return_url': 'http://localhost:8081/process',
'cancel_url': 'http://localhost:8081/cancel'
},
process endpoints retrieves PAYMENT ID and PAYER ID from query and captures the order - allowing me to do whatever i want to do ( e.g save payment in db ) in callback.
Now v2 seems like huge mess:
Following devs guide i have created
Paypal button (copy paste ):
import React, {useEffect} from "react";
import {PayPalButtons, PayPalScriptProvider, usePayPalScriptReducer} from "#paypal/react-paypal-js";
// This values are the props in the UI
const amount = "2";
const currency = "USD";
const style = {"layout": "vertical"};
const ButtonWrapper = ({currency, showSpinner}) => {
// usePayPalScriptReducer can be use only inside children of PayPalScriptProviders
// This is the main reason to wrap the PayPalButtons in a new component
const [{options, isPending}, dispatch] = usePayPalScriptReducer();
useEffect(() => {
dispatch({
type: "resetOptions",
value: {
...options,
currency: currency,
},
});
}, [currency, showSpinner]);
const createOrder = (data, actions) => {
console.log("create")
return fetch('http://localhost:8081/create', {
method: 'post'
}).then(function (res) {
return res.json();
}).then(function (orderData) {
console.log(orderData);
window.location = orderData.redirect;
});
}
// Call your server to finalize the transaction
const onApprove = (data, actions) => {
console.log("approve")
return fetch('/process', {
method: 'post'
}).then(function (res) {
return res.json();
}).then(function (orderData) {
// Three cases to handle:
// (1) Recoverable INSTRUMENT_DECLINED -> call actions.restart()
// (2) Other non-recoverable errors -> Show a failure message
// (3) Successful transaction -> Show confirmation or thank you
// This example reads a v2/checkout/orders capture response, propagated from the server
// You could use a different API or structure for your 'orderData'
var errorDetail = Array.isArray(orderData.details) && orderData.details[0];
if (errorDetail && errorDetail.issue === 'INSTRUMENT_DECLINED') {
return actions.restart(); // Recoverable state, per:
// https://developer.paypal.com/docs/checkout/integration-features/funding-failure/
}
if (errorDetail) {
var msg = 'Sorry, your transaction could not be processed.';
if (errorDetail.description) msg += '\n\n' + errorDetail.description;
if (orderData.debug_id) msg += ' (' + orderData.debug_id + ')';
return alert(msg); // Show a failure message (try to avoid alerts in production environments)
}
// Successful capture! For demo purposes:
console.log('Capture result', orderData, JSON.stringify(orderData, null, 2));
var transaction = orderData.purchase_units[0].payments.captures[0];
alert('Transaction ' + transaction.status + ': ' + transaction.id + '\n\nSee console for all available details');
});
}
return (<>
{(showSpinner && isPending) && <div className="spinner"/>}
<PayPalButtons
style={style}
disabled={false}
forceReRender={[amount, currency, style]}
fundingSource={undefined}
createOrder={(data, actions) => createOrder(data, actions)}
onApprove={(data, actions) => onApprove(data, actions)}
/>
</>
);
}
export default function PayPalButton() {
return (
<div style={{ maxWidth: "750px", minHeight: "200px" }}>
<PayPalScriptProvider
options={{
"client-id": "test",
components: "buttons",
currency: "USD"
}}
>
<ButtonWrapper
currency={currency}
showSpinner={false}
/>
</PayPalScriptProvider>
</div>
);
}
And then followed the flow from paypal's github example
created their HttpClient
const checkoutNodeJssdk = require('#paypal/checkout-server-sdk');
/**
* Returns PayPal HTTP client instance with environment which has access
* credentials context. This can be used invoke PayPal API's provided the
* credentials have the access to do so.
*/
function client() {
return new checkoutNodeJssdk.core.PayPalHttpClient(environment());
}
/**
* Setting up and Returns PayPal SDK environment with PayPal Access credentials.
* For demo purpose, we are using SandboxEnvironment. In production this will be
* LiveEnvironment.
*/
function environment() {
let clientId = process.env.PAYPAL_CLIENT_ID || '<clientId>';
let clientSecret = process.env.PAYPAL_CLIENT_SECRET || '<secret>';
if (process.env.NODE_ENV === 'production') {
return new checkoutNodeJssdk.core.LiveEnvironment(clientId, clientSecret);
}
return new checkoutNodeJssdk.core.SandboxEnvironment(clientId, clientSecret);
}
async function prettyPrint(jsonData, pre=""){
let pretty = "";
function capitalize(string) {
return string.charAt(0).toUpperCase() + string.slice(1).toLowerCase();
}
for (let key in jsonData){
if (jsonData.hasOwnProperty(key)){
if (isNaN(key))
pretty += pre + capitalize(key) + ": ";
else
pretty += pre + (parseInt(key) + 1) + ": ";
if (typeof jsonData[key] === "object"){
pretty += "\n";
pretty += await prettyPrint(jsonData[key], pre + "\t");
}
else {
pretty += jsonData[key] + "\n";
}
}
}
return pretty;
}
module.exports = {client: client, prettyPrint:prettyPrint};
created their CreateOrder:
/**
* PayPal SDK dependency
*/
const checkoutNodeJssdk = require('#paypal/checkout-server-sdk');
/**
* PayPal HTTP client dependency
*/
const payPalClient = require('./PayPalClient');
/**
* Setting up the JSON request body for creating the Order. The Intent in the
* request body should be set as "CAPTURE" for capture intent flow.
*
*/
function buildRequestBody() {
return {
"intent": "CAPTURE",
"application_context": {
"return_url": "http://localhost:8081/process",
"cancel_url": "https://www.example.com",
"locale": "en-US",
"landing_page": "BILLING",
"user_action": "CONTINUE"
},
"purchase_units": [
{
"soft_descriptor": "Donation",
"amount": {
"currency_code": "USD",
"value": "220.00"
}
}
]
};
}
/**
* This is the sample function which can be sued to create an order. It uses the
* JSON body returned by buildRequestBody() to create an new Order.
*/
async function createOrder(debug=false) {
try {
const request = new checkoutNodeJssdk.orders.OrdersCreateRequest();
request.headers["prefer"] = "return=representation";
request.requestBody(buildRequestBody());
const response = await payPalClient.client().execute(request);
if (debug){
console.log("Status Code: " + response.statusCode);
console.log("Status: " + response.result.status);
console.log("Order ID: " + response.result.id);
console.log("Intent: " + response.result.intent);
console.log("Links: ");
response.result.links.forEach((item, index) => {
let rel = item.rel;
let href = item.href;
let method = item.method;
let message = `\t${rel}: ${href}\tCall Type: ${method}`;
console.log(message);
});
console.log(`Gross Amount: ${response.result.purchase_units[0].amount.currency_code} ${response.result.purchase_units[0].amount.value}`);
// To toggle print the whole body comment/uncomment the below line
console.log(JSON.stringify(response.result, null, 4));
}
return response;
}
catch (e) {
console.log(e)
}
}
/**
* This is the driver function which invokes the createOrder function to create
* an sample order.
*/
if (require.main === module){
(async() => await createOrder(true))();
}
/**
* Exports the Create Order function. If needed this can be invoked from the
* order modules to execute the end to flow like create order, retrieve, capture
* and refund(Optional)
*/
module.exports = {createOrder:createOrder};
And endpoints:
const createUsersOrder = async (res) => {
try {
let response = await createOrder();
console.log("Creating Order...");
let orderId = "";
if (response.statusCode === 201){
console.log("Created Successfully");
orderId = response.result.id;
console.log("Links:");
response.result.links.forEach((item, index) => {
let rel = item.rel;
let href = item.href;
let method = item.method;
let message = `\t${rel}: ${href}\tCall Type: ${method}`;
console.log(message);
});
let links = {};
response.result.links.forEach(function (linkObj) {
links[linkObj.rel] = {
'href': linkObj.href,
'method': linkObj.method
};
})
//if redirect url present, redirect user
if (links.hasOwnProperty('approve')) {
var returnObj = {redirect : links['approve'].href}
console.log("Returning " + returnObj)
res.send(returnObj);
} else {
console.error('no redirect URI present');
}
}
console.log("Copy approve link and paste it in browser. Login with buyer account and follow the instructions.\nOnce approved hit enter...");
return
} catch (error) {
console.log('There was an error: ', error);
}
};
app.post("/create", function(req,res) {
createUsersOrder(res);
})
Here, this is called when button is clicked, as "createOrder" method is called. I create order just like in the v1 code, and then redirect browser to the url. However when the user approves transaction on paypal, and thus is being redirected to one of
"application_context": {
"return_url": "http://localhost:8081/process",
"cancel_url": "http://localhost:8081/cancel",
"locale": "en-US",
"landing_page": "BILLING",
"user_action": "CONTINUE"
},
return url ( /process route for success ), the request DOES NOT contain payment_id, only PAYER_ID. But the payment_id ( or order_id in v2 ) is needed to capture the order.
Since i have found literally zero blogs, tutorials, guide for v2 ( only millions for v1) i am confused where to get the order id. Do i really need to save it in DB after i create the order? OR is there any other trick?
Also, the button contains onApprove method, but after creating order, the browser is redirected to paypal, and the paypal redirects user to http://localhost:8081/process endpoint - thus the onApprove method is never invoked and useless (?).
This whole flow of v2 is really confusing, is there something i am missing?
Thanks for help
With your v2 code, do not use any redirects. At all. Your button should call 2 endpoints on your server. These two endpoints should (respectively) do the API operations of creating and capturing the order, and return the JSON result in each case (the capture route can do any server-side operations like storing the transaction result in the database before forwarding the JSON result to the client caller, since the client needs to handle any capture error situations as well as showing a success message). You can find a full stack node.js example in the PayPal integration guide, but it's fine to keep your #paypal/react-paypal-js code pretty much as-is for the front end.
Trying out my first NodeJS cloud function so far unsuccessfully despite working fine VS code. Getting following error
Function cannot be initialized. Error: function terminated.
Looking through the logs I see some potential issues
Detailed stack trace: ReferenceError: supabase_public_url is not defined
Provided module can't be loaded (doesn't specify)
Thoughts: Am I doing it wrong with the secret manager and using the pub/sub incorrect?
My Code index.js
import { createClient } from '#supabase/supabase-js'
import sgMail from "#sendgrid/mail"
import { SecretManagerServiceClient } from '#google-cloud/secret-manager'
//activate cloud secret manager
const client = new SecretManagerServiceClient()
const supabaseUrl = client.accessSecretVersion(supabase_public_url)
const supabaseKey = client.accessSecretVersion(supabase_service_key)
const sendgridKey = client.accessSecretVersion(sendgrid_service_key)
sgMail.setApiKey(sendgridKey)
const supabase = createClient(supabaseUrl, supabaseKey)
// get data for supabase where notifications coins are true
const supabaseNotifications = async() => {
let { data, error } = await supabase
.from('xxx')
.select('*, xxx!inner(coin, xx, combo_change, combo_signal, combo_prev_signal), xxx!inner(email)')
.eq('crypto_signals.combo_change', true)
if(error) {
console.error(error)
return
}
return data
}
//create an array of user emails from supabase data
const userEmail = (data) => {
try {
const emailList = []
for (let i of data) {
if (emailList.includes(i.profiles.email) != true) {
emailList.push(i.profiles.email)
} else {}
}
return emailList
}
catch(e) {
console.log(e)
}
}
// function to take email list and supabase data to generate emails to users
const sendEmail = (e, data ) => {
try {
for (let i of e) {
const signalList = []
for (let x of data) {
if(i == x.profiles.email) {
signalList.push(x)
} else {}
}
// create msg and send from my email to the user
const msg = {
to: i,
from:"xxxx",
subject: "Coin notification alert from CryptoOwl",
text: "One or more of you coins have a new signal",
html: signalList.toString()
}
sgMail.send(msg)
console.log(i)
}
}
catch(e) {
console.log(e)
}
}
// main function combines all 3 functions (supabase is await)
async function main(){
let supabaseData = await supabaseNotifications();
let supabaseEmails = userEmail(supabaseData);
let sendgridEmails = sendEmail(supabaseEmails, supabaseData);
}
exports.sendgridNotifications = (event, context) => {
main()
};
my package.json with type module to use import above
{
"type":"module",
"dependencies":{
"#sendgrid/mail":"^7.6.1",
"#supabase/supabase-js":"1.30.0",
"#google-cloud/secret-manager": "^3.11.0"
}
}
I'm not at all versed in Google Secret Manager but a rapid look at the Node.js library documentation shows (if I'm not mistaking) that accessSecretVersion() is an asynchronous method.
As a matter of facts, we find in the doc examples like the following one:
async function accessSecretVersion() {
const [version] = await client.accessSecretVersion({
name: name,
});
// Extract the payload as a string.
const payload = version.payload.data.toString();
// WARNING: Do not print the secret in a production environment - this
// snippet is showing how to access the secret material.
console.info(`Payload: ${payload}`);
}
See https://cloud.google.com/secret-manager/docs/samples/secretmanager-access-secret-version#secretmanager_access_secret_version-nodejs
How can you delete a Document with all it's collections and nested subcollections? (inside the functions environment)
In the RTDB you can ref.child('../someNode).setValue(null) and that completes the desired behavior.
I can think of two ways you could achieve the desired delete behavior, both with tremendously ghastly drawbacks.
Create a 'Super' function that will spider every document and delete them in a batch.
This function would be complicated, brittle to changes, and might take a lengthy execution time.
Add 'onDelete' triggers for each Document type, and make it delete any direct subcollections. You'll call delete on the root document, and the deletion calls will propagate down the 'tree'. This is sluggish, scales atrociously and is costly due to the colossal load of function executions.
Imagine you would have to delete a 'GROUP' and all it's children. It would be deeply chaotic with #1 and pricey with #2 (1 function call per doc)
groups > GROUP > projects > PROJECT > files > FILE > assets > ASSET
> urls > URL
> members > MEMBER
> questions > QUESTION > answers > ANSWER > replies > REPLY
> comments > COMMENT
> resources > RESOURCE > submissions > SUBMISSION
> requests > REQUEST
Is there a superior/favored/cleaner way to delete a document and all it's nested subcollections?
It ought to be possible considering you can do it from the console.
according to firebase documentation:
https://firebase.google.com/docs/firestore/solutions/delete-collections
Deleting collection with nested subcollections might be done easy and neat with node-JS on the server side.
const client = require('firebase-tools');
await client.firestore
.delete(collectionPath, {
project: process.env.GCLOUD_PROJECT,
recursive: true,
yes: true
});
Unfortunately, your analysis is spot on and indeed this use case does require a lot of ceremony. According to official documentation, there is no support for deep deletes in a single shot in firestore neither via client libraries nor rest-api nor the cli tool.
The cli is open sourced and its implementation lives here: https://github.com/firebase/firebase-tools/blob/master/src/firestore/delete.js. They basically implemented option 1. you described in your question, so you can take some inspiration from there.
Both options 1. and 2. are far from ideal situation and to make your solution 100% reliable you will need to keep a persistent queue with deletion tasks, as any error in the long running procedure will leave your system in some ill-defined state.
I would discourage to go with raw option 2. as recursive cloud function calls may very easily went wrong - for example, hitting max. limits.
In case the link changed, below the full source of https://github.com/firebase/firebase-tools/blob/master/src/firestore/delete.js:
"use strict";
var clc = require("cli-color");
var ProgressBar = require("progress");
var api = require("../api");
var firestore = require("../gcp/firestore");
var FirebaseError = require("../error");
var logger = require("../logger");
var utils = require("../utils");
/**
* Construct a new Firestore delete operation.
*
* #constructor
* #param {string} project the Firestore project ID.
* #param {string} path path to a document or collection.
* #param {boolean} options.recursive true if the delete should be recursive.
* #param {boolean} options.shallow true if the delete should be shallow (non-recursive).
* #param {boolean} options.allCollections true if the delete should universally remove all collections and docs.
*/
function FirestoreDelete(project, path, options) {
this.project = project;
this.path = path;
this.recursive = Boolean(options.recursive);
this.shallow = Boolean(options.shallow);
this.allCollections = Boolean(options.allCollections);
// Remove any leading or trailing slashes from the path
if (this.path) {
this.path = this.path.replace(/(^\/+|\/+$)/g, "");
}
this.isDocumentPath = this._isDocumentPath(this.path);
this.isCollectionPath = this._isCollectionPath(this.path);
this.allDescendants = this.recursive;
this.parent = "projects/" + project + "/databases/(default)/documents";
// When --all-collections is passed any other flags or arguments are ignored
if (!options.allCollections) {
this._validateOptions();
}
}
/**
* Validate all options, throwing an exception for any fatal errors.
*/
FirestoreDelete.prototype._validateOptions = function() {
if (this.recursive && this.shallow) {
throw new FirebaseError("Cannot pass recursive and shallow options together.");
}
if (this.isCollectionPath && !this.recursive && !this.shallow) {
throw new FirebaseError("Must pass recursive or shallow option when deleting a collection.");
}
var pieces = this.path.split("/");
if (pieces.length === 0) {
throw new FirebaseError("Path length must be greater than zero.");
}
var hasEmptySegment = pieces.some(function(piece) {
return piece.length === 0;
});
if (hasEmptySegment) {
throw new FirebaseError("Path must not have any empty segments.");
}
};
/**
* Determine if a path points to a document.
*
* #param {string} path a path to a Firestore document or collection.
* #return {boolean} true if the path points to a document, false
* if it points to a collection.
*/
FirestoreDelete.prototype._isDocumentPath = function(path) {
if (!path) {
return false;
}
var pieces = path.split("/");
return pieces.length % 2 === 0;
};
/**
* Determine if a path points to a collection.
*
* #param {string} path a path to a Firestore document or collection.
* #return {boolean} true if the path points to a collection, false
* if it points to a document.
*/
FirestoreDelete.prototype._isCollectionPath = function(path) {
if (!path) {
return false;
}
return !this._isDocumentPath(path);
};
/**
* Construct a StructuredQuery to find descendant documents of a collection.
*
* See:
* https://firebase.google.com/docs/firestore/reference/rest/v1beta1/StructuredQuery
*
* #param {boolean} allDescendants true if subcollections should be included.
* #param {number} batchSize maximum number of documents to target (limit).
* #param {string=} startAfter document name to start after (optional).
* #return {object} a StructuredQuery.
*/
FirestoreDelete.prototype._collectionDescendantsQuery = function(
allDescendants,
batchSize,
startAfter
) {
var nullChar = String.fromCharCode(0);
var startAt = this.parent + "/" + this.path + "/" + nullChar;
var endAt = this.parent + "/" + this.path + nullChar + "/" + nullChar;
var where = {
compositeFilter: {
op: "AND",
filters: [
{
fieldFilter: {
field: {
fieldPath: "__name__",
},
op: "GREATER_THAN_OR_EQUAL",
value: {
referenceValue: startAt,
},
},
},
{
fieldFilter: {
field: {
fieldPath: "__name__",
},
op: "LESS_THAN",
value: {
referenceValue: endAt,
},
},
},
],
},
};
var query = {
structuredQuery: {
where: where,
limit: batchSize,
from: [
{
allDescendants: allDescendants,
},
],
select: {
fields: [{ fieldPath: "__name__" }],
},
orderBy: [{ field: { fieldPath: "__name__" } }],
},
};
if (startAfter) {
query.structuredQuery.startAt = {
values: [{ referenceValue: startAfter }],
before: false,
};
}
return query;
};
/**
* Construct a StructuredQuery to find descendant documents of a document.
* The document itself will not be included
* among the results.
*
* See:
* https://firebase.google.com/docs/firestore/reference/rest/v1beta1/StructuredQuery
*
* #param {boolean} allDescendants true if subcollections should be included.
* #param {number} batchSize maximum number of documents to target (limit).
* #param {string=} startAfter document name to start after (optional).
* #return {object} a StructuredQuery.
*/
FirestoreDelete.prototype._docDescendantsQuery = function(allDescendants, batchSize, startAfter) {
var query = {
structuredQuery: {
limit: batchSize,
from: [
{
allDescendants: allDescendants,
},
],
select: {
fields: [{ fieldPath: "__name__" }],
},
orderBy: [{ field: { fieldPath: "__name__" } }],
},
};
if (startAfter) {
query.structuredQuery.startAt = {
values: [{ referenceValue: startAfter }],
before: false,
};
}
return query;
};
/**
* Query for a batch of 'descendants' of a given path.
*
* For document format see:
* https://firebase.google.com/docs/firestore/reference/rest/v1beta1/Document
*
* #param {boolean} allDescendants true if subcollections should be included,
* #param {number} batchSize the maximum size of the batch.
* #param {string=} startAfter the name of the document to start after (optional).
* #return {Promise<object[]>} a promise for an array of documents.
*/
FirestoreDelete.prototype._getDescendantBatch = function(allDescendants, batchSize, startAfter) {
var url;
var body;
if (this.isDocumentPath) {
url = this.parent + "/" + this.path + ":runQuery";
body = this._docDescendantsQuery(allDescendants, batchSize, startAfter);
} else {
url = this.parent + ":runQuery";
body = this._collectionDescendantsQuery(allDescendants, batchSize, startAfter);
}
return api
.request("POST", "/v1beta1/" + url, {
auth: true,
data: body,
origin: api.firestoreOrigin,
})
.then(function(res) {
// Return the 'document' property for each element in the response,
// where it exists.
return res.body
.filter(function(x) {
return x.document;
})
.map(function(x) {
return x.document;
});
});
};
/**
* Progress bar shared by the class.
*/
FirestoreDelete.progressBar = new ProgressBar("Deleted :current docs (:rate docs/s)", {
total: Number.MAX_SAFE_INTEGER,
});
/**
* Repeatedly query for descendants of a path and delete them in batches
* until no documents remain.
*
* #return {Promise} a promise for the entire operation.
*/
FirestoreDelete.prototype._recursiveBatchDelete = function() {
var self = this;
// Tunable deletion parameters
var readBatchSize = 7500;
var deleteBatchSize = 250;
var maxPendingDeletes = 15;
var maxQueueSize = deleteBatchSize * maxPendingDeletes * 2;
// All temporary variables for the deletion queue.
var queue = [];
var numPendingDeletes = 0;
var pagesRemaining = true;
var pageIncoming = false;
var lastDocName;
var failures = [];
var retried = {};
var queueLoop = function() {
if (queue.length == 0 && numPendingDeletes == 0 && !pagesRemaining) {
return true;
}
if (failures.length > 0) {
logger.debug("Found " + failures.length + " failed deletes, failing.");
return true;
}
if (queue.length <= maxQueueSize && pagesRemaining && !pageIncoming) {
pageIncoming = true;
self
._getDescendantBatch(self.allDescendants, readBatchSize, lastDocName)
.then(function(docs) {
pageIncoming = false;
if (docs.length == 0) {
pagesRemaining = false;
return;
}
queue = queue.concat(docs);
lastDocName = docs[docs.length - 1].name;
})
.catch(function(e) {
logger.debug("Failed to fetch page after " + lastDocName, e);
pageIncoming = false;
});
}
if (numPendingDeletes > maxPendingDeletes) {
return false;
}
if (queue.length == 0) {
return false;
}
var toDelete = [];
var numToDelete = Math.min(deleteBatchSize, queue.length);
for (var i = 0; i < numToDelete; i++) {
toDelete.push(queue.shift());
}
numPendingDeletes++;
firestore
.deleteDocuments(self.project, toDelete)
.then(function(numDeleted) {
FirestoreDelete.progressBar.tick(numDeleted);
numPendingDeletes--;
})
.catch(function(e) {
// For server errors, retry if the document has not yet been retried.
if (e.status >= 500 && e.status < 600) {
logger.debug("Server error deleting doc batch", e);
// Retry each doc up to one time
toDelete.forEach(function(doc) {
if (retried[doc.name]) {
logger.debug("Failed to delete doc " + doc.name + " multiple times.");
failures.push(doc.name);
} else {
retried[doc.name] = true;
queue.push(doc);
}
});
} else {
logger.debug("Fatal error deleting docs ", e);
failures = failures.concat(toDelete);
}
numPendingDeletes--;
});
return false;
};
return new Promise(function(resolve, reject) {
var intervalId = setInterval(function() {
if (queueLoop()) {
clearInterval(intervalId);
if (failures.length == 0) {
resolve();
} else {
reject("Failed to delete documents " + failures);
}
}
}, 0);
});
};
/**
* Delete everything under a given path. If the path represents
* a document the document is deleted and then all descendants
* are deleted.
*
* #return {Promise} a promise for the entire operation.
*/
FirestoreDelete.prototype._deletePath = function() {
var self = this;
var initialDelete;
if (this.isDocumentPath) {
var doc = { name: this.parent + "/" + this.path };
initialDelete = firestore.deleteDocument(doc).catch(function(err) {
logger.debug("deletePath:initialDelete:error", err);
if (self.allDescendants) {
// On a recursive delete, we are insensitive to
// failures of the initial delete
return Promise.resolve();
}
// For a shallow delete, this error is fatal.
return utils.reject("Unable to delete " + clc.cyan(this.path));
});
} else {
initialDelete = Promise.resolve();
}
return initialDelete.then(function() {
return self._recursiveBatchDelete();
});
};
/**
* Delete an entire database by finding and deleting each collection.
*
* #return {Promise} a promise for all of the operations combined.
*/
FirestoreDelete.prototype.deleteDatabase = function() {
var self = this;
return firestore
.listCollectionIds(this.project)
.catch(function(err) {
logger.debug("deleteDatabase:listCollectionIds:error", err);
return utils.reject("Unable to list collection IDs");
})
.then(function(collectionIds) {
var promises = [];
logger.info("Deleting the following collections: " + clc.cyan(collectionIds.join(", ")));
for (var i = 0; i < collectionIds.length; i++) {
var collectionId = collectionIds[i];
var deleteOp = new FirestoreDelete(self.project, collectionId, {
recursive: true,
});
promises.push(deleteOp.execute());
}
return Promise.all(promises);
});
};
/**
* Check if a path has any children. Useful for determining
* if deleting a path will affect more than one document.
*
* #return {Promise<boolean>} a promise that retruns true if the path has
* children and false otherwise.
*/
FirestoreDelete.prototype.checkHasChildren = function() {
return this._getDescendantBatch(true, 1).then(function(docs) {
return docs.length > 0;
});
};
/**
* Run the delete operation.
*/
FirestoreDelete.prototype.execute = function() {
var verifyRecurseSafe;
if (this.isDocumentPath && !this.recursive && !this.shallow) {
verifyRecurseSafe = this.checkHasChildren().then(function(multiple) {
if (multiple) {
return utils.reject("Document has children, must specify -r or --shallow.", { exit: 1 });
}
});
} else {
verifyRecurseSafe = Promise.resolve();
}
var self = this;
return verifyRecurseSafe.then(function() {
return self._deletePath();
});
};
module.exports = FirestoreDelete;
For those who don't want or can't use cloud functions, I found a recursiveDelete function in the admin sdk:
https://googleapis.dev/nodejs/firestore/latest/Firestore.html#recursiveDelete
// Recursively delete a reference and log the references of failures.
const bulkWriter = firestore.bulkWriter();
bulkWriter
.onWriteError((error) => {
if (error.failedAttempts < MAX_RETRY_ATTEMPTS) {
return true;
} else {
console.log('Failed write at document: ', error.documentRef.path);
return false;
}
});
await firestore.recursiveDelete(docRef, bulkWriter);
i don't know how much helpful for you but test it and compare the execution time which i get use it from fire store doc
/** Delete a collection in batches to avoid out-of-memory errors.
* Batch size may be tuned based on document size (atmost 1MB) and application requirements.
*/
void deleteCollection(CollectionReference collection, int batchSize) {
try {
// retrieve a small batch of documents to avoid out-of-memory errors
ApiFuture<QuerySnapshot> future = collection.limit(batchSize).get();
int deleted = 0;
// future.get() blocks on document retrieval
List<QueryDocumentSnapshot> documents = future.get().getDocuments();
for (QueryDocumentSnapshot document : documents) {
document.getReference().delete();
++deleted;
}
if (deleted >= batchSize) {
// retrieve and delete another batch
deleteCollection(collection, batchSize);
}
} catch (Exception e) {
System.err.println("Error deleting collection : " + e.getMessage());
}
}
As mentioned above, you need to write good bit of code for this. For each document that is to be deleted you need to check if it has one or more collections. If it does, then you need to queue those up for deletion too. I wrote the code below to do this. It's not tested to be scalable to large data sets, which is fine for me as I'm using it to clean up after small scale integration tests. If you need something more scalable, feel free to take this as a starting point and play around with batching more.
class FirebaseDeleter {
constructor(database, collections) {
this._database = database;
this._pendingCollections = [];
}
run() {
return new Promise((resolve, reject) => {
this._callback = resolve;
this._database.getCollections().then(collections => {
this._pendingCollections = collections;
this._processNext();
});
});
}
_processNext() {
const collections = this._pendingCollections;
this._pendingCollections = [];
const promises = collections.map(collection => {
return this.deleteCollection(collection, 10000);
});
Promise.all(promises).then(() => {
if (this._pendingCollections.length == 0) {
this._callback();
} else {
process.nextTick(() => {
this._processNext();
});
}
});
}
deleteCollection(collectionRef, batchSize) {
var query = collectionRef;
return new Promise((resolve, reject) => {
this.deleteQueryBatch(query, batchSize, resolve, reject);
});
}
deleteQueryBatch(query, batchSize, resolve, reject) {
query
.get()
.then(snapshot => {
// When there are no documents left, we are done
if (snapshot.size == 0) {
return 0;
}
// Delete documents in a batch
var batch = this._database.batch();
const collectionPromises = [];
snapshot.docs.forEach(doc => {
collectionPromises.push(
doc.ref.getCollections().then(collections => {
collections.forEach(collection => {
this._pendingCollections.push(collection);
});
})
);
batch.delete(doc.ref);
});
// Wait until we know if all the documents have collections before deleting them.
return Promise.all(collectionPromises).then(() => {
return batch.commit().then(() => {
return snapshot.size;
});
});
})
.then(numDeleted => {
if (numDeleted === 0) {
resolve();
return;
}
// Recurse on the next process tick, to avoid
// exploding the stack.
process.nextTick(() => {
this.deleteQueryBatch(query, batchSize, resolve, reject);
});
})
.catch(reject);
}
}
Solution using Node.js Admin SDK
export const deleteDocument = async (doc: FirebaseFirestore.DocumentReference) => {
const collections = await doc.listCollections()
await Promise.all(collections.map(collection => deleteCollection(collection)))
await doc.delete()
}
export const deleteCollection = async (collection: FirebaseFirestore.CollectionReference) => {
const query = collection.limit(100)
while (true) {
const snap = await query.get()
if (snap.empty) {
return
}
await Promise.all(snap.docs.map(doc => deleteDocument(doc.ref)))
}
}
There is now a simple way delete a document and all of its subcollections using NodeJS.
This was made available in nodejs-firestore version v4.11.0.
From the docs:
recursiveDelete()
Recursively deletes all documents and subcollections at and under the specified level.
import * as admin from 'firebase-admin'
const ref = admin.firestore().doc('my_document')
admin.firestore().recursiveDelete(ref)
You can write a handler which will recursive delete all nested descendants when triggers onDelete Firestore event.
Example of handler:
const deleteDocumentWithDescendants = async (documentSnap: FirebaseFirestore.QueryDocumentSnapshot) => {
return documentSnap.ref.listCollections().then((subCollections) => {
subCollections.forEach((subCollection) => {
return subCollection.get().then((snap) => {
snap.forEach((doc) => {
doc.ref.delete();
deleteDocumentWithDescendants(doc);
});
});
});
});
};
// On any document delete
export const onDocumentDelete = async (documentSnap: FirebaseFirestore.QueryDocumentSnapshot) => {
await deleteDocumentWithDescendants(documentSnap);
};
Tie it up with firestore event:
exports.onDeleteDocument = functions.firestore.document('{collectionId}/{docId}')
.onDelete(onDocumentDelete);
// You can add all the collection hierarchy to object
private collectionsHierarchy = {
groups: [
[
'groups',
'projects',
'files',
'assets',
'urls',
'members'
]
]
};
async deleteDocument(rootDocument: string) {
// if (!rootDocument.startsWith(`groups/${this.groupId()}`)) {
// rootDocument = `groups/${this.groupId()}/${rootDocument}`;
// }
const batchSize: number = 100;
let root = await this.db
.doc(rootDocument)
.get()
.toPromise();
if (!root.exists) {
return;
}
const segments = rootDocument.split('/');
const documentCollection = segments[segments.length - 2];
const allHierarchies = this.collectionsHierarchy[documentCollection];
for (let i = 0; i < allHierarchies.length; i = i + 1) {
const hierarchy = allHierarchies[i];
const collectionIndex = hierarchy.indexOf(documentCollection) + 1;
const nextCollections: [] = hierarchy.slice(collectionIndex);
const stack = [`${root.ref.path}/${nextCollections.shift()}`];
while (stack.length) {
const path = stack.pop();
const collectionRef = this.db.firestore.collection(path);
const query = collectionRef.orderBy('__name__').limit(batchSize);
let deletedIems = await this.deleteQueryBatch(query, batchSize);
const nextCollection = nextCollections.shift();
deletedIems = deletedIems.map(di => `${di}/${nextCollection}`);
stack.push(...deletedIems);
}
}
await root.ref.delete();
}
private async deleteQueryBatch(
query: firebase.firestore.Query,
batchSize: number
) {
let deletedItems: string[] = [];
let snapshot = await query.get();
if (snapshot.size === 0) {
return deletedItems;
}
const batch = this.db.firestore.batch();
snapshot.docs.forEach(doc => {
deletedItems.push(doc.ref.path);
batch.delete(doc.ref);
});
await batch.commit();
if (snapshot.size === 0) {
return deletedItems;
}
const result = await this.deleteQueryBatch(query, batchSize);
return [...deletedItems, ...result];
}
Another solution using Node.js Admin SDK with Batch.
const traverseDocumentRecursively = async (
docRef: FirebaseFirestore.DocumentReference<FirebaseFirestore.DocumentData>,
accumulatedRefs: FirebaseFirestore.DocumentReference<FirebaseFirestore.DocumentData>[],
) => {
const collections = await docRef.listCollections();
if (collections.length > 0) {
for (const collection of collections) {
const snapshot = await collection.get();
for (const doc of snapshot.docs) {
accumulatedRefs.push(doc.ref);
await traverseDocumentRecursively(doc.ref, accumulatedRefs);
}
}
}
};
import { chunk } from 'lodash';
const doc = admin.firestore().collection('users').doc('001');
const accumulatedRefs: FirebaseFirestore.DocumentReference<FirebaseFirestore.DocumentData>[] = [];
await traverseDocumentRecursively(doc, accumulatedRefs);
await Promise.all(
// Each transaction or batch of writes can write to a maximum of 500 documents
chunk(accumulatedRefs, 500).map((chunkedRefs) => {
const batch = admin.firestore().batch();
for (const ref of chunkedRefs) {
batch.delete(ref);
}
return batch.commit();
}),
);
Not sure if this is helpful for anyone here, but I am frequently facing the error "Fatal error deleting docs <list of docs>" when using firebase-tools.firestore.delete method (firebase-tools version 9.22.0).
I am currently handling these deletion failures using the returned error message in order to avoid rewriting the code cited at Oleg Bondarenko's answer. It uses admin.firestore to effectively delete the failed docs.
It's a poor solution since it relies on the error message, but at least it doesn't force us to copy the whole FirestoreDelete code to modify just a few lines of it:
firebase_tools.firestore
.delete(path, {
project: JSON.parse(process.env.FIREBASE_CONFIG!).projectId,
recursive: true,
yes: true,
token: getToken(),
})
.catch((err: Error) => {
if (err.name == "FirebaseError") {
// If recursive delete fails to delete some of the documents,
// parse the failures from the error message and delete it manually
const failedDeletingDocs = err.message.match(
/.*Fatal error deleting docs ([^\.]+)/
);
if (failedDeletingDocs) {
const docs = failedDeletingDocs[1].split(", ");
const docRefs = docs.map((doc) =>
firestore.doc(doc.slice(doc.search(/\(default\)\/documents/) + 19))
);
firestore
.runTransaction(async (t) => {
docRefs.forEach((doc) => t.delete(doc));
return docs;
})
.then((docs) =>
console.log(
"Succesfully deleted docs after failing: " + docs.join(", ")
)
)
.catch((err) => console.error(err));
}
}
});
If you are looking to delete user data, a solution to consider in 2022 is the Delete User Data Firebase Extension.
Once this is active, you can simply delete the user from Firebase Auth to trigger the recursive deletion of the user documents:
import admin from "firebase-admin";
admin.auth().deleteUser(userId);
You can call firebase.firestore().doc("whatever").set() and that will delete everything in that document.
The only way .set does not erase everything is if you set the merge flag to true.
See Firestore Documentation on Add Data
var cityRef = db.collection('cities').doc('BJ');
var setWithMerge = cityRef.set({
capital: true
}, { merge: true });