Return observable while doing task - Firebase Functions - node.js

Question

I have a function in my Firebase Cloud Functions that needs to run a scraping task. While the function is scraping the public website I need to return to the consumer a progress feedback. (eg. "Now it's at 10%", "Now it's at 30%", etc...).
I tought using Observables but I get weird results because the scraping function is async but it should return immediately the observable.
This is the simplified code of my Firebase Functions Backend:
exports.scrapeFunction = functions.region('europe-west2').https.onCall((data) => {
const res$ = new BehaviorSubject<Response>({
completion: 0,
completionMsg: 'One moment...',
error: undefined,
user: undefined,
});
scrape(data.id, res$);
return res$.asObservable();
});
const scrape = async (id: number, res$: BehaviorSubject<Response>) => {
const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
res$.next({
completion: 10,
completionMsg: 'I\'m loading the browser...'
})
await page.goto('https://example.com');
res$.next({
completion: 20,
completionMsg: 'Page opened...'
})
/* Other scraping async functions... */
}
On my frontend, instead I've a code similar to this:
response$ = Observable<Response> | undefined;
ngOnInit(): void {
const scrapeFunction= getFunctions(undefined, 'europe-west2');
const verifyClient = httpsCallable(functions, 'scrapeFunction');
const { data } = await scrapeFunction({ id: 0001 });
console.log('Data is', data);
this.response$ = (data as any).source as Observable<Response>;
this.response$.subscribe((res) => console.log(res));
return;
}
Logging in the console I get:
Data is
{source: {...}}
source:
closed: false
currentObservers: null
hasError: false
isStopped: false
observers: []
thrownError: null
_value: {completion: 0, completionMsg: 'One moment...'}
[[Prototype]]: Object
[[Prototype]]: Object
I also get the following error because the function is not returning an Observable as I know, but an object with the first key "source".
ERROR Error: Uncaught (in promise): TypeError: _this.response$.subscribe is not a function
TypeError: _this.response$.subscribe is not a function
at dashboard.component.ts:35:29
at Generator.next (<anonymous>)
at asyncGeneratorStep (asyncToGenerator.js:3:1)
at _next (asyncToGenerator.js:25:1)
at ZoneDelegate.invoke (zone.js:372:1)
at Object.onInvoke (core.js:28692:1)
at ZoneDelegate.invoke (zone.js:371:1)
at Zone.run (zone.js:134:1)
at zone.js:1276:1
at ZoneDelegate.invokeTask (zone.js:406:1)
at resolvePromise (zone.js:1213:1)
at zone.js:1120:1
at asyncGeneratorStep (asyncToGenerator.js:6:1)
at _next (asyncToGenerator.js:25:1)
at ZoneDelegate.invoke (zone.js:372:1)
at Object.onInvoke (core.js:28692:1)
at ZoneDelegate.invoke (zone.js:371:1)
at Zone.run (zone.js:134:1)
at zone.js:1276:1
at ZoneDelegate.invokeTask (zone.js:406:1)
Can someone understand this behavior?
[UPDATE 1]
I edited my index.ts file in Firebase Functions to be like this:
const res$ = new BehaviorSubject<Response>({
completion: 0,
completionMsg: 'One moment...',
});
exports.verifyClient = functions.https.onCall((data) => {
scrapeCheck(data.id);
return verifyClientRes$.asObservable();
});
const scrapeCheck = async (id: number) => {
await scrape(id, res$);
res$.complete();
return;
};
/* The scrape function is actually on an external file */
const scrape = (/* ... */) => {
/* ... */
}
** But i still got the same error on the console.
[UPDATE 2]
I created a StackBlitz project. You can find it here:
https://stackblitz.com/edit/wherescrypto?file=functions/src/index.ts
The function that is causing this error is in "functions/src/index.ts" and it's called "verifyClient"

Answer 1

Reviewing some documentation that could help to fix the issue you have, I was wondering if you already considered trying it with a realtime database or firestore.
You'll have to push your updates somewhere, the client app can see them that way. This solution was mentioned in a comment in this stackoverflow question.

Answer 2

Putting scrape in setTimeout will do the trick here. so that scrape is invoked after observable is returned.
exports.scrapeFunction = functions.region('europe-west2').https.onCall((data) => {
const res$ = new BehaviorSubject<Response>({
completion: 0,
completionMsg: 'One moment...',
error: undefined,
user: undefined,
});
setTimeout(() => scrape(data.id, res$));
return res$.asObservable();
});