I just migrated my website over to firebase from localhost and its working fine however I do have the issue of my firebase functions taking a pretty significant amount of time to resolve. One of the core features is pulling files from a google cloud bucket which was taking only 3 seconds on localhost and is now taking around x3 as long after migrating.
Is there anyway for me to speed up my firebase function query time? If not then is there a way for me to at least wait for a request to resolve before redirecting to a new page?
Here is the code for pulling the file in case it helps at all.
app.get('/gatherFromStorage/:filepath',async(req,res) => {
try{
const {filepath} = req.params;
const file = bucket.file(filepath)
let fileDat = []
const newStream = file.createReadStream()
newStream.setEncoding('utf8')
.on('data',function(chunk){
fileDat.push(chunk)
})
.on('end', function() {
console.log('done')
res.json(fileDat)
})
.on('error',function(err){
console.log(err)
});
}catch(error){
res.status(500).send(error)
console.log(error)
}
})
Also this question may come off as silly but I just don't know the answer. When I create a express endpoint should each endpoint be its own firebase function or is it fine that I wrap all my endpoints into one firebase function?
Related
I´ve got a weird problem.
Using Node, React, Express, MongoDB -> MERN Stack.
So my page generates a PDF file which then gets send to the backend (as blob data) and is being stored on there.
The problem I have, now I need to send a payment ID along with that blob data to save the order in the data base. I need both in one post request, to make it as smooth as possible:
await axios
.post(process.env.REACT_APP_SERVER_API + '/payment/cash', {
blobData,
paymentId
})
.then(async (res) => ...
like so.
Before, when I just sent the blob data, I could simply access the data in the backend by writing:
exports.createCashOrder = async (req, res) => {
const { filename } = req.file; // THIS RIGHT HERE
const fileHash = await genFileHash(filename);
try {
await saveOrder(filename, fileHash, "cash", paymentId);
//await sendOrderCreatedEmail(req.body, fileHash);
//await sendOrderReceivedConfirmEmail(req.body, fileHash);
res.send({ filename: filename });
}
But that doesn't work anymore. I dont have access to that file object anymore when sending that request object.
Neither by trying
req.body.blobData
req.body.blobData.file
req.file
Any idea how to achieve that, except from making two seperate post requests?
Glad for any help, cheers!
Send the data as a form
await axios
.postForm(process.env.REACT_APP_SERVER_API + '/payment/cash', {
blobData,
paymentId
})
.then(async (res) => ...
And then use multer middleware to handle the form in express.
I have built a small app that I deployed to heroku. Locally, the whole thing is working as expected. But when deployed, the Network.webSocketFrameReceived event is never triggered. It is a node app that runs on express with a minimal websocket server.
The goal of the app is to open some url using headless chrome (i am using puppeteer here), record the websocket frames and parse them if they contain some specific fields, close connection when successful. Then move to next url.
async function openUrlAndParseFrames(page, url) {
await new Promise(async function (resolve) {
const parseWebsocketFrame = (response) => {
console.log('parsing websocket frame...', response);
let payload;
try {
// some parsing here
} catch (e) {
console.error(`Error while parsing payload ${response.response.payloadData}`)
}
}
console.log('Go to url', url);
await page.goto(url);
const cdp = await page.target().createCDPSession();
await cdp.send('Network.enable');
await cdp.send('Page.enable');
cdp.on('Network.webSocketFrameReceived', parseWebsocketFrame);
});
}
Is it not possible to make this websocket connection on heroku using puppeteer? I never receive the "parsing websocket frame..." logs...
PS:
I am aware of this special args I need to set for puppeteer to run on heroku
puppeteer.launch({ args: ['--no-sandbox', '--disable-setuid-sandbox'] });
Also I added the buildpacks heroku/nodejs and https://github.com/jontewks/puppeteer-heroku-buildpack
I found the answer myself. The real problem was, that the IP range (from Heroku) was blocked and I didn't even access the page I was trying to but was blocked with a 403 from CloudFront.
I figured it out by logging the page content. const websiteContent = await page.content(); Which showed the error page html.
After trying various things I decided to move away from Heroku and now successfully deployed to Google App Engine.
I have a website that is making a GET call to my backend database. The first time the website calls the API it takes 11 seconds, clearly, this is too long. But if I refresh the page and give it another go, it is super fast in less than a second.
I tried to find some solutions, so I opened DevTools and found this:
For some reason, the TTFB (Time to First Byte) takes 10 seconds!
How can I reduce the TTFB the first time the website calls the Rest API?
Here is my React code which is using Axios to fetch the rest API:
axios
.get(
"https://MY.API",
{
headers,
}
)
.then((response) => {
this.setState({
response: response.data,
});
})
.catch((err) => {
console.error(err);
});
Here is my backend code using Express, Mongoose, and MongoDB
router.get("/", async (req, res) => {
try {
const response = await Model.find();
res.json(response);
} catch (err) {
res.json({ message: err });
}
});
I would say that this is a pretty standard piece of code. I don't know why the TTFB is so much.
What tips I can implement to reduce the original wait time? This is annoying to my users.
Thanks!
I have built two docker images. One with nginx that serves my angular web app and another with node.js that serves a basic express app. I have tried to access the express app from my browser in two different tabs at the same time.
In one tab the angular dev server (ng serve) serves up the web page. In the other tab the docker nginx container serves up the web page.
While accessing the node.js express app at the same time from both tabs the data starts to mix and mingle and the results returned to both tabs are a mix mash of the two requests (one from each browser tab)...
I'll try and make this more simple by showing my express app code here...but to answer this question you may not even need to know what the code is at all...so maybe check the question as stated below the code first.
'use strict';
/***********************************
GOOGLE GMAIL AND OAUTH SETUP
***********************************/
const fs = require('fs');
const {google} = require('googleapis');
const gmail = google.gmail('v1');
const clientSecretJson = JSON.parse(fs.readFileSync('./client_secret.json'));
const oauth2Client = new google.auth.OAuth2(
clientSecretJson.web.client_id,
clientSecretJson.web.client_secret,
'https://us-central1-labelorganizer.cloudfunctions.net/oauth2callback'
);
/***********************************
EXPRESS WITH CORS SETUP
***********************************/
const PORT = 8000;
const HOST = '0.0.0.0';
const express = require('express');
const cors = require('cors');
const cookieParser = require('cookie-parser');
const bodyParser = require('body-parser');
const whiteList = [
'http://localhost:4200',
'http://localhost:80',
'http://localhost',
];
const googleApi = express();
googleApi.use(
cors({
origin: whiteList
}),
cookieParser(),
bodyParser()
);
function getPageOfThreads(pageToken, userId, labelIds) {
return new Promise((resolve, reject) => {
gmail.users.threads.list(
{
'auth': oauth2Client,
'userId': userId,
'labelIds': labelIds,
'pageToken': pageToken
},
(error, response) => {
if (error) {
console.error(error);
reject(error);
}
resolve(response.data);
}
)
});
}
async function getPages(nextPageToken, userId, labelIds, result) {
while (nextPageToken) {
let pageOfThreads = await getPageOfThreads(nextPageToken, userId, labelIds);
console.log(pageOfThreads.nextPageToken);
pageOfThreads.threads.forEach((thread) => {
result = result.concat(thread.id);
})
nextPageToken = pageOfThreads.nextPageToken;
}
return result;
}
googleApi.post('/threads', (req, res) => {
console.log(req.body);
let threadIds = [];
oauth2Client.credentials = req.body.token;
let getAllThreadIds = new Promise((resolve, reject) => {
gmail.users.threads.list(
{ 'auth': oauth2Client, 'userId': 'me', 'maxResults': 500 },
(err, response) => {
if (err) {
console.error(err)
reject(err);
}
if (response.data.threads) {
response.data.threads.forEach((thread) => {
threadIds = threadIds.concat(thread.id);
});
}
if (response.data.nextPageToken) {
getPages(response.data.nextPageToken, 'me', ['INBOX'], threadIds).then(result => {
resolve(result);
}).catch((err) => {
console.error(err);
reject(err);
});
} else {
resolve(threadIds);
}
}
);
});
getAllThreadIds
.then((result) => {
res.send({ threadIds: result });
})
.catch((error) => {
res.status(500).send({ error: 'Request failed with error: ' + error })
});
});
googleApi.get('/', (req, res) => res.send('Hello World!'))
googleApi.listen(PORT, HOST);
console.log(`Running on http://${HOST}:${PORT}`);
The angular app makes a simple request to the express app and waits for the reply...which it properly receives...but when I try to make two requests at the exact same time data starts to get mixed together and results are given back to each browser tab from different accounts...
...and the question is... When running containers in the cloud is this kind of thing an issue? Does one need to spin up a new container for each client that wants to actively connect to the express service so that their data doesn't get mixed?
...or is this an issue I am seeing because the express app is being accessed from locally inside my machine? If two machines with two different ip address tried to access this express server at the same time would this sort of data mixing still be an issue or would each get back it's own set of results?
Is this why people use CaaS instead of IaaS solutions?
FYI: this is demo code and the data will not be actually going back to the consumer directly...plans are to have it placed into a database and then re-extracted from the database to download all of the metadata headers for each email.
-Thank you for your time
I can only clear up a small part of this question:
When running containers in the cloud is this kind of thing an issue?
No. Docker is not causing any of the quirky behaviour that you are describing.
Does one need to spin up a new container for each client?
A docker container generally can serve as much users as the application inside of it can. So as long as your application can handle a lot of users (and it should), you don't have to start the same application in multiple containers. That said, when you expect a very large number of customers, there exist docker tools like Docker Compose, Docker Swarm and a lot of alternatives that will enable you to scale up later. For now, you don't need to worry about this at all.
I think I may have found out the issue with my code...and this is actually very important if you are using the node.js googleapis client library...
It is entirely necessary to create a new oauth2Client for each request that comes in
const oauth2Client = new google.auth.OAuth2(
clientSecretJson.web.client_id,
clientSecretJson.web.client_secret,
'https://us-central1-labelorganizer.cloudfunctions.net/oauth2callback'
);
Problem:
When this oauth2Client is shared it is shared by each and every person that connects at the same time...So it is necessary to create a new one each and every time a user connects to my /threads endpoint so that they do not share the same memory space (i.e. access_token etc.) while the processing is done.
Setting the client secret etc. and creating the oauth2Client just once at the top and then simply resetting the token for each request leads to the conflicts mentioned above.
Solution:
For now simply moving the creation of this oauth2Client into each and every request that comes in makes this work properly.
Each client that connects to the service NEEDS to have their own newly created oauth2Client instance or these types of conflicts will occur...
...it's kind of a no brainer but I still find it odd that there is nothing about this in the docs? and their own examples (https://github.com/googleapis/google-api-nodejs-client) seem to show only one instance being created for the whole of their app...but those examples are snippets so...
I have a Node.js web scraper that might take over an hour to run. It times out trying to run on App Engine standard environment. What's the best way to deploy it?
Also, it it triggered to run once per day with cron.yaml which hits an Express route. Is there a better way to do this?
Here is a simplified snippet of the code. I can run it locally, and deploy it to App Engine. It runs fine with a small amount of links in dlLinkArray. But with a larger amount (thousands) it doesn't seem to do anything. Usage reports show that it runs for a few seconds.
const Storage = require('#google-cloud/storage');
const storage = new Storage();
function startDownload(){
dlLinkArray = [/*Array of objects with URL and Filename {link: 'http://source.com', filename: 'file123456'} */]; //About 10,000 links/files
var promises = [];
dlLinkArray.forEach(record =>{ //create array of nested promises
promises.push(
uploadFile(bucketName, record.link, record.filename)
.then((x) => {
if(x[1].name) //rename file from whatever is on the remote server to a usefull ID
return renameFile(bucketName, x[1].name, record.filename + ".pdf"); //renameFile uses storage.file.move to rename, returns a promise
else
return x;
})
);
});
return Promise.all(promises);
}
function uploadFile(bucketName, fileURL, reName) {
// Uploads a remove file to the Cloud Storage bucket
return storage
.bucket(bucketName)
.upload(fileURL, {
gzip: true,
metadata: {
cacheControl: 'public, max-age=31536000',
},
});
}
/*Express Route*/
app.get('/api/whatever/download', (req, res) => {
buckets2.startDownload().then(() => console.log("DONE"));
res.status(200).send("Download Started");
});
I suspect that the problem might occur due to the request deadline. For the App Engine standard it is set at 60 seconds by default. However, if you use manual scaling, requests can run up to 24 hours in the standard environment.