I built an app with Vuejs which is hosted on firebase, I recently added dynamic rendering with rendertron to improve SEO, I'm hosting the rendertron on Heroku. The rendertron client work well.
In order to send requests coming from bots like googlebot to rendertron and recieve a compiled HTML file, I used firebase function, it checks for the user agent, if it's a bot then it sends it to the rendertron link, if it's not, it fetches the app and resend result.
Here's the function code:
const functions = require('firebase-functions');
const express = require('express');
const fetch = require('node-fetch');
const url = require('url');
const app = express();
const appUrl = 'khbich.com';
const renderUrl = 'https://khbich-render.herokuapp.com/render';
function generateUrl(request){
return url.format({
protocol:request.protocol,
host:appUrl,
pathname:request.originalUrl
});
}
function detectBot(userAgent){
let bots = [
"googlebot",
"bingbot",
"facebookexternalhit",
"twitterbot",
"linkedinbot",
"facebot"
]
const agent = userAgent.toLowerCase()
for(let bot of bots){
if(agent.indexOf(bot)>-1){
console.log('bot-detected',bot,agent)
}
}
}
app.get('*', (req,res)=>{
let isBot = detectBot(req.headers['user-agent']);
if(isBot){
let botUrl= generateUrl(req);
fetch(`${renderUrl}/${botUrl}`)
.then(res => res.text())
.then(body=>{
res.set('Cache-Control','public','max-age=300','s-maxage=600')
res.set('Vary','User-Agent');
res.send(body.toString())
})
}
else{
fetch(`https://${appUrl}`)
.then(res=>res.text())
.then(body=>{
res.send(body.toString())
})
}
});
I used the function as an entry point for firebase hosting, so it's invoked whenever someone enters the app.
I checked on the firebase dashboard to see if it's working, and I noticed that it crashed for exceeding the number of requests per 100 second quota, I don't have much users when I checked, and the function invocations reached 370 calls in one minute.
I don't see why I had a large number of calls all at once, I'm thinking that maybe since I'm fetching the website if the user agent is not a bot, then the function is re-invoked causing an infinite loop of invocations, but I don't know if that's really why ?
If it's an infinite loop, how can I redirect users to their entered url without reinvoking the function ? will a redirect work ?
Related
I'm working on a simple local image server that provides images to a web application with some JSON. The web application has pagination that will do a get request "/images?page=X&limit&200" to an express.js server that returns the JSON files in a single array. I want to take advantage of the browser's internal caching such that if a user goes to a previous page the express.js returns an ETAG. I was wondering how this could be achieved with express.js? For this application, I really just want the computation of the ETAG to take in three parameters the page, the directory, and the limit (It doesn't need to consider the whole JSON body). Also this application is for local use only, so I want the server to do the heavy lifting since I figured it be faster than the browser. I did see https://www.npmjs.com/package/etag which seems promising, but I'm not sure how to use it with express.js
Here's a boilerplate of the express.js code I have below:
var express = require('express');
var app = express();
var fs = require('fs');
app.get('/', async (req, res) =>{
let files = [];
let directory = fs.readdirSync("mypath");
let page = parseInt(req.query.page);
let limit = parseInt(req.query.limit);
for (let i = 0; i < limit; ++i) {
files.push(new Promise((resolve) => {
fs.readFile(files[i + page * limit].name, (err, data) => {
// format the data so easy to use for UI
resolve(JSON.parse(data));
});
});
}
let results = await Promise.all(files);
// compute an etag here and attach it the results.
res.send(results);
});
app.listen(3000);
When your server sends an ETag to the client, it must also be prepared to check the ETag that the client sends back to the server in the If-None-Match header in a subsequent "conditional" request.
If it matches, the server shall respond with status 304; otherwise there is no benefit in using ETags.
var serverEtag = "<compute from page, directory and limit>";
var clientEtag = req.get("If-None-Match");
if (clientEtag === serverEtag) res.status(304).end();
else {
// Your code from above
res.set("ETag", serverEtag);
res.send(results);
}
The computation of the serverEtag could be based on the time of the last modification in the directory, so that it changes whenever any of the images in that directory changes. Importantly, this could be done without carrying out the fs.readFile statements from your code.
I created some functions in firebase cloud functions, but all of them are works. But i have a new function which is not work properly. I don't know why but i think it has same pattern with others.
this is my code:
const functions = require('firebase-functions');
const appVideo = express();
const cors = require('cors')({ origin: true });
appVideo.use(cors);
appVideo.get('/update-video', async(req, res) => {
console.log('updateStatusVideo idCourse', req.query.idCourse, ' idMateri: ', req.query.idMateri, ' idVideo:', req.query.idVideo);
res.status(200).send('Oke')
})
exports.video = functions.https.onRequest(appVideo)
I often call partial deploy like
firebase deploy --only functions:video. But when i execute the functions https through browser it often return
Request failed with status code 404
other weird things is when i inspect the browser and switch to console, i found
Failed to load resource: the server responded with a status of 500 ()
this is the url of function in firebase:
https://us-central1-my-apps.cloudfunctions.net/video [modified for confidential]
Please help
When you export this line:
exports.video = functions.https.onRequest(appVideo);
You define a Cloud Function called video that is deployed as https://us-central1-PROJECT_ID.cloudfunctions.net/video where PROJECT_ID is whatever your Firebase Project ID is.
Because you use a express application for this exported function, any URL that is handled must first start with https://us-central1-PROJECT_ID.cloudfunctions.net/video (the BASE_URL).
This line:
appVideo.get('/update-video', ...)
attaches a listener to BASE_URL/update-video, which becomes https://us-central1-PROJECT_ID.cloudfunctions.net/video/update-video.
If you want to use just https://us-central1-PROJECT_ID.cloudfunctions.net/video as-is, you'll need to change to using
appVideo.get('/', ...)
If you want to use just https://us-central1-PROJECT_ID.cloudfunctions.net/update-video, you'll need to change to using
appVideo.get('/', ...)
and
exports.update = {};
exports.update.video = functions.https.onRequest(appVideo);
// "update.video" is deployed as "update-video"
Note: This last part abuses deploying groups to get the desired URL
I am using actionssdk and I build my Action fulfilments using Javascript and node.js + Express.
I am looking for a way to get the url (protocol + host name + port) of the server where the fulfilment is hosted.
Is there a simple way to do this? E.g. in the MAIN intent? Is there some conv-property I can use? Can I get hold of a req-parameter in the MAIN-intent, from which I can deduct hostname etc?
const express = require('express');
const expressApp = express();
const { actionssdk, ... } = require('actions-on-google');
const app = actionssdk({
ordersv3: true,
clientId: ...
});
expressApp.post('/fulfilment', app);
app.intent('actions.intent.MAIN', (conv) => {
let myUrl: string = ... // ???????
...
});
(background: obviously I know myself to where I deployed my fulfilment code. But I have a reusable template for fulfilment code in which I want to refer to the host url, and I do not want to type that in manually each time I develop a new Action).
You can get access to the request object in a middleware via Framework Metadata which is by default of type BuiltinFrameworkMetadata which contains objects used by Express
For example, you can use it like this, which will be ran before each request:
app.middleware((conv, framework) => {
console.log(framework.express.request.headers.host)
})
Im using SuperRewards (SR) To make Coin transaction for users in my website,maybe you are familiar with SR.
Whenever a transaction happens SuperRewards sends a postback (Post Request) to my server containing information about the transaction and coins etc...
So my question is how to handle Postbacks or Post request ( i really dont know the diffrence ) from other website to my server USing Nodejs Express ?
Picture 1 App testing
picture 2 app Testing
Code
You would handle it like any other request coming to your Express app. Since it's a GET request, you would have a GET route defined.
According to section Fields and Formats, you get back quite a few query strings. Knowing that we can do the following:
app.get('/super-rewards', async (req, res) => {
// `new` is a reserved keyword, so we can't use `new` as a variable name.
const newCurrency = req.query['new']
const {
id,
uid,
oid,
total,
sig
} = req.query
})
Additionally, the documentation states that sig should match the MD5 hash of your secret key, if I'm understanding that correctly. So a full example would be something like:
const crypto = require('crypto')
const express = require('express')
const app = express()
app.get('/super-rewards', async (req, res) => {
// `new` is a reserved keyword, so we can't use `new` as a variable name.
const newCurrency = req.query['new']
const {
id,
uid,
oid,
total,
sig
} = req.query
const secretHash = crypto.createHash('md5').update(process.env.SECRET_KEY).digest('hex')
if (secretHash !== sig) {
throw new Error('Invalid transaction')
}
})
Aside, it is a GET request because the documentation clearly states that:
Postbacks are sent from our server to yours using a HTTP GET request (...)
I contacted the support team and they told me i must use a public domain not Local host that s why it wasnt working so the problem is solved and thank you for your time :)
I'm trying to get a Skype bot up and running based off of the echo example but I'm struggling to make a successful POST to my app. When I send a post to /v1/chat I get back a status of 201 (successful creation), and nothing in the body. My console.log does not print anything either, which leads me to believe that the botService.on('personalMessage', ...) function is not being run. Does anyone have any insight into how these POST requests should be formatted? I cannot seem to find anything in the documentation.
My code:
const fs = require('fs');
const restify = require('restify');
const skype = require('skype-sdk');
const botService = new skype.BotService({
messaging: {
botId: '28:<bot’s id="ID176db9ab-e313-4d76-a60c-bc2a280e9825">',
serverUrl : "https://apis.skype.com",
requestTimeout : 15000,
appId: process.env.APP_ID,
appSecret: process.env.APP_SECRET
}
});
botService.on('contactAdded', (bot, data) => {
console.log('contact added');
bot.reply('Hello ${data.fromDisplayName}!', true);
});
botService.on('personalMessage', (bot, data) => {
console.log('message incoming');
console.log(data);
bot.reply('Hey ${data.from}. Thank you for your message: "${data.content}".', true);
});
const server = restify.createServer();
server.post('/v1/chat', skype.messagingHandler(botService));
const port = process.env.PORT || 8080;
server.listen(port);
console.log('Listening for incoming requests on port ' + port);
Final Edit & Solution: I think the problem caused by Heroku somehow(it could be something with their free tier ,1 dyno). After some effort, I uploaded the program to Azure, and it is now working perfectly.
Potential Solution: You need to change the server address in the server.post command. If you run your program in "https:\www.yourwebsite.com/v1/chat" , you need to modify this;
server.post('/v1/chat', skype.messagingHandler(botService));
to this;
server.post('https:\\www.yourwebsite.com/v1/chat', skype.messagingHandler(botService));
Of course, don't forget to specify your app id, bot id, and app secret. If you don't have one, you need to generate a password in your Skype application page.
I have the exact problem with the OP. I followed the tutorial, and it doesn't specify how to modify our code to comply with our server. So, after running the program it only returns this;
{"code":"ResourceNotFound","message":"/ does not exist"}
In the Echo example in the Skype Bot Webpage; it says;
"We'll assume the bot's URL for messaging was set to https://echobot.azurewebsites.net/v1/chat during registration."
Make sure that Procfile and worker processes are setup.
My bot is working fine on heroku itself