I'm trying to create a dynamic sitemap for my website, it has lots of pages that change often.
The sitemap needs to be accessed from www.mywebsite.com/sitemap.xml
My current attempt queries the database for all the pages, get's each pages url and passes it to an EJS template that creates what looks like XML.
I have two problems here
The route to the page cannot have a file suffix. e.g. '.xml'
The page is automatically treated as html
I realise that there are other options for creating a sitemap using modules like "express-sitemap," but I haven't been able to find any easily understood (i am new to this) documentation for them, and this seems like a good way of doing things to me
yes you can use express-sitemap
To generate Sitemap automatically
var sitemap = require('express-sitemap')();
var app = require('express')();
sitemap.generate(app);
To generate dynamically..
for suppose you have products pages and you have specified url for them..
You can create a dynamic file everytime and place it in your public folder.
const Product = require('./model/product')
const sitemap = require('sitemap');
let sitemapData;
const generateSitemap = async () => {
const products = await Product.find({},{path: 1});
const urls = products.map({path} => `/products/${path}`)
sitemapData = sitemap.createSitemap ({
hostname: 'http://example.com',
cacheTime: 600000, // 600 sec - cache purge period
urls
});
}
You can use this function in a routine or with a cron and generate sitemap regularly..
setInterval(generateSitemap, 360000); //running every hour
other thing that you can do is:
use sitemapData variable and do things like this.
sitemapData.add({url: '/product-a/'}); // when some product is added
sitemapData.add({url: '/product-b/', changefreq: 'monthly', priority: 0.7});
sitemapData.del({url: '/product-c/'}); // when something is removed
sitemapData.del('/product-d/');
you can serve it in a route like this:
app.get('/sitemap.xml', function(req, res) {
sitemapData.toXML( function (err, xml) {
if (err) {
return res.status(500).end();
}
res.header('Content-Type', 'application/xml');
res.send( xml );
});
});
Here is how I made a txt sitemap. I find Google Search Console has an easier time fetching txt sitemaps vs xml sitemaps. But if you want to make a xml sitemap, you can look at this blog for the correct formatting. This code uses Mongoose and is saved as /pages/sitemap.txt.js.
// pages/sitemap.txt.js
import dbConnect from "../utils/dbConnect";
import Pduct from "../models/Pduct";
const createSitemap = (posts) => `${posts
.map(({ slug }) => {
return `https://[YOUR DOMAIN]/${slug}`;
})
.join("\n")}
`;
export async function getServerSideProps({ res }) {
await dbConnect();
const request = await Pduct.find({}, "slug").lean();
res.setHeader("Content-Type", "text");
res.write(createSitemap(request));
res.end();
}
export default () => null;
Related
I have an API which uses Node.js + Express on the backend.
For one of the API endpoints, I'd like to use the Express response object method of "sendFile", documented here:
https://expressjs.com/en/api.html#res.sendFile
The API should return a Javascript file through the sendFile method.
What I can't figure out is how to read in the .js file on the front end so that I can use the JavaScript functions defined in the file. The sendFile portion appears to be working -- it's just the use of the file which I can't figure out.
Here's what I'm doing on the backend:
app.get("/api/member", async (req, res) => {
options = {
root: path.join(__dirname, '/static'),
dotfiles: 'deny'
}
res.sendFile("member.js", options, (err) => {
if (err) {
console.log(err)
next(err)
} else {
console.log('Sent file')
}
})
});
This seems to be working fine, as I can navigate to the endpoint on my localhost and it loads the JS file. The file member.js simply contains some javascript function definitions.
But, I can't figure out how to consume/use the file once it arrives to the front end.
Here's what I have currently on the frontend:
async function refreshJS() {
const url = `${baseUrl}/member`;
const response = await fetch(url, { credentials: "include" });
const script = document.createElement("script")
script.type = "text/javascript"
script.src = response.body
document.head.appendChild(script)
eval(script)
}
I've spent a lot of time looking through the console/debugger to find the text associated with the JS functions -- but they're nowhere to be found.
I've tested this general framework by loading JS files locally through the console and it worked, so I think it's wrapped up in a misunderstanding of where the JS functions live in the API response. For example, if I replace the command above of:
script.src = response.body
with
script.src = "member.js"
then everything works fine provided I have the file locally.
The examples that I've reviewed seem to deal exclusively with sending an HTML file which is loaded on the frontend. But, I can't find supporting documentation from the fetch API to understand how to use the JS file contents.
I'm trying to do a reverse image search using googlethis on an image the user uploads. It supports reverse image searching, but only with a Google-reachable image URL. Currently, I upload the image to file.io, which deletes it after it gets downloaded.
This is the current application flow:
User POSTs file -> Server uploads file to file.io -> Google downloads the file -> Server does things with the reverse image search
However, I want to skip the middleman and have Google download files directly from the server:
User POSTs file -> Server serves file at unique URL -> Google downloads the file -> Server deletes the file -> Server does things with the reverse image search
I've looked at Serving Temporary Files with NodeJs but it just shows how to serve a file at a static endpoint. If I added a route to /unique-url, the route would stay there forever (a very slow memory leak! Probably! I'm not really sure!)
The only way I can think of is to save each file with a UUID and add a parameter: /download?id=1234567890, which would probably work, but if possible, I want to do things in memory.
So:
How do I do this using normal files?
How do I do this in-memory?
Currently working (pseudo) code:
app.post('/', (req, res) => {
const imagePath = saveImageTemporarily(req)
const tempUrl = uploadToFileIo(imagePath)
const reverseImageResults = reverseGoogleSearch(tempUrl)
deleteFile(imagePath)
doThingsWithResults(reverseImageResults).then((result) => { res.send(result) })
}
The other answer is a good one if you are able to use Redis -- it offers lots of helpful features like setting a time-to-live on entries so they're disposed of automatically. But if you can't use Redis...
The basic idea here is that you want to expose a (temporary) URL like example.com/image/123456 from which Google can download an image. You want to store the image in memory until after Google accesses it. So it sounds like there are two (related) parts to this question:
Store the file in memory temporarily
Rather than saving it to a file, why not create a Buffer holding the image data. Once you're done with it, release your reference to the buffer and the Node garbage collector will dispose of it.
let image = Buffer.from(myImageData);
// do something with the image
image = null; // the garbage collector will dispose of it now
Serve the file when Google asks for it
This is a straightforward route which determines which image to serve based on a route parameter. The query parameter you mention will work, and there's nothing wrong with that. Or you could do it as a route parameter:
app.get('/image/:id', (req, res) => {
const id = req.params.id;
res.status(200).send(/* send the image data here */);
});
Putting it all together
It might look something like this:
// store image buffers here
const imageStore = {};
app.post('/image', (req, res) => {
// get your image data here; there are a number of ways to do this,
// so I leave it up to you
const imageData = req.body;
// and generate the ID however you want
const imageId = generateUuid();
// save the image in your store
imageStore[imageId] = imageData;
// return the image ID to the client
res.status(200).send(imageId);
});
app.get('/image/:id', (req, res) => {
const imageId = req.params.id;
// I don't know off the top of my head how to correctly send an image
// like this, so I'll leave it to you to figure out. You'll also need to
// set the appropriate headers so Google recognizes that it's an image
res.status(200).send(imageStore[imageid]);
// done sending? delete it!
delete imageStore[imageId];
});
I would use REDIS for the in-memory DB, and on the server, I would transform the image to base64 to store it in Redis.
In Redis, you can also set TTL on the images.
Check my code below
import {
nanoid
} from 'nanoid'
function base64_encode(file) {
// read binary data
var bitmap = fs.readFileSync(file);
// convert binary data to base64 encoded string
return new Buffer(bitmap).toString('base64');
}
app.post('/', async(req, res) => {
const client = redisClient;
const imagePath = saveImageTemporarily(req)
//const tempUrl = uploadToFileIo(imagePath)
var base64str = base64_encode(imagePath);
const id = nanoid()
await client.set(id, JSON.stringify({
id,
image: base64str
}));
const reverseImageResults = reverseGoogleSearch(JSON.parse(await client.get(id)).image)
await client.del(id);
doThingsWithResults(reverseImageResults).then((result) => {
res.send(result)
})
}
We're working on an app that will allow users to store templates with images on them, and pull those templates up later. This is for an AR environment using Viro on React Native.
We're trying to dynamically load an image into the component, and receiving errors when we require the filepath, which has been set to a variable:
const exampleUri = '/some/uri'
render() {
return(
<Viro3DObject
source={require(exampleUri)}
/>)
}
The URI for the source prop has to be dynamic, as the URIs are pulled from a Database.
We've tried storing the entire request in the database (in models/element.js):
const Sequelize = require('sequelize');
const db = require('../db');
const Element = db.define('element', {
sourceViro3DObject: {
type: Sequelize.STRING
}
});
sourceViro3DObject: `require('../../assets/emoji_heart/emoji_heart.vrx')`
When we called it in the React Native class component:
getObjectData = async () => {
try {
const {data} = await axios.get(`/api/elements/${this.props.elementId}`)
this.setState({sourceViro3DObject: data.sourceViro3DObject})
} catch (err) {
console.log(err)
}
}
async componentDidMount() {
await this.getObjectData()
}
But this simply sets state.sourceViro3DObject to a string:
'require('../../assets/emoji_heart/emoji_heart.vrx')'
We've tried setting the filepath directly to state as a string:
state.sourceViro3DObject = '../../assets/emoji_heart/emoji_heart.vrx'
and then call require on it:
require(this.state.sourceViro3DObject)
and received the following error:
Invalid prop `source` supplied to `Viro3DObject`
We've seen recommendations of storing the URIs in an object, but that can't work for us as we don't know what image is going to be used until it's pulled from the database. We can't hard-code them anywhere.
We'd appreciate any help with this!
In our application we store our reports in a user defined folders. User can add their own folders during runtime. Iam showing the history of those files in a web page. on clicking the file name i should show the file from the folder. How can i show the files from a non public directory.Since its given during runtime i havent added them as static dir to the express server.
One idea we tried was to use node-static-server and create a file server with the folder and serve the file. for each file we create this. it works fine but i get an error saying "port already in use". is there any better idea to do this? is this the right approach?
You can do this in NodeJS using a express.static:
const FS = require('fs')
const express = require('express')
const bp = require('body-parser')
const app = express()
function fileTest(req, res, next){
if (/\.|\/|\\/.test(req.params.file))
return res.sendStatus(400)
return next();
}
app.get(
'/static/:file',
fileTest,
function(req, res, next){
req.url = req.url.replace('/static','')
next()
},
express.static(
'./static',
{
fallthrough: false
}
)
)
app.post(
'/static/:file',
fileTest,
bp.text(),
function (req, res) {
FS.writeFile(
'./static/'+req.params.file,
req.body,
function (err) {
if(err)
return res.sendStatus(500)
return res.sendStatus(200)
}
)
}
)
app.listen(
1337
)
This is a simple example showing a server that will:
[POST]
Take a text body and load it into memory( pitfall: large bodies in memory )
Based on the URL, save it as a file in the static folder
[GET]
Search for a file
If found return file
The good news is that you can make the file and then request the file without restarting the server. Bad news is that this is a VERY SLOW server( comparatively to other options ).
As with all examples no good practices were followed, so be sure to adapt it to your needs.
Things to think about as you adopt it:
How do I allow people to save files to other folders?
How do I disallow people from saving files to other folders I don't want them to?
PROPER AUTHORIZATION
I have a "partial" template that I want to use both client-side and server-side.
Is there some method or filter or something that's very similar to include except that instead of executing the template immediately, it returns a client-compiled function which I could then assign to a JS variable and use throughout my script?
At present, I'm doing this:
exports.list = function(req, res){
res.render('file/list', {
...
fileItemTemplate: jade.compile(fs.readFileSync(path.join(req.app.get('views'),'file','file-item.jade')), {client: true})
});
};
And then in my template I have:
ul#folder-list.thumbnails
each file in files
include file-item
...
script(type='text/javascript')
var fileItemTemplate = !{fileItemTemplate};
And in this way I can render some items to HTML on page-load, and then add some more in later by rendering the partial as data comes in.
This works, but it doesn't feel very DRY because I have to read in the file, and deal with filepaths and stuff in the route, and then essentially redeclare the exact same variable client-side.
Is there a nice way of doing this?
Something like this would be ideal:
script(type='text/javascript')
var fileItemTemplate = !{compile file-item};
A possible solution could be JadeAsset. See also the discussion here.
You can hook assets into Express:
assets.on('complete', function() {
var app = express.createServer();
app.configure(function() {
app.use(assets); // that's all you need to do
});
app.listen(8000);
});
To create your Jade assets:
var assets = new AssetRack([
new rack.JadeAsset({
url: '/templates.js',
dirname: __dirname + '/templates'
})
]);