Get the HTML content of a page after the JS loading - node.js

Question

I'm trying to get the results of a search in the Rust documentation. I made this code to do it :
let HTMLParser = require('node-html-parser');
let https = require('https');
const search = "foo";
let options = {
host: "doc.rust-lang.org",
path: "/std/index.html?search=" + search
};
let request = https.get(options, (res) => {
if (res.statusCode != 200) return console.log(`An error occured : ${res.statusCode}. Retry later.`);
res.setEncoding("utf8");
let output = "";
res.on("data", (chunk) => {
output += chunk
});
res.on("end", () => {
let root = HTMLParser.parse(output);
console.log(root.querySelector(".search-results")); // print "null" because the search is not done when the request response come
});
request.end();
});
But when I run this code, I get the HTML content of the index.html page like if I requested this page without the ?search="foo". I found that the page change dynamically with some JS when we search for something, and then the base content is set to hidden and the search div become visible. So it seems that the JS didn't load when I get the request result, but I needs it to get the results of the search in the documentation. I don't know how I can do that.
Thank you in advance for your answers !

Answer 1

The Rust doc page does not seem to hit a backend when a search is performed. I noticed this using the browser developer tools.
It looks like the page loads a search-index which contains the readily available docs. You can use this js to search for docs. The logic is written in the main.js.
Let me know if you are looking for more info, as I have not found out how the link generation on each doc item is created.
EDIT
All the logic required to build the url is in main.js. The method is as follows. If you take a close look at the aliases.js, main.js, storage.js and search-index.js files, you can reuse almost all of it to create the links and the required search outputs.
function buildHrefAndPath(item) {
var displayPath;
var href;
var type = itemTypes[item.ty];
var name = item.name;
if (type === 'mod') {
displayPath = item.path + '::';
href = rootPath + item.path.replace(/::/g, '/') + '/' + name + '/index.html'
} else if (type === 'primitive' || type === 'keyword') {
displayPath = '';
href = rootPath + item.path.replace(/::/g, '/') + '/' + type + '.' + name + '.html'
} else if (type === 'externcrate') {
displayPath = '';
href = rootPath + name + '/index.html'
} else if (item.parent !== undefined) {
var myparent = item.parent;
var anchor = '#' + type + '.' + name;
var parentType = itemTypes[myparent.ty];
if (parentType === 'primitive') {
displayPath = myparent.name + '::'
} else {
displayPath = item.path + '::' + myparent.name + '::'
}
href = rootPath + item.path.replace(/::/g, '/') + '/' + parentType + '.' + myparent.name + '.html' + anchor
} else {
displayPath = item.path + '::';
href = rootPath + item.path.replace(/::/g, '/') + '/' + type + '.' + name + '.html'
}
return [displayPath,
href]
}