I have an axios request that gets a webpage and I need pull a string from the response data between 2 reference strings.
Example: I want to get Stuff from the response data string.
"theID":1234,"theMap":"Stuff"}]};
axios
.get(url)
.then((response) => {
const firstvariable = `theMap":"`;
const secondvariable = `"}]}`;
const data = response.data;
const source = data.match(
new RegExp(firstvariable + "(.*)" + secondvariable)
);
console.log(source); // should be Stuff
})
.catch((err) => {});
Regardless of what regex I try from searching on stackoverflow / googling, nothing works.
By default, Axios takes care of deserializing the JSON into a plain ol' object. So, all you need is something like this:
const response = await axios.get(url);
// since didn't actually give us an example of your JSON,
// you'll need to figure out the correct path.
const stuff = response.data?.path?.to?.stuff;
Related
As far as I know (correct me if i'm wrong please) the flow of downloading a file should be that the frontend make a call to an api route and everything else is going on on the server.
My task was to read from firestore and write it to the CSV file, I populated the CSV file with the data and now when I try to send it to the frontend only thing that is in the file after the download it the first line containing headers name and email (the file that was written on my computer is correctly willed with the data). This is my route
import { NextApiHandler } from "next";
import fs from "fs";
import { stringify } from "csv-stringify";
import { firestore } from "../../firestore";
import { unstable_getServerSession } from "next-auth/next";
import { authOptions } from "./auth/[...nextauth]";
const exportFromFirestoreHandler: NextApiHandler = async (req, res) => {
const session = await unstable_getServerSession(req, res, authOptions);
if (!session) {
return res.status(401).json({ message: "You must be authorized here" });
}
const filename = "guestlist.csv";
const writableStream = fs.createWriteStream(filename);
const columns = ["name", "email"];
const stringifier = stringify({ header: true, columns });
const querySnapshot = await firestore.collection("paprockibrzozowski").get();
await querySnapshot.docs.forEach((entry) => {
stringifier.write([entry.data().name, entry.data().email], "utf-8");
});
stringifier.pipe(writableStream);
const csvFile = await fs.promises.readFile(
`${process.cwd()}/${filename}`,
"utf-8"
);
res.status(200).setHeader("Content-Type", "text/csv").send(csvFile);
};
export default exportFromFirestoreHandler;
since I await querySnapshot and await readFile I would expect that the entire content of the file would be sent to the frontend. Can you please tell me what am I doing wrong?
Thanks
If anyone will struggle with this same stuff here is the answer base on the # Nelloverflowc thank you for getting me this far, hoverver files not always were populated with data, at first I tried like so
stringifier.on("close", async () => {
const csvFile = fs.readFileSync(`${process.cwd()}/${filename}`, "utf-8");
res
.status(200)
.setHeader("Content-Type", "text/csv")
.setHeader("Content-Disposition", `attachment; filename=${filename}`)
.send(csvFile);
});
stringifier.end();
the api of https://csv.js.org/ must have changed becuase instead of on.('finish') it is on close now, so reading file sync did the job regarding always getting the file populated with the correct data, but along with it there was an error
API resolved without sending a response for /api/export-from-db, this may result in stalled requests.
the solution to that is to convert file into readable stream like so
try {
const csvFile = fs.createReadStream(`${process.cwd()}/${filename}`);
res
.status(200)
.setHeader("Content-Type", "text/csv")
.setHeader("Content-Disposition", `attachment; filename=${filename}`)
.send(csvFile);
} catch (error) {
res.status(400).json({ error });
}
Here is the tread and the discussion that helped me
Node.js send file in response
The await on that forEach is most definitely not doing what you expect it to do, also you probably shouldn't use await and forEach together
Either switch to using the Sync API for the csv-stringify library or do something along these lines (assuming the first .get() actually contains the actual values from a promise):
[...]
stringifier.pipe(writableStream);
stringifier.on('finish', () => {
const csvFile = await fs.promises.readFile(
`${process.cwd()}/${filename}`,
"utf-8"
);
res.status(200).setHeader("Content-Type", "text/csv").send(csvFile);
});
for (const entry of querySnapshot.docs) {
stringifier.write([entry.data().name, entry.data().email], "utf-8");
);
stringifier.end();
[...]
I'm trying to find a specific text. In my case, I have no idea of selectors, elements, parents, or anything else in the HTML code of the target. Just trying to find out if this page has robots.txt. Doing that by searching for 'User-agent:'.
Is someone who knows how to search for a specific text in the parse, without knowing any other piece of information on the page?
getApiTest = async () => {
axios.get('http://webilizerr.com/robots.txt')
.then(res => {
const $ = cheerio.load(res.data)
console.log($(this).text().trim() === 'User-agent:'
)
}).catch(err => console.error(err))
};
Thanks for your time.
You can simply use a regular expression to check whether "User-agent" is part of the returned HTML.
Be aware: If the scraped page doesn't have a robots.txt file and returns a 404 status code, which should normally be the case, axios throws an error. You should consider this in your catch statement.
Following a working example:
const axios = require("axios");
const cheerio = require("cheerio");
const getApiTest = async () => {
try {
const res = await axios.get("https://www.finger.digital/robots.txt");
const $ = cheerio.load(res.data);
const userAgentRegExp = new RegExp(/User-agent/g);
const userAgentRegExpResult = userAgentRegExp.exec($.text());
if (!userAgentRegExpResult) {
console.log("Doesn't have robots.txt");
return;
}
console.log("Has robots.txt");
} catch (error) {
console.error(error);
console.log("Doesn't have robots.txt");
}
};
getApiTest();
Here is what I am trying to do.
I am introducing functionality to enable users to search for local restaurants.
I created a HTTP cloud function, so that when the client delivers a keyword, the function will call an external API to search for the keyword, fetch the responses, and deliver the results.
In doing #2, I need to make two separate url requests and merge the results.
When I checked, the function does call the API, fetch the results and merge them without any issue. However, for some reason, it only returns null to the client.
Below is the code: could someone take a look and advise me on where I went wrong?
exports.restaurantSearch = functions.https.onCall((data,context)=>{
const request = data.request;
const k = encodeURIComponent(request);
const url1 = "an_url_to_call_the_external_API"+k;
const url2 = "another_url_to_call_the_external_API"+k;
const url_array = [ url1, url2 ];
const result_array = [];
const info_array = [];
url_array.forEach(url=>{
return fetch(url, {headers: {"Authorization": "API_KEY"}})
.then(response=>{
return response.json()
})
.then(res=>{
result_array.push(res.documents);
if (result_array.length===2) {
const new_result_array_2 = [...new Set((result_array))];
new_result_array_2.forEach(nra=>{
info_array.push([nra.place_name,nra.address_name])
})
//info_array is not null at this point, but the below code only return null when checked from the client
return info_array;
}
})
.catch(error=>{
console.log(error)
return 'error';
})
})
});
Thanks a lot in advance!
You should use Promise.all() instead of running each promise (fetch request) separately in a forEach loop. Also I don't see the function returning anything if result_array.length is not 2. I can see there are only 2 requests that you are making but it's good to handle all possible cases so try adding a return statement if the condition is not satisfied. Try refactoring your code to this (I've used an async function):
exports.restaurantSearch = functions.https.onCall(async (data, context) => {
// Do note the async ^^^^^
const request = data.request;
const k = encodeURIComponent(request);
const url1 = "an_url_to_call_the_external_API" + k;
const url2 = "another_url_to_call_the_external_API" + k;
const url_array = [url1, url2];
const responses = await Promise.all(url_array.map((url) => fetch(url, { headers: { "Authorization": "API_KEY" } })))
const responses_array = await Promise.all(responses.map((response) => response.json()))
console.log(responses_array)
const result_array: any[] = responses_array.map((res) => res.documents)
// Although this if statement is redundant if you will be running exactly 2 promises
if (result_array.length === 2) {
const new_result_array_2 = [...new Set((result_array))];
const info_array = new_result_array_2.map(({place_name, address_name}) => ({place_name, address_name}))
return {data: info_array}
}
return {error: "Array length incorrect"}
});
If you'll be running 2 promises only, other option would be:
// Directly adding promises in Promise.all() instead of using map
const [res1, res2] = await Promise.all([fetch("url1"), fetch("url2")])
const [data1, data2] = await Promise.all([res1.json(), res2.json()])
Also check Fetch multiple links inside of forEach loop
I have mocked up the response from the front end in Node.js as seen below.
the attestationObject parameter is what is returned once the Yubikey has signed the challenge and its been converted to base64 for transport to the node server.
What i'm getting is an ArrayBuffer { byteLength: 226 } but I have no idea what to do with it.
I know i need to check the domain name the signed it and I need to store something with the users credentials so they can login again.
I understand there is loads of options, I just want to get a bare minimum passwordless register and login working.
const cbor = require("cbor");
const attestationObject = "o2NmbXRkbm9uZWdhdHRTdG10oGhhdXRoRGF0YVjE4mQ5WmgO3yl24XjxRqkP9LjqRYP-GsIubALB-5K_CK5FXMrOUa3OAAI1vMYKZIsLJfHwVQMAQABcapsmHtrsLJtfZ7RDcRm0iDgMlc5-CuP2XcNOwDy0uU2mU44ENk-EqtthH7huq8AipYfY0EvmfPRqQI-zI5GlAQIDJiABIVggZplpmQSKsJvg78INyrQUgBo9dv0vaZL6Qp15rOd6wMQiWCAx-ZeQ6T_xTMlY9cG3EWY54wT9Hd6EX7P7Ak-9uwauCA"
const clientDataJSON = "eyJjaGFsbGVuZ2UiOiJlVGR1TjJGaGFIaHhhRFJzT0RsdU1qTnRhMjgiLCJvcmlnaW4iOiJodHRwczovL2UzMDI3MTU3Lm5ncm9rLmlvIiwidHlwZSI6IndlYmF1dGhuLmNyZWF0ZSJ9"
const id = "AFxqmyYe2uwsm19ntENxGbSIOAyVzn4K4_Zdw07APLS5TaZTjgQ2T4Sq22EfuG6rwCKlh9jQS-Z89GpAj7MjkQ"
const rawid = "AFxqmyYe2uwsm19ntENxGbSIOAyVzn4K4_Zdw07APLS5TaZTjgQ2T4Sq22EfuG6rwCKlh9jQS-Z89GpAj7MjkQ"
convertToBuffer(attestationObject)
.then((buffer) => {
return parseAttestationObject(buffer)
})
.then((json) => {
console.log(json)
})
.catch((err) => {
console.log(err)
})
function convertToBuffer(base64) {
return new Promise((resolve, reject) => {
if (typeof base64 === "string") {
base64 = base64.replace(/-/g, "+").replace(/_/g, "/");
base64 = Buffer.from(base64, "base64");
base64 = new Uint8Array(base64);
resolve(base64.buffer);
}
})
}
function parseAttestationObject(attestationObject){
return new Promise((resolve, reject) => {
const authData = cbor.decodeAllSync(Buffer.from(attestationObject));
const authnrDataArrayBuffer = authData[0].authData.buffer;
console.log(authnrDataArrayBuffer)
// What do I do with this authnrDataArrayBuffer? What needs saving to the database?
})
}
It would be helpful if you would be a bit more exact about the exact issue, but in a nutshell:
You'd want to store the rawI This is the identifier that you need to pass in the allowCredentials object in the authentication step, so you'll need it.
The attestationobject is a CBOR encoded value. After some manipulations you should be able to extract a public key from this. You'll be able to to use this certificate to verify the response from the authenticator in the authentication step.
I'm leaving out any specific implementation steps, but please have a look at https://github.com/fido-alliance/webauthn-demo as this project implements webauthn for node.js as well, so you should be able to extract all relevant code from it.
// this is your attestationObject which is web safe base64 encode string
var attestationObject = "o2NmbXRkbm9uZWdhdHRTdG10oGhhdXRoRGF0YVjE4mQ5WmgO3yl24XjxRqkP9LjqRYP-GsIubALB-5K_CK5FXMrOUa3OAAI1vMYKZIsLJfHwVQMAQABcapsmHtrsLJtfZ7RDcRm0iDgMlc5-CuP2XcNOwDy0uU2mU44ENk-EqtthH7huq8AipYfY0EvmfPRqQI-zI5GlAQIDJiABIVggZplpmQSKsJvg78INyrQUgBo9dv0vaZL6Qp15rOd6wMQiWCAx-ZeQ6T_xTMlY9cG3EWY54wT9Hd6EX7P7Ak-9uwauCA";
// need to convert to base64 encode string
attestationObject = attestationObject.replace(/\-/g, '+').replace(/_/g, '/') + '=='.substring(0, (3*attestationObject.length)%4);
// do a base64 decode
var attCbor = Buffer.from(attestationObject, 'base64');
// decode to have CBOR object, using cbor module
const cbor = require("cbor");
var attCborObj = cbor.decodeAllSync(attCbor)[0];
console.log(attCborObj);
I'm trying to consolidate a bunch of route usage throughout my Express API, and I'm hoping there's a way I can do something like this:
const app = express()
const get = {
fetchByHostname({
name
}) {
return `hey ${name}`
}
}
const map = {
'/public/hostname/:hostname': get.fetchByHostname
}
app.use((req, res, next) => {
const url = req.originalUrl
const args = { ...req.body, ...req.query }
const method = map[url] // this won't work
const result = method(args)
return res.json({
data: result
})
})
I'm trying to avoid passing round the req and res objects and just handle the response to the client in one place. Is there an Express/Node/.js module or way to match the URL, like my map object above?
I really don't understand what you are trying to achieve, but from what i can see, your fectchByHostname({name})should be fetchByHostname(name) and you might be able to return hey $name. You should be sure you are using ES6 because with you args. Else you have to define the as in es5 args = {body: req.body, query: req.query};. Hope it helps.