Getting the below error when trying to fetch (using npm node-fetch) html from the link below:
Failed to fetch page: { FetchError: request to
https://www1.nseindia.com/marketinfo/companyTracker/compInfo.jsp?symbol=TCS&series=EQ
failed, reason: read ECONNRESET
at ClientRequest
I am using the following snippet :
const DomParser = require("dom-parser");
const parser = new DomParser();
const fetch = require("node-fetch");
router.get("/info", (req, res, next) => {
var url =
"https://www1.nseindia.com/marketinfo/companyTracker/compInfo.jsp?symbol=TCS&series=EQ";
fetch(url)
.then(function(response) {
// When the page is loaded convert it to text
return response.text();
})
.then(function(html) {
// Initialize the DOM parser
// Parse the text
var doc = parser.parseFromString(html, "text/html");
// You can now even select part of that html as you would in the regular DOM
// Example:
// var docArticle = doc.querySelector('article').innerHTML;
console.log(doc);
})
.catch(function(err) {
console.log("Failed to fetch page: ", err);
});
});
The response was consoled log few times before showing the error and now its throwing err everytime I call /info.
I have tried the snippet in repl online editor. it returns Promise {pending}.
I would use some modern promise based packages to do this job. some are got, axios etc. node-fetch was last published 8 months ago. It might not be able to handle the encoding or compressing.
here is an example using axios which works.
const axios = require("axios");
const DomParser = require("dom-parser");
const parser = new DomParser();
var url =
"https://www1.nseindia.com/marketinfo/companyTracker/compInfo.jsp?symbol=TCS&series=EQ";
axios(url)
.then(response => response.data)
.then(html => {
// Initialize the DOM parser
// Parse the text
var doc = parser.parseFromString(html, "text/html");
// You can now even select part of that html as you would in the regular DOM
// Example:
// var docArticle = doc.querySelector('article').innerHTML;
console.log(doc);
})
.catch(function(err) {
console.log("Failed to fetch page: ", err);
});
As far as I know you cannot use fetch in NodeJS. You need to rely in different methods. I personally usually use Axios.
Check also if this part of the code actually returns a promise:
return response.text()
Before chaining another .then.
https://www.valentinog.com/blog/http-js/
Related
I'm trying to use a proxy with node fetch to grab data from an api. It works fine on lumtest.com
const url = 'https://lumtest.com/myip.json';
When I'm trying to get a response from my own express test api or typicode I get the following error, note that removing the proxy agent returns the correct json response so i'm leading towards missing headers but I have also tried adding in user-agent, method, content-type but no luck:
FetchError: invalid json response body at http://localhost:8000/api/test reason: Unexpected end of JSON input
The test api in express is super simple
router.get('/test', (req,res) => {
res.json({
msg: 'Success'
})
})
The call im making with the proxy looks like the following
async function getData() {
const url = 'https://jsonplaceholder.typicode.com/todos/1';
try{
const response = await fetch(url, {
agent: new HttpsProxyAgent.HttpsProxyAgent(`https://username-zone-data_center:pass#zproxy.lum-superproxy.io:22225`)
});
const json = await response.json();
console.log('json', json);
} catch (e) {
console.log('Error: ', e)
}
}
Changing the url to const url = 'https://lumtest.com/myip.json' or https://httpbin.org/ip?json works though...
Any suggestions appreciated.
I use node.js expess with MVC pattern and use axios for get json url. The all logic request api i wrote it in Controller , I try to create some get api with axios like this.
this my chartController.js , it's some logic for get json from url with req.body.symbol (it's just symbol of stock from user submitted input form , I want to get that symbol to my axios get url in code below in const chartChartPage )
'use strict';
const axios = require('axios');
const request = require('request');
/* For Get my Api Token Data */
const dataTokens = require('../config/apitokens');
const chartChartPage = async (req,res) => {
try {
const symbol = req.body.symbol;
const url = `https://${dataTokens.sandbox}.iexapis.com/stable/stock/${symbol}/chart/1m?token=${dataTokens.tokens}`;
const fetchData = await axios.get(url);
res.status(200).json(fetchData.data);
}
catch (error){
res.status(400).send(error.message);
}
}
module.exports = {
chartPage,
chartChartPage
}
Now, i have to added some routes in my chart.js
i think i should add router.post('/', chartChartPage) for get that symbol in input when user submitted Maybe i'm wrong.
var express = require('express');
var router = express.Router();
var {chartPage , chartChartPage} = require('../controllers/chartControllers');
router.get('/', chartPage);
router.post('/', chartChartPage);
module.exports = router;
and in my chartChartPage.js (it's just js file for my template)
and use ajax to get that data from url above (with chartChartPage) to get data for build chart stock
and try to console.log that data but that's not work in console
$(function(){
chartChartPage();
});
const chartChartPage = async () => {
await $.ajax({
url: '/chart',
type: 'POST',
dataType: 'json',
success: (response) => {
if (response != null) {
console.log(response);
}
},
error: (err) => {
console.log(err);
}
});
}
and when i submitted form i got that all json data in my page, but i want when submitted it's render to same page and work in console.log in ajax get url above. How can i fix.?
enter image description here
My end goal is that I want to be able to create a test that satisfies the following statement:
verify that requests to valid URLs return a 200 HTTP status code
A valid URL for example would be /about-page or /jobs, basically any directory that I add in my content folder that contains a file with the extension /index.md.
This is my code so far:
app.js
const readFilePromise = util.promisify(fs.readFile)
app.get('/*', (req, res) => {
readFilePromise(path.join(__dirname, 'content', req.url) + '/index.md', 'utf8')
.then(data => {
convertData(data, res)
})
.catch(err => {
res.status(404).send('Page doesn\'t exist!')
})
})
const convertData = (data, res) => {
const convertedData = md.render(data)
readFilePromise(path.join(__dirname, '/template.html'), 'utf8')
.then(data => {
data = data.replace(/\{\{content\}\}/, convertedData)
res.send(data)
})
.catch(err => {
console.log(err)
})
}
app.listen(3000)
module.exports = app
After reading this article, it mentions that
Requests are asynchronous, which means you must be able to conduct asynchronous tests.
So I wrote the following test:
app.test.js
const app = require('./app.js')
const request = supertest(app)
const supertest = require('supertest')
it('Gets the test endpoint and returns a 200 status', async done => {
const res = await request.get('/*')
expect(res.status).toBe(200)
done()
})
When I run the test, it fails with a 404 status, rather than returning a 200 status. I thought this might be due to my app.js not being in the async/await style, so I changed app.js to:
const readFilePromise = util.promisify(fs.readFile)
app.get('/*', async (req, res) => {
try {
await readFilePromise(path.join(__dirname, 'content', req.url) + '/index.md', 'utf8')
} catch (err) {
res.status(404).send('Page doesn\'t exist!')
}
try {
const convertedData = md.render(data)
await readFilePromise(path.join(__dirname, '/template.html'), 'utf8')
data = data.replace(/\{\{content\}\}/, convertedData)
res.send(data)
} catch (err) {
console.log(err)
}
})
app.listen(3000)
module.exports = app
I tried running the test again, but it still fails with a 404. I think my set up within app.test.js is wrong, but I'm not sure exactly what, as I've tried using the various set ups as the article. How would I fix this?
Separately, when I try going to a URL using the async/await style in app.js, I get a ReferenceError: data is not defined error, but I'm not sure how to define data in the async/await format.
I explained here how to set up app for the test environment: supertest not found error testing express endpoint
You did not mention how you set the database environment, make sure your database is not empty. Then make your get request. but just checking status for get request is not enough because if your db is empty you will still get 200.
const response = await request(app).get("/route").send().expect(200);
expect(response.body.length).toBeGreaterThan(0)
Better approach would be connect to a different database, post your data first and then check the response
const response = await request(app).get("/api/tickets").send().expect(200);
expect(response.body.length).toEqual(2); // if you post two items
Also before you every test make sure you start with empty database inside beforeEach()
app.get("/dashboard", function(req, res) {
const country = "Singapore";
// I used a constant in desperation
const url =
"https://api.covid19api.com/live/country/" + country;
https.get(url, function(response) {
response.on("data", function(data) {
const caseData = JSON.parse(data);
// Error stems from here
res.render("cases", { caseData: caseData });
});
});
});
You're not waiting for the entire message before trying to parse it.
The data event is invoked for each chunk of data from the HTTP response. The response may not necessarily fit in a single chunk.
You'll have to wait for the end event before trying to parse the JSON... or preferably use a higher-level library such as node-fetch for a more comfortable promise-based API.
We are using navigator.sendBeacon function to send data to Koa server, in which we are using bodyparser.
If we not wrapped data into form then by default this function send data as request payload. How I can able to access this data on Koa server?
Example -
navigator.sendBeacon('http://localhost:3000/cookies/', 'test=payload')
At server, request body is blank.
Considering that
Koa does not parse request body, so you need to use either koa-bodyparser or koa-body,
koa-bodyparser by default has only json and form parsing enabled,
From your screenshot, it is clear that navigator.sendBeacon set the Content-Type to text,
You need to change the Koa server code, so that it parses text data.
Example:
const Koa = require('koa'),
bodyParser = require('koa-bodyparser'),
app = (module.exports = new Koa());
app.use(bodyParser({ enableTypes: ['json', 'text'] }));
app.use(async (ctx) => {
// ctx.request.body should contain the parsed data.
console.log('Received data:', ctx.request.body);
ctx.body = ctx.request.body;
});
if (!module.parent) app.listen(3000);
Tested with
koa 2.7.0,
koa-bodyparser 4.2.1.
Although koa doesn't parse request body and for some reason you don't want to use koa-bodyparser you can still use the raw http to collect the body from request object.
app.use(async (ctx) => {
try {
// notice that I'm wrapping event emitter in a `promise`
ctx.body = await new Promise((resolve, reject) => {
let data = '';
// this is same as your raw `http.request.on('data', () => ()`
ctx.req.on('data', chunk => {
data += chunk;
};
ctx.req.on('error', err => {
reject(err);
};
ctx.req.on('end', () => {
resolve(data);
};
});
} catch (e) {
console.error(e);
}
});