I am using the below code, to start a server that accepts http requests, and then forwards them to an https api, then relay the response.
I have a problem in that the api gives a 'Content-Encoding: gzip' response, which seems to be causing me trouble.
If I don't relay the gzip response header, the C# code I'm testing gets a response that is just random symbols, the compressed data I assume, as does Postman. When I do include as per my example below, I get StatusCode 0: "The magic number in GZip header is not correct. Make sure you are passing in a GZip stream."
I've tried passing the response headers back as:
res.writeHead(cres.statusCode, cres.headers});
But that just seems to result in the jumbled output again. How can I fix this?
const https = require('https')
const port = 55555
const requestHandler = (req, res) => {
console.log(req.url)
var options = {
host: 'my.api.com',
path: '/7f308b16-d165-4062-b00f-76970783442e'+req.url,
path: req.url,
method: 'GET',
headers: req.headers
};
var json = '';
var creq = https.request(options, function(cres) {
// set encoding
cres.setEncoding('utf8');
// wait for data
cres.on('data', function(chunk){
console.log('data: '+chunk);
json += chunk;
});
cres.on('close', function(){
console.log('close: '+cres.statusCode);
res.writeHead(cres.statusCode);
res.end();
});
cres.on('end', function(){
console.log('end: '+json.toString());
console.log(res.headers)
res.writeHead(cres.statusCode, {'Content-Encoding': 'gzip', 'Content-Type':'text/json; charset=utf-8', 'Cache-Control':'private' });
res.end(json);
});
}).on('error', function(e) {
// we got an error, return 500 error to client and log error
console.log(e.message);
res.writeHead(500);
res.end();
});
creq.end();
}
const server = http.createServer(requestHandler)
server.listen(port, (err) => {
if (err) {
return console.log('something bad happened', err)
}
console.log(`server is listening on ${port}`)
})```
Related
Basically I want to test 500+ live website URLs and get the statusCode of them using protractor and getting help from node's http module.
But I'm not getting anything in response and all console statements related to the http request are ignored in my terminal.
I've tried using some external libraries such as axios and protractor-http-client but all are giving me same result.
I'm using TypeScript.
import request from 'request' // tried this one earlier instead of http
import https from 'https';
async testFunction () {
const req = await https.get('https://www.google.com');
let body = '';
req.on('data',(d) => {
body += d;
});
req.on('end',(resp) => {
console.log(resp.statusCode); // Not printing in terminal at all
body = JSON.parse(body);
});
req.on('response', (resp) => { // also tried information instead of response
console.log(resp.statusCode); // Not printing in terminal at all
});
console.log(body); // printing as empty
}
}
I would expect the response status code be 200.
I'm new to both protractor and node. So any kind of help and suggestion would be appreciated.
const http = require('http');
const res = http.request({
host: 'google.de',
method: 'get',
path: '/'
}, (response) => {
let dataStr = [];
const receiveData = chunk => {
const b = Buffer.from(chunk);
dataStr.push(b);
};
response.on('end', () => {
console.log('status Code:', response.statusCode);
console.log('response:', Buffer.concat(dataStr).toString());
});
response.on('error', (e) => console.error(e));
response.on('data', receiveData);
});
res.end();
I am using the Node.js framework and Express module to write an API wrapper that redirects requests to another server. I can successfully redirect the request to the target server and I receive a valid response containing a JSON payload. However, after the initial request, if I try another request I get the following error.
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
This is a sample of the code I wrote for the HTTP GET Express route:
app.get('/companyrecords/:name', function(req, res) {
const options = {
protocol: 'http:',
hostname: 'myhost',
port: 5001,
method: 'GET',
path: '/path/to/resource/?name=name',
auth: 'username:password',
headers: {
Connection: 'close'
}
}
const myAppReq = http.request(options, (myAppRes) =>{
console.log(`STATUS: ${myAppRes.statusCode}`);
myAppRes.on('data', (chunk) => {
console.log(`BODY: ${chunk}`);
res.send(chunk);
});
myAppRes.on('end', () => {
res.end('No more data to send.');
});
});
myAppReq.on('error', (err) => {
console.error(`Problem with request: ${err.message}`);
});
myAppReq.write('');
myAppReq.end();
});
Not sure why I am getting this error since I am calling the req.write() method so that the request's headers are sent. When looking at the error stack trace it appears the error occurs when I call the res.send() method inside the callback to the 'data' event. Perhaps I'm not understanding the flow of execution among the request or the sequence in which the events are being emitted. Any guidance/information would be greatly appreciated.
You shouldn't be sending the response inside data event callback because response will be sent when you will receive the first chunk of data. What you should do is to write the chunk to response stream and send response inside end event callback:
const myAppReq = http.request(options, (myAppRes) =>{
myAppRes.on('data', (chunk) => {
res.write(chunk);
});
myAppRes.on('end', () => {
res.end();
});
});
Stack Overflow community, greetings. I'm trying to pass the response of a new request on the request object using the Node HTTP Module for a basic autocomplete search app for my website (i.e using Node as a proxy that will transform and redirect the requests within the server).
The flow basically is:
Client Browser - Node - ElasticSearch - Node - Client Browser
I've started with:
Listen to requests with http.createServer (function (req,res)
Get the body from the req object and use it in a new request with http.request(options, function (newReqResponse)
Get the body from that newReqResponse object and send it back to the client on the res object
The problem is that the content of newReqResponse is always outdated (trails behind the last typed character). i.e.:
If I type "te", the content of newReqResponse corresponds to that if I had typed only "t".
If I type "test", it corresponds to that if I had typed "tes".
And so on.
I've tried to solve it using Node.js streams and using the file system module to write and read files sync and async, but the result is the same. Here's a sample of the whole -code- picture:
var http = require('http');
var fs = require('fs');
http.createServer(function (req, res) {
var reqBody = '';
var newReqResponseBody = "";
req.on('data', function (chunk) {
reqBody += chunk;
fs.writeFile('reqbody.json', reqBody, function(err) {
if (err) {
throw err;
}
});
var options = {
hostname: '127.0.0.1',
port: 9500,
method: 'GET',
path: '/_search',
headers: { host: 'es',
'content-length': Buffer.byteLength(reqBody),
'content-type': 'application/json',
accept: 'application/json' },
};
var newReq = http.request(options, function (newReqResponse) {
newReqResponse.setEncoding("UTF-8");
newReqResponse.on('data', function (ch) {
newReqResponseBody += ch;
});
newReqResponse.on("end", function() {
fs.writeFile("newReqResponseBody.json", newReqResponseBody, function(err) {
if (err) {
throw err;
}
});
});
});
newReq.on("error", function(err) {
console.log(`problem with request: ${err.message}`);
});
newReq.write(reqBody);
newReq.end();
});
req.on('end', function() {
var responseBody = fs.readFileSync('newReqResponseBody.json', 'utf8');
console.log(responseBody);
res.end(responseBody);
});
}).listen(3000, '127.0.0.1');
Is there a workaround to work with requests and responses within the http server? If there isn't, I'll be very grateful if you give me any directions on how to solve this.
Since the planned use for Node is rather basic, I prefer to stick with core modules rather than having to get new ones from npm, unless that it's necessary.
Thanks in advance.
EDIT:
All I had to do was to call res.end(responseBody) within the newReqResponse.on("end") callback, which is totally counterintuitive for me, but... it works.
Glad you solved your own problem. However, I see room for improvement (not sure if you're new), especially if you're transferring data. Which you can do with streams.
You can see that I didn't calculate the content length, you're asked not to and should get ignore (for this specific case) according to HTTP specification as streams pass data in chunks with 'Transfer-Encoding': 'chunked' header.
const fs = require('fs');
const http = require('http');
const options = {
hostname: '127.0.0.1',
port: 9500,
method: 'GET',
path: '/_search',
headers: {
'Content-Type': 'application/json'
}
};
http.createServer((req, res) => {
req.pipe(fs.createWriteStream('reqBody.json'));
let request = http.request(options, (newRes) => {
newRes.pipe(res);
});
fs.createReadStream('reqBody.json').pipe(request);
res.setHeader('Content-Type', 'application/json');
}).listen(3000, '127.0.0.1');
You can shorten this snippet more if you don't want your data saved in the future and only want to pipe the req stream to request.
function delete(id, response) {
var https = require('https');
var linkpath = "/v1/endpoint/" + id + "/?token=" + AUTH_KEY;
var req = https.request({
hostname: 'api.foo.com',
port: 443,
path: linkpath,
agent: false,
method: 'DELETE',
}, (res) => {
if (res.statusCode !== 200) {
response.send('HTTP ' + res.statusCode + ' ' + res.statusMessage);
}
res.on('error', function (err) {
response.send(err);
});
res.on('end', function (data) {
response.send(data);
});
});
req.on('error', function(e) {
response.send(e.message);
});
req.end();
}
This code, adapted from my (working) code that uses a POST request to do other things with this API, nets me a status code of 500 from the endpoint.
I don't know how to debug this. I can't send the URL manually to the server because it's a DELETE operation instead of a GET or POST.
Has anyone seen this problem? Or do you have ideas on how to debug it?
Postman (https://www.getpostman.com/) is a great tool for manually sending specific HTTP requests, including DELETE!
There are all sorts of tools that will let you manually send any HTTP to the server. For instance, you can get quite a bit of information with curl, which will happily send a DELETE request.
For example:
curl -v -X "DELETE" https://jsonplaceholder.typicode.com/posts/1
will return the request and response headers as well as the body of the return value if any.
I'm trying to create the express.js proxy to an external website to obtaining audio data from there. I know about modules like http-proxy, but think they are excessive for the case when only one url-bound request goes through proxy. I'm using the code below:
express.get('/proxy', function (req, res) {
var options ={
host: "website.com",
port: 80,
path: "/audio/test.mp3",
method: 'GET'
};
http.get(options, function (audioRes) {
var data = [], dataLen = 0;
audioRes.on('data', function(chunk) {
data.push(chunk);
dataLen += chunk.length;
})
.on('end', function() {
var buf = new Buffer(dataLen);
res.set(audioRes.headers);
res.send(buf);
});
})
.on('error', function (error) {
console.log(error.message);
});
});
I get response, but it cannot be decoded as a valid audio. While debugging with the Fiddler, I found out that the number of bites sent by a server mismatches the number specified in the Content-Length header (which indicates fewer bytes being retrieved).
I cannot figure out how to properly return the exact response that's been retrieved from the remote server. Would be grateful for any help.
To send request via proxy, you can set the proxy url in Host header. Also you have to specify the full URL of the external resource you are trying to access via proxy.
var http = require("http");
var options = {
host: "proxy",
port: 8080,
path: "http://www.google.com", //full URL
headers: {
Host: "10.1.2.3" //your proxy location
}
};
http.get(options, function(res) {
console.log(res);
});
I am not sure why it is not returning full response. Can you post your options.
Update
Try this inside /proxy after putting the options
http.get(options, function (audioRes) {
audioRes.pipe(res);
})
.on('error', function (error) {
console.log(error.message);
});