By my yet little understanding on how NodeJS/Javascript works, I believe it should be possible. Is it?
I would like to be able to send a function to a client based on http request, something like this (pseudo short code):
// Server
server.get("/function",()=>{
return function () => {
console.log('test)
}
// client
request ("server/function",(res)=>{
func = res.body
func()
// # result
// test
I have tried to convert to string to send the function from the server and convert back to an object on the client, but it return errors that I couldn't understand why as if I send a simple key value json object it works.
// server
const http = require('http');
const port = 3000;
const hostname = '127.0.0.1';
const server = http.createServer((req, res) => {
if (req.url === '/function') {
res.statusCode = 200;
function func () {
console.log('test')
}
res.end(func.toString())
}
});
server.listen(3000)
// client
const http = require('http');
const httpGet = url => {
return new Promise((resolve, reject) => {
http.get(url, res => {
res.setEncoding('utf8');
let body = '';
res.on('data', chunk => body += chunk);
res.on('end', () => resolve(body));
}).on('error', reject);
});
};
const call = async () => {
const body = await httpGet('http://localhost:3000/function')
console.log(`${body}`)
func = JSON.parse(body) // try to redefine function based on the response
}
const func = () => {
console.log('before')
}
func() // run the func defined here
call()
func() // run the function got from the request?
The idea is to have a client that will be able to execute almoust any code without the need of updating the client itself.
I'm not sure why you want to do this, it is really unsafe and you shouldn't have web applications that do this. But here goes:
Let's say you have a function:
function test() {
console.log("test")
}
And as string:
const funcString = test.toString()
// ^ is now "function test() {\n console.log(\"test\")\n }"
You need to first get the function body:
const funcBody = funcString.replace(/^[^{]+{/, '').replace(/}[^}]*$/, '')
// ^ is now "console.log(\"test\")"
Then create a function from it:
const newTest = Function(funcBody)
Now if you call newTest() you will see test printed in your console.
Note that the regular expression I have used to get the function body will not work on all functions. For example, for const test = () => 1 you will need a different regular expression.
I'm not sure this is the best idea.
Historically, the client-server relationship you are describing was inversed through remote procedure calls where the client would invoke a specific endpoint on a remote server. It sounds like your biggest draw towards having the client arbitrarily execute the code was the removal of updating the client code. What happens if you want to make backwards-incompatible changes to the server code? I think you will have better success using versioned API endpoints to execute code server-side based on client-side logic, which you will find many RPC and/or REST frameworks for within npm.
The ideia is to have a client that will be able to execute almoust any code without the need of updating the client itself.
One final thing to keep in mind are the security implications of this. What happens if I find your client and substitute in my malicious JavaScript?
Related
I know what is wrong with my code and I have looked into the best way of solving it, however with my lack of experience, I am having a hard time finding a good answer.
I need my first route(/data) to be fully completed before the second(/logo) express route sends the data. In short, I just need the variable symbolUrl to be completed before it goes into the second fetch call. Here is the code down below to explain
app.use(express.static('public'));
const url =
'https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest';
const qString =
'?CMC_PRO_API_KEY=' + process.env.apiKey + '&start=1&limit=10&convert=USD';
let symbol = [];
app.get('/data', async (req, res) => {
const fetch_res = await fetch(url + qString);
const coinData = await fetch_res.json();
for (let i = 0; i < 9; i++) {
symbol.push(coinData.data[i]['symbol']);
};
res.json(coinData);
});
app.get('/logo', async (req, res) => {
const symbolUrl = symbol.join(',');
const url2 = 'https://pro-api.coinmarketcap.com/v1/cryptocurrency/info';
const qString2 = `?CMC_PRO_API_KEY=${apiKey}%symbol=${symbolUrl}`;
const fetch_res2 = await fetch(url2 + qString2);
const coinLogo = await fetch_res2.json();
res.json(coinLogo);
});
The issue I am trying to solve with this project is that I want to send the data(/data) to be sent to the front end first because this API call will load the majority of the page. Then my second call will load images and other larger files afterward. HOWEVER, the API I am working with to get the logos(images) of the specific crypto coins I want, I need a different endpoint as well as use %symbol=${symbolUrl} in the API call to get the correct tokens I want to call.
client code:
fetch('http://localhost:2000/data')
.then(async (response) => {
return response.json();
})
.then(async (data) => {
const parsedData = data['data'];
// console.log(data['data'][0]['name'])
await parsedData.forEach((element) => {
// this just has about 20 lines of code generating the the look of the page. It works as intended
});
fetch('http://localhost:2000/logo')
.then(async (response) => {
return response.json();
})
.then(async (logo) => {
console.log(logo)});
***I have tried putting this in an async function and awaiting the first fetch call
All I need to be done is for app.get(/data) to be fully complete before doing my second app.get. I have done testing and I know that is the issue. I apologize if it is something easy, but I couldn't find anything on making an app.get synchronous and I have tried putting both in a async function, however that did not work.
You cannot send responses in fragments like you're trying to do, it would throw an error saying Can't set headers after they are sent to client
The proper method to implement what you are trying to do is to define the first layer as middleware, and then allow the second layer to return the response. Here layer basically means a function handler.
In order to control when the execution passes to the next layer / next function handler, express has a third parameter (request, response, next). You're only using request and response, researching about next will solve your concern.
Express next function, what is it really for?
First handler
app.get('something_unique', async (req, res, next) => {
// do whatever you want to do first
// save data into res.locals
res.locals.foo = {...}
next()
})
Second Handler
app.get('something_unique', (req, res) => {
const data = res.locals.foo;
// whatever you want
return res.json({ anything })
})
More:
Express next function, what is it really for?
Error: Can't set headers after they are sent to the client
Passing variables to the next middleware using next() in Express.js
I'm not sure what client code you're really running as it sounds like you've bee trying several things, but this should work to sequence the /data request and the /logo request so that the /logo request is not run until the response from the /data request has been received.:
async function run() {
const r1 = await fetch('http://localhost:2000/data');
const data = await r1.json();
const parsedData = data.data;
parsedData.forEach((element) => {
// this just has about 20 lines of code generating
// the the look of the page. It works as intended
});
const r2 = await fetch('http://localhost:2000/logo');
const logo = await r2.json();
return logo;
}
run().then(logo => {
console.log(logo);
}).catch(err => {
// handle errors here
console.log(err);
});
If there is any asynchronous code inside the .forEach(), then we will have to see that also to properly sequence that.
As I've said in my comments, stuffing the data from the first request into a server-side variable is probably the wrong design on the server because two separate clients both issuing /data requests will conflict with one another, creating race conditions. But, you haven't explained what this data is really for or why you're stuffing it into a variable on the server for us to suggest an alternate design.
What are the consequences of making request global (or singleton), so that it is accessible all over the server and does not have to be passed in each function call? For example:
index.js:
const http = require('http');
const { saveReq } = require('./shared');
const {
doSomethingWithReqPassingItAsParameter,
doSomethingWithReqPassingItAsGlobal
} = require('./lib');
const requestListener = function (req, res) {
// approach 1
doSomethingWithReqPassingItAsParameter(req);
// approach 2
saveReq(req);
doSomethingWithReqPassingItAsGlobal();
res.writeHead(200);
res.end('Hello, World!');
}
const server = http.createServer(requestListener);
server.listen(8080);
lib.js:
const { loadReq } = require('./shared');
const doSomethingWithReqPassingItAsParameter = (req) => {
console.log('req as parameter', req.url);
};
const doSomethingWithReqPassingItAsGlobal = () => {
console.log('req as global', loadReq().url);
};
module.exports = {
doSomethingWithReqPassingItAsParameter,
doSomethingWithReqPassingItAsGlobal,
};
shared.js
var request;
const saveReq = (r) => request = r;
const loadReq = () => request;
module.exports = {
saveReq,
loadReq,
}
This is very convenient for large projects with many levels of function calls, but how parallel requests will be handled? I know that nodejs is single-threaded, does it mean than each http request will be run from end to finish separately or they can overlap, thus using a global request object would make a mess?
The consequences are that your server will only work for one request at a time and as soon as you have more than one request in flight at the same time, data will be mixed up between requests dealing to bugs, crashes, security issues and incorrect results.
Simply put, you cannot program a server that way. Pass the req object or data from it to any functions that need it. That keeps the appropriate req object associated with the right execution to avoid all the problems of trying to store a req object in some sort of global location where multiple requests in flight at the same time will/can conflict.
There is a relatively new thing in nodejs called "async local storage" that could perhaps be used for this. You can read a little about it here, though it's my personal opinion that it's still better to pass your request data to the functions that want to use it rather than the async local storage for this.
I hope you could please help me out, I'm running Node.js and trying to get a the city name from a API and it keeps showing an error
saying it Cannot read property city_name of undefined.
It gets stuck on this line in the code:
const cityName = weatherData.data.city_name;
Any clue why its doing that? Please
// Creating the server of the weather app
const express = require('express');
const app = express();
const { StringDecoder } = require('string_decoder');
const decoder = new StringDecoder('utf8');
const https = require('https');
app.get('/', (req, res) => {
const weatherPath = "https://api.weatherbit.io/v2.0/current?key=41c0f84d717a4764a26d144aa33a9443&city=melbourne,Australia"
// Calling the weather app
https.get(weatherPath, (response) => {
console.log(response.statusCode);
// Getting the data from the weather app
response.on('data', (d) => {
//console.log(d);
// Converting the buffer data from the weather app
console.log(decoder.write(d));
const weatherData = decoder.write(d);
const cityName = weatherData.data.city_name;
console.log(cityName);
});
});
res.send("The server is up and running on the web");
});
app.listen(3000, () =>
{
console.log('Server is running on port 3000');
});
Your data is a string and hence doesn't have those properties. You would need to JSON.parse it first.
But there is another issue, your code will break as soon as more data is returned, because you listen only for a single chunk of data. You have to add up all chunks (add to the existing chunks on every data event) and process the full data on the end event.
But in general the https.get method is very bare-bones, it will be a lot simpler to use a package like node-fetch:
// Creating the server of the weather app
const express = require('express');
const app = express();
const fetch = require('node-fetch')
app.get('/', (req, res) => {
const weatherPath = "https://api.weatherbit.io/v2.0/current?key=41c0f84d717a4764a26d144aa33a9443&city=melbourne,Australia"
// Calling the weather app
fetch(weatherPath)
.then(response => response.json())
.then(weatherData => {
// Getting the data from the weather app
const cityName = weatherData.data[0].city_name;
console.log(cityName);
}).catch(e => {
console.error('An error occured!', e);
});
res.send("The server is up and running on the web");
});
app.listen(3000, () => {
console.log('Server is running on port 3000');
});
Additional information
To address your comment:
I converted the data into a string though in the command line it looked like a JSON.
The term "JSON" is often used in a confusing way. Technically JSON (JavaScript Object Notation) is a serialization format, a string representation of an object or other basic JavaScript datatype (with limitations). The concept of a "live" object exists only in memory inside your script. So, the API is always sending you a string of characters. This string "is" JSON, i.e. uses JSON as method of representing structured data that, when parsed (!), can be turned back into a JavaScript object (in memory). So you are right that it looked like JSON since it is, but it's still a string at that point.
It's like sending you a blueprint (2D representation - JSON string) of a house (3D object - the original object). (You obviously can't send a house in a letter so people are sending blueprints (JSON) instead.) It looks like a house, because it is representing one, but you can't yet open its door (access a property) or something. At that point it's still just something printed on a piece of paper (a string) that people recognize as a blueprint (it is valid JSON). You have to first build an actual house (parse the JSON back into an object) from the blueprint.
(Of course this isn't made any better by using a variable name like json to represent the data parsed from JSON like it sometimes happens.)
I tried to hit the API and the response :
{"data":[{"rh":73,"pod":"n","lon":144.96332,"pres":1025.6,"timezone":"Australia\/Melbourne","ob_time":"2020-07-09 09:05","country_code":"AU","clouds":50,"ts":1594285500,"solar_rad":0,"state_code":"07","city_name":"Melbourne","wind_spd":1,"wind_cdir_full":"north-northwest","wind_cdir":"NNW","slp":1026.3,"vis":5,"h_angle":-90,"sunset":"07:16","dni":0,"dewpt":8.2,"snow":0,"uv":0,"precip":0,"wind_dir":348,"sunrise":"21:34","ghi":0,"dhi":0,"aqi":61,"lat":-37.814,"weather":{"icon":"c02n","code":"802","description":"Scattered clouds"},"datetime":"2020-07-09:09","temp":12.8,"station":"E5657","elev_angle":-20.02,"app_temp":12.8}],"count":1}
Edit :
i dont try your code in application before, i try by browser
This is mycode
// Calling the weather app
https.get(weatherPath, (response) => {
response.setEncoding('utf8')
let chunks = []
// Getting the data from the weather app
response.on('data', (d) => {
chunks.push(d);
});
response.on('end', () => {
let data = JSON.parse(chunks.join(''))
console.log(data.data[0].city_name)
});
});
weather.data is an array, so when you try access weather.data.city_name will be undefined. You must access weather.data[0].city_name.
I have solved using advance node module used popularly called axios
Have a look at Code,
// Creating the server of the weather app
const express = require('express');
const app = express();
const axios = require('axios');
app.get('/', (req, res) =>
{
//In production we do not need this
process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = 0;
const weatherPath = "https://api.weatherbit.io/v2.0/current?key=41c0f84d717a4764a26d144aa33a9443&city=melbourne,Australia";
axios.get(weatherPath)
.then(function (response)
{
// handle success
let cityName = response.data.data[0].city_name;
console.log(cityName);
})
.catch(function (error)
{
// handle error
console.log(error);
})
.finally(function ()
{
// always executed
});
res.send("The server is up and running on the web");
});
app.listen(3000, () =>
{
console.log('Server is running on port 3000');
});
I'm trying to write some tests using Lab and Sinon for various HTTP requests that are called in a file of mine. I followed the Fake XMLHttpRequest example at http://sinonjs.org/ but when I run my tests it appears to not actually capture any requests.
Here is the (relevant) testing code:
context('when provided a valid payload', function () {
let xhr;
let requests;
before(function (done) {
xhr = sinon.useFakeXMLHttpRequest();
requests = [];
xhr.onCreate = function (req) { requests.push(req); };
done();
});
after(function (done) {
// clean up globals
xhr.restore();
done();
});
it('responds with the ticket id', (done) => {
create(internals.validOptions, sinon.spy());
console.log(requests); // Logs empty array []
done();
});
});
create is the function I imported from the other file, here:
internals.create = async function (request, reply) {
const engineeringTicket = request.payload.type === 'engineering';
const urgentTicket = request.payload.urgency === 'urgent';
if (validation.isValid(request.payload)) {
const attachmentPaths = formatUploads(request.payload.attachments);
const ticketData = await getTicket(request.payload, attachmentPaths);
if (engineeringTicket) {
const issueData = getIssue(request.payload);
const response = await jira.createIssue(issueData);
jira.addAttachment(response.id, attachmentPaths);
if (urgentTicket) {
const message = slack.getMessage(response);
slack.postToSlack(message);
}
}
zendesk.submitTicket(ticketData, function (error, statusCode, result) {
if (!error) {
reply(result).code(statusCode);
} else {
console.log(error);
}
});
} else {
reply({ errors: validation.errors }).code(400); // wrap in Boom
}
};
as you can see it calls jira.createIssue and zendesk.submitTicket, both of which use an HTTP request to post some payload to an API. However, after running the test, the requests variable is still empty and seems to have captured no requests. It is definitely not actually submitting the requests as no tickets/issues have been created, what do I need to fix to actually capture the requests?
Your problem is apparent from the tags: you are running the code in NodeJS, but the networking stubs in Sinon is for XMLHttpRequest, which is a browser specific API. It does not exist in Node, and as such, the setup will never work.
That means if this should have worked you would have needed to run the tests in a browser. The Karma test runner can help you with this if you need to automate it.
To make this work in Node you can either go for an approach where you try to stub out at a higher level - meaning stubbing the methods of zendesk and jira, or you can continue with the approach of stubbing network responses (which makes the tests a bit more brittle).
To continue stubbing out HTTP calls, you can do this in Node using Nock. Saving the requests like you did above is done like this:
var requests = [];
var scope = nock('http://www.google.com')
.get('/cat-poems')
.reply(function(uri, requestBody) {
requests.push( {uri, requestBody} );
});
To get some insights on how you can stub out at a higher level, I wrote this answer on using dependency injection and Sinon, while this article by Morgan Roderick gives an intro to link seams.
I am trying to implement the ._read function of a readable stream, a problem happens when ._read is called and there isn't data, the documentation says that I can push('') until more data comes, and I should only return false when the stream will never have more data.
https://nodejs.org/api/stream.html#stream_readable_read_size_1
But it also says that if I need to do that then something is terribly wrong with my design.
https://nodejs.org/api/stream.html#stream_stream_push
But I can't find an alternative to that.
code:
var http = require('http');
var https = require('https');
var Readable = require('stream').Readable;
var router = require('express').Router();
var buffer = [];
router.post('/', function(clientRequest, clientResponse) {
var delayedMSStream = new Readable;
delayedMSStream._read = function() {
var a=buffer.shift();
if(typeof a === 'undefined'){
this.push('');
return true;
}
else {
this.push(a);
if(a===null) {
return false;
}
return true;
}
};
//I need to get a url from example.com
https.request({hostname:'example.com'}, function(exampleResponse){
data='';
exampleResponse.on('data',function(chunk){data+=chunk});
exampleResponse.on('end',function(){
var MSRequestOptions = {hostname: data, method: 'POST'};
var MSRequest = https.request(MSRequestOptions, function(MSResponse){
MSResponse.on('end', function () {
console.log("MSResponse.on(end)");//>>>
});//end MSResponse.on(end)
}); //end MSRequest
delayedMSStream.pipe(MSRequest);
});
});
clientRequest.on('data', function (chunk) {
buffer.push(chunk);
});
clientRequest.on('end', function () {//when done streaming audio
buffer.push(null);
});
});//end router.post('/')
explanation:
client sends a POST request streaming audio to my server, my server requests a url from example.com, when example.com responds with the url, my server streams the audio to it.
What's a smarter way to do it?
So if I undertstand the code correctly, you:
receive a request,
make your own request to a remote endpoint and fetch a URL
make a new request to that URL and pipe that to original response.
There are ways to do this other then yours, and even your way would look cleaner to me if you just improve the naming a bit. Also, splitting the huge request into a few functions with smaller responsibility scopes might help.
I would make the endpoint this way:
let http = require('http');
let https = require('https');
let Readable = require('stream').Readable;
let router = require('express').Router();
let buffer = [];
/**
* Gets some data from a remote host. Calls back when done.
* We cannot pipe this directly into your stream chain as we need the complete data to get the end result.
*/
function getHostname(cb) {
https.request({
hostname: 'example.com'
}, function(response) {
let data = '';
response.on('error', err => cb(err)); // shortened for brewity
response.on('data', function(chunk) {
data = data + chunk;
});
response.on('end', function() {
// we're done here.
cb(null, data.toString());
});
});
}
router.post('/', function(request, response) {
// first let's get that url.
getHostname(function(err, hostname) {
if (err) { return response.status(500).end(); }
// now make that other request which we can stream.
https.request({
hostname: hostname,
method: 'POST'
}, function(dataStream) {
dataStream.pipe(response);
});
});
});
Now, as said in the comments, with streams2, you don't have to manage your streams. With node versions pre 0.10 you have had to listen to 'read', 'data' etc events, with newer node versions, it's handled. Furthermore, you don't even need it here, streams are smart enough to handle backpressure on their own.