I'm ingesting a codebase that is a React-NodeJS stack. One of the concepts that I am trying to grasp is regarding the back-end API and how its handled client side.
The codebase is essentially dumping an entire collection from MongoDB with an API call, and then doing a good amount of parsing and client side logic with React in order to render custom views. The HTTP responses here are pretty large, and will only get larger as data is added to the DB.
Is there any advantage/disadvantage to this approach, as opposed to creating multiple endpoints in NodeJS, and utilizing something like Mongoose to return filtered data to the client, making rendering easy and responses smaller.
Things to take into consideration could be resource consumption, how this would be billed if in the cloud, the impact of SPA's, etc.
Hopefully i get some more clarity at the end of this?
Client-side processing is best because as you know our server-side resources are free and can easily handle requests.but Sending large amounts of data to the client for processing will incur client overhead and make their browsing experience less acceptable, data security may be compromised, or the network may be overwhelmed and bandwidth consumed. Processing data server side will increase your server load per client.
So to avoid these issues, it is best to first hand over some of these conflicts to the side of the database handled (with filtering and special conditions) and then filter the security data processed in server side with command and coding that Do not send clients.
But let's do the heavy processing on the client machine. SPA has other benefits, of course.
So I mostly do server-side processing - unless it's really basic stuff like simple sorting etc etc.
Also, don't assume JavaScript is enabled. You have to fall back gracefully and that would require the server to do the processing anyhow.
this link say to you differences between server-side and client-side programming
Related
I'm trying to learn Node.js and adequate design approaches.
I've implemented a little API server (using express) that fetches a set of data from several remote sites, according to client requests that use the API.
This process can take some time (several fecth / await), so I want the user to know how is his request doing. I've read about socket.io / websockets but maybe that's somewhat an overkill solution for this case.
So what I did is:
For each client request, a requestID is generated and returned to the client.
With that ID, the client can query the API (via another endpoint) to know his request status at any time.
Using setTimeout() on the client page and some DOM manipulation, I can update and display the current request status every X, like a polling approach.
Although the solution works fine, even with several clients connecting concurrently, maybe there's a better solution?. Are there any caveats I'm not considering?
TL;DR The approach you're using is just fine, although it may not scale very well. Websockets are a different approach to solve the same problem, but again, may not scale very well.
You've identified what are basically the only two options for real-time (or close to it) updates on a web site:
polling the server - the client requests information periodically
using Websockets - the server can push updates to the client when something happens
There are a couple of things to consider.
How important are "real time" updates? If the user can wait several seconds (or longer), then go with polling.
What sort of load can the server handle? If load is a concern, then Websockets might be the way to go.
That last question is really the crux of the issue. If you're expecting a few or a few dozen clients to use this functionality, then either solution will work just fine.
If you're expecting thousands or more to be connecting, then polling starts to become a concern, because now we're talking about many repeated requests to the server. Of course, if the interval is longer, the load will be lower.
It is my understanding that the overhead for Websockets is lower, but still can be a concern when you're talking about large numbers of clients. Again, a lot of clients means the server is managing a lot of open connections.
The way large services handle this is to design their applications in such a way that they can be distributed over many identical servers and which server you connect to is managed by a load balancer. This is true for either polling or Websockets.
I'm on the newer side of web development. I use React so obviously I focus on client side rendering, but for a certain application I was thinking to make a request to my server for every page (this probably isn't necessary but just a workaround due to ignorance on my part); however, the thought came to me, how many fetch requests are too much?
I want to divide this a little bit, I know different fetch requests can take different amounts of time, a GET request for 1 item of data is faster than a POST request that adds 20 rows and you can't account for all the variations.
But in general,
How long does a fetch request to a server (performing some sort of CRUD operation on the database) take?
** 2. How long does a fetch request to a server (NOT performing any operation to a database) take?**
Option 2 is obviously faster (if we're just imagining simple requests) and I know this can probably vary from server to server, but I think it would be helpful to know, so I can structure my site more efficiently and have some sort of knowledge about this topic?
There is no general way to know how long a request will take, it depends on so many factors like internet speed (assuming it is not a local server), the amount of data being submitted (in a POST) or retrieved, the amount of processing done on the server before returning the response etc.
If you have the answers to all of the above questions you can just calculate the time as a simple calculation.
I need to track moving cars.
Should I post the location every time the location changes, and send it over the socket?
Or should make a REST API and post the location (from the tracked device) and check it (with the tracker device) every 10 seconds, regardless if the location changed or not?
(The App is being made with React Native)
Building HTTP requests by frequent updates requires more resources then sending messages through websocket. Keeping websocket connections open by a lot of users requires more resources than using HTTP. In my opinion the answer depends on the user count, the update frequency, whether you apply the REST constraints (no server side session) and which version of HTTP you use (HTTP2 is more efficient than HTTP1.1 as far as I know). I don't think this is something we can tell you without measurements.
The same is true if you want to push data from the server to the client. If you do it frequently and the update must be almost immediate, then websocket is probably a better choice than polling. If you do the rarely and the delay (polling frequency) can be a few minutes, then polling might be better.
Note that I am not an expert of load scaling, this is just a layman's logic.
I would use WebSockets. For small deployments and low-frequency updates basically anything works, but with WebSockets you have technology that scales better in the long term. (And no, I would not consider this premature optimization, since the choice of technology here does not mean unnecessary initial overhead.)
Shameless plug: If you're using WebSocket, you could take a look at Crossbar.io - http://crossbar.io, or WAMP (http://wamp-proto.org) in general, which provides messaging mechanisms on top of WebSocket and should work well for you use case. I work for the company which is at the core of this, but it's open source software.
TL:DR; Are there any drawbacks / pitfalls to use a RESTful API instead of a TCP/Socket connection for a master-slave pattern?
I'm writing a web application that fetches data from an 3rd party API, calculates statistics (and more), and presents it. The fetched data is stored in a database.
I'm writing this web application with NodeJS and AngularJS.
So there are 2 basic opertions the app will do:
Answer HTTP requests from the front end
Update the database. This includes: Fetching data, saving data to database, and caching results of some heavy queries every 5 minutes in a in-memory database like redis.
I want to seperate these operations in two applications. The HTTP server shall be the master. The second application is a just a slave, of which as many instances can be spawned.
The master will implement some kind of task-processor which just distributes tasks to idle slaves. The Slave is very dumb. It can report about its status (idle/busy and some details like current load etc). You can start tasks on a slave. The master will take care of queueing tasks and so on.
I guess this is common server/agent pattern.
So I've started implenting a TCP Server and Client, but it just feels like so much overhead to do this. A bit like reinventing the wheel. So I thought I just could use a HTTP server on my client which just has two endpoints like
[GET] /status
[POST] /execute/:task
Am I on the right track here?
TL;DR; There are drawbacks to rolling your own REST API for master-slave architecture.
Your server/agent pattern is commonly referred to as microservices.
Rolling your own REST API might work, but is probably not optimal. Dealing with delivery semantics (for example, at most once vs at least once), transient failures, polling, etc will likely cause a lot of pain before you get it right.
There are libraries/services to provide varying levels of convenience:
Seneca - http://senecajs.org
Pigato - http://prdn.github.io/pigato/
Kong by Mashape - http://getkong.org
Webtask by Auth0 (paid) - https://webtask.io
As a hypothetical example, let's say that I wanted to make an application that displays peoples twitter networks. I would provide an API that would allow a client to query on a single username. That user's top x tweets would be sent to the client. Then, each person that had been mentioned by the initial person would be scanned. Their top x tweets would be sent to the client. This process would recursively continue, breadth-first, until a pre-defined depth was reached. The client would be receiving the data in real time, displaying statistics such as number of users scanned, number of known users remaining to scan, and a growing list of the tweet data. None of the processing is complicated (regex of small amounts of text), but many, many network requests would be spawned from a single initial request.
I really want the fantastic realtime capabilities of node.js with socket.io, but I feel like this is an abuse of those technologies - they're not meant for heavy server-side lifting. Is there a more appropriate toolset for what I am trying to accomplish, or a particular way to use these tools to that end? Milewise is doing something similar-ish, but I think that my application would consume significantly more network resources than theirs.
Thanks.
The best network transport which you can get on the web now are WebSockets which offers persistent bi-directional real-time connection between server and client. Although not every browser supports them, socket.io gives you a couple of fallback solutions which may however decrease the network performance when compared to WebSockets as stated in this article:
During making connection with WebSocket, client and server exchange
data per frame which is 2 bytes each, compared to 8 kilo bytes of http
header when you do continuous polling.
...
Reducing kilobytes of data
to 2 bytes…and reducing latency from 150ms to 50ms is far more than
marginal. In fact, these two factors alone are enough to make
WebSocket seriously interesting to Google.
Apart from network transport, other things may also be important, for example how are you fetching, formating and processing the data on the server side. In node.js heavy CPU bound computations may block processing of other asynchronous operations, therefore these kind of operations should be dispatched to separate threads or processes in order to prevent blocking.