Let me start with a brief explanation of this question. I've been recently getting started into web development. I know the bits and bytes of modern MVC architectures and have already been using it. But now that I am planning to build a Dashboard, I am confused about how to do it.
The traditional flow of data is, it is loaded from database at the backend, processed a little and sent in a response to the frontend. This is what I have worked on till now.
But talking about dashboards, assume that the data that should be shown on the dashboard requires too much processing. Like suppose the dashboard shows how much the person has spent/earned till now.
Now, adding some background context. If the application was about showing how much the person has spent at an online shopping site till now, it might not need too much processing. Loading a few hundreds of records might be okay (maybe!).
But if the dashboard is about showing how much an intraday trader has spent/earned till now (since he/she joined) would be too heavy. In this case, loading tens of thousands of records and calculating (adding/subtracting) will result in load on the server.
In this case, I am thinking about having a separate table for reports which is updated as and when necessary (for ex: when a new trade is executed by the user). And then simply reading and sending the data to frontend. Also, a bit of processing maybe done if needed, but all the heavy work is already done.
These two examples can be classified into two categories:
Processing in real-time
Processing ahead of time
Now, the question I have here is, "Should I always use the 2nd approach?" I have this question because if the processing is too less, then creating separate tables for reports is just a waste of storage. So, there are trade-offs that I can think of.
Now, coming to the second part of this question. This part refers to how should I be handling data changes at frontend. Three possible ways are:
Periodic changes (okay if the data updates are less frequent)
Instant / Real-Time changes (using something like data streaming maybe?)
Instant / Real-Time changes, but only sending the updates (new changes that took place) and not whole data to frontend
Now, coming to what I think about this part. If the data changes are less frequent i.e. once in a day, it's okay to go for the first approach. But I am still unaware about the 2nd and 3rd one. Also, should I use data streaming and can anyone provide some insights of how streaming differs from a normal request-response cycle? I think this might be related to websockets or something similar but I am unaware.
PS: I am currently using Django as my backend, and am planning to use React/Vue or similar framework as my frontend. (The answer will be independent of this, but just for information)
With modern front-end technologies, it doesn't really matter if you use pulling data from the server (periodic update) or pushing data to the front-end (instant/real-time). For angular and React you should look at Observables, using this on the front-end both options are treated exactly the same.
In an event-driven front-end, it doesn't matter how you get the data you will write your program to handle the event of "when my data arrives, I will do XYZ"
Generally speaking, it would be better to break your next question down in to 1 or 2 very specific questions. Instead of asking a very broad question with tags "front-end", "back-end" and "everything-in-the middle", you could ask several specific questions focused on each aspect.
Related
I am making a stock market simulator website for an event. It will be an event where about 200-300 people will participate as investors.
There will be 3 databases, one for all available stock (Industry, Company, Price etc.), one for all investors (Name, Stocks Owned etc.) and one to store all trades that happened. It also needs to run fast as transaction related calculations need to happen quickly to avoid any conflict with other investors.
I have the website written in React, but I'm not sure about what to use for the backend. I was going to use a python framework with firebase, but I think this would be too slow as it is an app that needs to be working live. Also, any pointer towards a good architecture to use would be nice too! I was thinking a basic microservices architecture (without actual RestAPI's but just file calls as this increases hosting cost), but I'm not sure if this would be the best for my use case.
Any help is appreciated!
We are currently in the process of organising a student conference.
The issue is that we offer several different events at the same time over the course of a week. The conference runs the whole day.
It's currently been operating on a first come, first served basis, however this has led to dramatic problems in the past, namely the server crashing almost immediately, as 1000+ students all try to get the best events as quickly as they can.
Is anyone aware of a way to best handle this so that each user has the best chance of enrolling in the events they wish to attend, firstly without the server crashing and secondly with people registering for events which have a maximum capacity, all within a few minutes? Perhaps somehow staggering the registration process or something similar?
I'm aware this is a very broad question, however I'm not sure where to look to when trying to solve this problem...
Broad questions have equally broad answers. There are broadly two ways to handle it
Write more performant code so that a single server can handle the load.
Optimize backend code; cache data; minimize DB queries; optimize DB queries; optimize third party calls; consider storing intermediate things in memory; make judicious use of transactions trading off consistency with performance if possible; partition DB.
Horizontally scale - deploy multiple servers. Put a load balancer in front of your multiple front end servers. Horizontally scale DB by introducing multiple read slaves.
There are no quick fixes. It all starts with analysis first - which parts of your code are taking most time and most resources and then systematically attacking them.
Some quick fixes are possible; e.g. search results may be cached. The cache might be stale; so there would be situations where the search page shows that there are seats available, when in reality the event is full. You handle such cases on registration time. For caching web pages use a caching proxy.
I am developing an app that uses long-polling for instant notifications about changes made to base.
App structure is as follows:
there is a questions base and admins are answering them in real time. I have a actions table where actions are stored with timestamps. So every admins browser is polling method that queries database for those actions and if there appears a new action from last check it issues a question update with ajax.
I came to 2 questions i cant find an answer to so i came here.
I have multiple clients that are polling one database. With help of long-polling technique i was able to reduce the latency of process but the server load is still same since its asking database for changes every 2 seconds. Is there a way to reduce amount of database queries somehow because at the moment it seems wrong that lets say 10 clients are asking essentially same information each every 2 seconds. maybe there is technique to reduce server load.
Whether there is a solution to first question or not either way i need to somehow calculate server needs for this kind of app as i might need to increase amount of administrators using it. How could i do that.
I am considering developing a web site which has many characteristics of a social networking site. The website, I am considering will have a lot of apps, which will interact with the database, and also, scraping other websites for information and a multiuser chat. Also, it will feature a forum, blog, and other similar CRUD applications. The key things I am looking at is
Response time
Max number of developers may be 1 to 3 during the initial stages
I expect the website to scale up to around 1000 concurrent users in a year, and then hopefully an exponential growth.
The users are expected to spend a lot of time, in the site.
With this requirements in mind, I looked at Django, and Web2Py, since I am knowledgable in Python. It fits the bill mostly, but, I am concerned about the scalability, and as it scales, I will require more servers to be added. This means, additional cost, and I don't have any ideas to monetize the app in the near future for various reasons. So, I have to be satisfied with a limited amount of resources.
Can you kindly advice me?
Thx
Ik
From what you had described, Node.js is perfect. Not only does it have a low memory footprint and can it handle thousands of concurrent clients out of the box, but you can definitely use it for scraping websites (see this and this), creating chats (check nodechat and this other nice tutorial)
The respond time depends on your application, but if you code the right way (don't block the event loop of Node.js, keep you 'heavy-lifting' outside the server process) Node.js is really fast.
This depends on you, but consider Node.js is JavaScript on the server-side, so there is already a great pool of developers that already know JS and could learn Node.js specific things fast.
There were some official benchmarks on the nodejs blog some weeks ago, look here: http://blog.nodejs.org/2011/11/05/node-v0-6-0/ A simple server with Node.js can handle 5-6 thousands of requests per second, so you can imagine that's really something.
Spending a lot of time on the site means that they'll be making many requests, so look at my point above 3).
http://highscalability.com/blog/2011/2/22/is-nodejs-becoming-a-part-of-the-stack-simplegeo-says-yes.html
Scaling node.js
I wonder how high traffic websites handle traffic logging, for example a website like myspace.com receives a lot of hits, I can imagine it would take a lot of space to log all those requests, so, do they log every single request or how do they handle this?
If you view source on a MySpace page, you get the answer:
<script type="text/javascript">
var pageTracker = _gat._getTracker("UA-6293770-1");
pageTracker._setDomainName(".myspace.com");
pageTracker._setSampleRate("1"); //sets sampling rate to 1 percent
pageTracker._trackPageview();
</script>
That script means they're using Google Analytics.
They can't just gauge traffic using IIS logs because they may sell ads to third parties, and third parties won't take your word for how much traffic you get. They want independent numbers from a separate company, and that's where Google Analytics comes in.
Just for future reference - whenever you've got a question about how a web site is doing something, try viewing the source. You'd be amazed at what you can find there in plain view.
We had a similar issue with out Intranet which is used by hundreds of people. The disk activity was huge and performance was being hurt.
The short answer is Asynchronous non-blocking logging.
probably like google analytics.
Use Javascript to load a page on a difference server, etc.
Don't how they track it since I don't work there. I am pretty sure that they have enough storage to record every little thing about their user if they wanted.
If I were them, I would use AwStats if I just wanted to know basic stuff about my users.
It is more likely that they have developed their own scripts for tracking their users. Stuff they would log
-ip_address
-referrer
-time
-browser
-OS
and so on. Then a script to see different data about the user varying by day, weeks, or months. As brulak said, something along the line of Analytics, but since they have access to actual database, they can learn much more about their users.
ZXTM traffic shaping and logging, speaking from experience here
I'd be extremely surprised if they didn't log every single request, yes, and operations with particularly high traffic volumes usually roll their own log-management solutions against the raw server logs, in some form or other -- sometimes as simple batch-type processes, sometimes as complete subsystems.
One company I worked for, back in the dot-com heyday, got upwards of twenty million pageviews a day; for that site (actually a set of them, running across a few dozen machines in all, as I recall), our ops team wrote a quite sophisticated, clustered solution in C that parsed, translated (into relational storage), compressed and distributed the logs daily. Log files, especially verbose ones, pile up fast, and the commercial solutions available at the time just couldn't cut it.
If by logging you mean for collecting server related information (request and response times, db and cpu usage per request etc) I think they sample only the 10% or 1% of the traffic. That gives the same results (provide developers with auditing information) without filling in the disks or slowing the site down.