I am considering developing a web site which has many characteristics of a social networking site. The website, I am considering will have a lot of apps, which will interact with the database, and also, scraping other websites for information and a multiuser chat. Also, it will feature a forum, blog, and other similar CRUD applications. The key things I am looking at is
Response time
Max number of developers may be 1 to 3 during the initial stages
I expect the website to scale up to around 1000 concurrent users in a year, and then hopefully an exponential growth.
The users are expected to spend a lot of time, in the site.
With this requirements in mind, I looked at Django, and Web2Py, since I am knowledgable in Python. It fits the bill mostly, but, I am concerned about the scalability, and as it scales, I will require more servers to be added. This means, additional cost, and I don't have any ideas to monetize the app in the near future for various reasons. So, I have to be satisfied with a limited amount of resources.
Can you kindly advice me?
Thx
Ik
From what you had described, Node.js is perfect. Not only does it have a low memory footprint and can it handle thousands of concurrent clients out of the box, but you can definitely use it for scraping websites (see this and this), creating chats (check nodechat and this other nice tutorial)
The respond time depends on your application, but if you code the right way (don't block the event loop of Node.js, keep you 'heavy-lifting' outside the server process) Node.js is really fast.
This depends on you, but consider Node.js is JavaScript on the server-side, so there is already a great pool of developers that already know JS and could learn Node.js specific things fast.
There were some official benchmarks on the nodejs blog some weeks ago, look here: http://blog.nodejs.org/2011/11/05/node-v0-6-0/ A simple server with Node.js can handle 5-6 thousands of requests per second, so you can imagine that's really something.
Spending a lot of time on the site means that they'll be making many requests, so look at my point above 3).
http://highscalability.com/blog/2011/2/22/is-nodejs-becoming-a-part-of-the-stack-simplegeo-says-yes.html
Scaling node.js
Related
Can anyone help me to decide which technology(s) to use in a work project?
I need help with some technical issues related to the job I joined 2 weeks ago and would like to know if someone with more experience could help me.
The help I need is to make one decision about which technology(es) I should use in to continue the development of an app that connects patients and doctors in a specific area of medicine, having a dashboard with chat (persistent), calendar/schedule, financial and other things less impactful.
The front-end is "practically" ready with NextJS 12 and (as I understand it) has a basic "pre-implementation" of session management with cookies and Firebase to Google and Facebook Auth) and the main question are about:
whether it will be better and/or necessary to make a separate back end (outside of NextJS's "serverless" API concept) fully or partially (in this case, which parts will be separated?),
Reason of this question: it's my first job as a Node JR backend developer and the company that hired me suffered from an "exodus" of 80% of employees due to problems in negotiations about salary and workload, so I was "kind of in the lead" of the project over the which I have doubts.
Thanks in advance!
I've searched a lot for several hours about NextJS serverless concepts in many cases, but I have no time to test implementations and the app should have a chat, store documents and payments.
I confess that I am inclined to prefer to make a separate backend, in view of the more well-known traditional advantages, but I would still like to have a little more understanding (especially about the more punctual differences between custom server and Next in this case) to defend the decision that is taken.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
I am programming since quite some years now.
Until now I was mainly focused on writing "normal" applications which run inside a console or with a GUI, sometimes also applications which interact with hardware components such as sensors / actors / ...
During this time I got to know a lot of cool programming principles and tools such as object orientation, modularization, unit-testing, test-driven-development, desing-patterns, code-analysis, ..
Also I have some first experience with hosting a wordpress blog, running static web-sites on a nginx webserver, and writing some small php-forms. But I feel like there is still too much magic in all these web-development topics. And I would like to fill this gap and learn a bit more about all these connected scripting / programming languages and technologies. (Because I hate, when I don't understand how things are working :D :D )
I started with some online "Web-development bootcamp" course at udemy to get a rough overview. This took quite some days now and I think HTML, CSS and Javascript for DOM manipulation / animations are clear to me now. Also I heard a lot about NodeJS and all it's derivate languages and databases like Mongo-DB. But still I feel like there is a lot of things unclear to me.
To get to a better understanding I wanted to development some small web-application. Nothing very special, just some website where you have to login to, are able to generate some data and this data is then persisted into a database and once you login again you are able to see the data again.
I first started with developing some classes in Javascript to represent the data in the browser while you are logged in. But I very soon realized that the Javascript which can run inside the browser is very limited and already for unit-testing and modularization into separate files that include each other I actually needed to do some crazy work-arounds or use other server-side languages like nodejs / php / ... .
After some time coding I decided to take one step back, trying to understand the basic design patterns of web-applications and not running for a long time into the wrong direction.
My questions are:
Is there some typical way to go / best practice while developing web-applications?
What are the typical key players? I know there is the difference between front-end, back-end and databases.
But are there some do's and don't's that good WebDev's follow?
For example:
which code is usually written in back-end / server-side languages?
What is usually done in the front-end? (Only desing and animations?)
Do I have to move all business logic into the back-end, also for security reasons or is this maybe also a bad idea because of peformance reasons?
What programming languages are more or less dead and not to be used in the future?
What things are typically reused from frameworks, for example authentication and session handling?
Also I felt like some things I know from other programming languages are not so easy in languages like javascript / nodejs. I am willing to spend time and effort into learning all these things but I would also like to keep the quality standards that I know from C++ /
Python. On the other side I also wondered if these patterns that I have in my head are maybe just boundaries that are completely useless in modern web-development? (e.g. typing, object orientation, modularization / splitting the code to be very reusable )
What do you think am I on the wrong track here, or do I maybe simply use the wrong languages?
I hope the long text is not knocking everyone down / keeping everyone from answering me :o
I would really appreciate your help and guidance to understand everything a bit better and to not repeat the things already a lot of others have done wrong ;)
BR, mezorian
First off, most of the questions are very opionated (at least the answers are) and your question will probably be closed for that reason. So I will post my answer before completing it and expand on it after.
First off a good roadmap to become a web developer. I like it mainly because it shows the crazyness the web development world has come to (don't be shocked!): https://levelup.gitconnected.com/the-2020-web-developer-roadmap-76503ddfb327
Trying to answer some of your question (answers are my opinion):
Is there some typical way to go / best practice while developing web-applications?
I'm tempted to say there are as many ways to do web development as there are web-developers in the world, but that might be a bit exaggerated. If you want some guidelines, I'd pick one of the major web frameworks and learn the way they do web development. With web frameworks I mean all kinds of frameworks starting with JS-frameworks all the way to static site generators, etc. They all have their ecosystem and their own rules.
What are the typical key players? I know there is the difference between front-end, back-end and databases.
(personal opinion) I work with Go in the backend. I love it because it brings back some simplicity in the crazy world of choices being a web developer. Since you know C, Go will probably be easy for you. It has static typing, structs, etc, but no need to manually manage your memory. It is also much faster than most other backend languages used in web development (Python, NodeJS, PHP, Ruby, etc).
In the front-end I have used native JS, jQuery, React, Vue, etc. I'm still waiting for something that makes things easy again. Flutter seems to be something that has a good approach, but is not really a web front framework (yet). (Don't do public websites with Flutter! They are not indexable.) We'll see where it goes.
Databases I will not go into here as that is another huge topic. Let's just say that I'm more a fan of using multiple databases for their specific strengths rather than a big one that is supposed to be good at anything.
which code is usually written in back-end / server-side languages?
Even this depends largely on your choices (framework and preference). One thing for sure has to be in the backend and that is security related stuff. Anything you put in frontend code is visible to an experienced user.
Apart from that there are some ecosystems where you don't write any backend code but talk to a (cloud) service that is basically like a database with a web endpoint on top with secured login. (for example https://firebase.google.com/.) Here the security related stuff is baked into the service.
If you do both, keeping business logic in the backend is probably a good idea. If the frontend calculates something (for quick response), the backend should double check that (e.g. calculating the total in a cart). But this is too general. There can always be use cases where some business logic needs to be implemented in the front-end.
Do I have to move all business logic into the back-end, also for security reasons or is this maybe also a bad idea because of performance reasons?
Performance can be a problem, but mostly because the roundtrip time to the server and back. If you do that for every tiny information, the UI will become sluggish. You might want to think about doing e.g. a calculation client-side.
JS-Frameworks like React, Vue don't request html from the backend, but data and build the html based on that data client-side. I'd use them if I have a very data driven website / webapp, especially if it is user-dependent. Transferring only the data and building the html for every site from it in the browser based on user settings and data, saves a lot of roundtrips.
If you are worried about server performance: For the server to hit its limit, you'd need heavy usage of your website for that to become an issue (at least with Go). If you get there you can still use horizontal scaling (multiple instances of you server) to solve that. Unless you are working for a large company with millions of users daily, I'd not worry about scaling for now.
What programming languages are more or less dead and not to be used in the future?
Warning: Very opionated!
I'd say PHP is dead. Many headhunters I've spoken with agree with me. Companies are desperately looking for PHP developers, because many developers are moving on from PHP to something "cooler". You'll definitely find a job with PHP, but might not be so happy with your job. For me it is also a sign of how modern a company really is (if PHP is not it's main backend language (any more)).
Python currently has a big boom. Mostly because of AI development. I'm not sure if that boom is also in the web development, but I'd say not. I used Python before Go (5+ years ago) and before that PHP (8+ years ago). I rarely get Python web developer job offers (at least compared to PHP and Go).
Go is the language of the cloud. It is perfect for concurrent programming which is an essential part in web development (every http call should be handled concurrently). It is fast and light weight and doesn't need anything installed on the server to run (compiles to a single binary without dependencies).
NodeJS: Haven't used. I'm not a fan of Javascript (but it was (and kind of still is) the only option in the browser), so I never liked the idea of using it also in the backend.
TypeScript: might be an alternative to JavaScript (thinking of frontend here) if you like a more structured language.
It sounded like you want to build a user baser web app with data being managed by each user. This is what I would (probably) do in that case:
Backend in Go
Go serves static files (start html, css, js, images, etc.)
Go server has an api endpoint that serves data (e.g. REST style)
Vue (or React) in the frontend
Vue requests data from the api to build the user-specific content
I've been looking around for a Node.js hosting service that suits my (probably rather exceptional) needs. It's basically a web app for CMS editors to preview CMS pages. (The actual website is statically hosted.)
So it handles only few requests, but all page requests (*.html) trigger quite a series of actions, to simplify let's say it rebuilds a good part of the website.
What I need is a service that delivers high performance on few occasions. Also it should support continuous deployment, so that when we update something, the app stays always on. (also the simpler the better: I'm a frontend developer, not a dev ops).
I've tried Google Cloud: painfully slow update mechanism, rather complex, but stable and fast, but only if you pay a lot.
Heroku is very simple, but their plans are for standard web apps, focusing on many requests, high bandwidth etc. Still, the $250 plan is rather ok in terms of performance. But again: pricey.
Jelastic would offer this flexible vertical scaling. But it's hard to do Continuous deployment and I have not yet figured out how to update an app without interruption of service.
I also though about renting a virtual private server, but again I would not know how to provide continuous delivery. Also, I'd rather have a dedicated service.
It feels like there must be a simple service that I've just missed. I'm grateful for any help or hints!
I have finally found an answer to my search: Google Cloud Run
You'll only pay by usage, and it still offers very good performance. For my scenario it decreases cost by a crazy factor: we'll pay less than 1% of the previous solution (driven by Google App Engine).
I've also tried AWS Lambda, but I had many issues with the rather extensive node app (like dynamic requires).
I'm currently working on an api for an app that I'm developing, which I don't want to tell too much about. I'm a solo dev with no patents so it's probably a good idea to keep it anonymous. It uses socket.io together with node.js to interact with clients and the other way around, which I might be swapping out sometime later for elixir and it's sockets, but that isn't relevant for now. Now I'm trying to look into cloud hosting, but I'm having a rough time finding a good service to use.
These are my requirements:
24/7 uptime
Low memory and performance necessary (at least to start with). About 1+ gig with 2+ cores will most likely suffice (need 2 threads or more for node to handle async programming well)
Preferably free for like maybe even a year, or just really cheap, but that might be munch to ask
Must somehow be able to run some sort of database. Haven't really settled on this yet, but I want to implement a custom currency at some point, and probably have the ability to add some cooldowns. So it can be fairly simple and small. If anybody has any tips on what database I should use, that would also be very welcome. I was thinking of Cassandra because of the blazing fast performance and expandability. But I also wanna look into remote databases, especially if I'm gonna go international with the product
Ability to keep open socket.io connections, as you've probably guessed :P
Low ping decently high bandwith internet. The socket.io connections are lightweight and not a lot of data has to be sent. Mostly packets of a few kilobytes every now and then for all of the clients.
If this information is too vague or you want to know some other requirements I haven't thought of, let me know.
Check out Heroku (PaaS), they have a free version to start with
Say I have a bunch of webservers each serving 100's of requests/s, and I want to see real time stats like:
Request rate over last 5s, 60s, 5 min etc
Number of unique users seen again per time window
Or in general for a bunch of timestamped events, I want to see real-time derived statistics - what's the best way to go about it?
I've considered having each GET request update a global counter somewhere, then sampling that at various intervals, but at the event rates I'm seeing it's hard to get a distributed counter that's fast enough.
Any ideas welcome!
Added: Servers are Linux running Apache/mod_wsgi, with a Python (Django) stack.
Added: To give a sense of the event rates I want to track stats for, they're coming in at over 10K events/s. Even incrementing a distributed counter at that rate is a challenge.
You might like to help us try out the beta of our agent for application performance monitoring in Python web applications.
http://newrelic.com
It delves more into the application performance rather than just the web server, but since any bottlenecks aren't generate going to be the web server, but your application then that is going to be more useful anyway.
Disclaimer. I work for New Relic and this is the project I am working on. It is a paid product, but the beta means it is free for now with all features. Later when that changes, if you didn't want to pay for it, their is still a Lite subscription level which is free and which gives you basic web metrics reporting which still covers some of what you are after. Anyway, right now would be a great opportunity to make use of it to debug your performance while you can.
Virtually all good servers provide this kind of functionality out of the box. For example, Apache has the mod_status module and Glassfish supports JMX. Furthermore, there are many commercial packages for monitoring clusters, such as Hyperic and Zenoss.
What web or application server are you using? It is difficult to provide a solution without that information.
Look at using WebSockets, their overhead is much smaller than a HTTP request, they are very well suited to real-time web applications. See: http://nodeknockout.com/ for Node based websocket examples.
http://en.wikipedia.org/wiki/WebSocket
You will need to run a daemon if you want to run it on your apache server.
Also take a look at:
http://kaazing.com/ if you wan't less hassle, but are willing to fork out some cash.
On the Windows side, Perfmonance monitor is the tool you should investigate.
As Jared O'Connor said, you should precise what kind of web server you want to monitor.