I know the following question might be a little bit vague but still I need some answers about it.
I'm about to implement a JSON based WebService that will feed information to a couple of websites and mobile Apps.
Since most of time we're doing CPU intensive MySQL queries and need to cache data, PHP is no longer enough and we decided to move to Node.js.
How can one plan, manage, implement and maintain complex business logic with Node.js since we're talking about fully async programming and not something run-time oriented like PHP, Java etc...?
I've tried to put something very simple together with Node.js and after some minutes all my code starts be full of nasty callbacks, overkill dependency injection and it becomes a mess. How should I convert the typical synchronous workflow into something Node friendly?
I've noticed most of the Node.js community is looking at premisses as a way to solve this issues, but still I don't see it is enough, its still a mess...
Any tips?
Managing complex business logic with node is possible. I can give you some tips :
Code organization point of view
Think async
Easy to say, but when I began node, comming from C programming, it wasn't easy at all.
If you have something like :
read(function(){
//some stuff
write(function(){
//some other stuff
[etc...]
}
}
Your code should probably be :
var onRead = function(){
//some stuff
write(onWrite);
}
var onWrite = function(){
//some other stuff
}
read(onRead);
or even better : Make a onRead module ? wich lead me to second advice :
Make modules
A lot of them. And load them on start.
A 50 line module can't really be that complicated to read, can it?
It takes some time to get accustomed with this, but you'll feel at home really quickly.
Use modules designed to remove "nested callback hells"
async is one of them. There are many out there.
I don't really use those. Often it's just a workaround for bad design, but they can be really useful in some situations.
Software architecture point of view
Doesn't change as much as you can expect. Mostly you'll want something similar to what you'd do on a stateless php app, apply above tips and it should be OK.
If you want more precise advice, maybe you should paste a code sample which is representative of your problems to let us explain how you can improve it.
Related
I became infected with functional programming and reactive approach. For inspiration and ideas, I use Haskell and an awesome Rick Hickey article. In the python world I found for myself RxPy and funcy libraries. Now I have thousands of lines of imperative code, which I want to make functional. If everything is simplified, I have an interface of getters and setters for the database and a kernel that works like a state machine. Here's how it looks on pseudo code:
class State(object):
def __init__(id):
self.data = database_interface.get_state(id)
self.status = data['status']
def process(self):
if self.status == 'init':
self.handle_init()
elif self.status == 'request_data':
self.handle_request_data()
elif self.status == 'idle':
self.handle_idle()
# etc...
...
def on_new_message(msg):
id = msg['id']
state = State(id)
state.process()
I have a lot of if and for imperative business logic in my state handlers. And I'm really embarrassed how to move from the current model to reactive and functional. Here everything is very simplistic, but who already had a similar experience will understand me. I need advice on where to move next, from ideas to practice, larger than simple utilities or trivial REST api in a functional style. Also, I will be very helped by links to the source codes of really large projects, wherever I could get ideas. Thanks to everyone who responds, having a real experience porting the imperative code to the functional one. Yes, I understand that it will not be porting the code, but rewriting it from scratch. Again, I need examples of projects with a large number of business logic, where there is work with data and mutations of data. Anyway, thank you.
STOP. You have thousands of lines of code in production. It works. It may be a big ball of mud, but it works. I'm a big fan of FP, and write functionalish Python and very functional JavaScript for work myself, but that strikes me as backwards-thinking.
But if you can't resist the siren song of the shiny new thing, start one module at a time. Architecture such that you can't do that? That's a way bigger problem. Refactor to fix that first. Then go through the modules one at a time. If it can be made pure, make it pure. If you are having a hard time making modules pure, refactor them so that you that you have some impure modules and a bunch of pure ones instead of all mixed modules. Note that outwardly pure modules may use side-effects internally, as long as they don't leak beyond the module scope and the caller can't tell the difference that's fine. For now.
Now that you've done that you can change the internal implementation of those modules piecemeal without affecting the rest of the program. Try to replace custom classes with generic data structures where possible, especially at module boundaries. Note that doing so can have a negative impact on perf: profile!
Note that a lot of these tips overlap with what many would consider best practices for defining good object oriented architecture as well (program to interfaces, don't let implementation details leak, KISS, etc.). That is not a coincidence.
Here's another good talk for inspiration. Here's a link to clean architecture of Uncle Bob. Although geared towards UI programming its relevant.
DON'T go read/listen and say "I've got to implement the hexagonal ports-and-adapters pattern with a message queue for IPC or we're doomed!". Again, you have working code. Change it carefully, ideally a bit here and there, respecting what that means (especially to your users).
If all of this seems rather harsh, it's because I've been down this road, and trust me, it hurts. If you don't want to see your new-found enthusiasm vanish in a cloud of bug-reports then tread slowly and softly. I kinda like node.js but have shot myself in the foot by forever staining its reputation at my employer by doing the kind of thing you're talking about doing. Take heed, spare yourself some pain simply by slowing down.
Before you start developing something useful in Node.js, what's your process? Do you create tests on VowJS, Expresso? Do you use Selenium tests? When?
I'm interested in gaining a nice workflow to develop all my node.js applications similar to Rails (Cucumber, Rspec, Code).
Sorry for the amount of questions.
Let me know how it works out with you.
The first thing I do is to write some documentation or do some wireframes. It helps to visualize what do I want to implement.
Then I code the interface/skeleton of my module/application, without implementations.
Then I add specs and tests using testosterone (although vows and expresso are more popular options) and I make them pass by implementing them.
If you find that a private method needs to be tested (it deals with I/O, has complex logic ...) move it to a another class and test it independently.
Stub your I/O calls as much as you can. Tests will run faster and you will not have to deal with side effects. I recommend gently.
My testing methodology isn't up the snuff as in for example Java/Junit and I should really work more on this(improve). I should really practice TDD more.
I played a little bit with expresso and liked to the fact that you could generate code coverage reports. What I thought was missing was something like #before #beforeclass #after which you can find in java.
I also played a bit with nodeunit which does have setup/teardown. I still like to play a little bit more with this framework.
I don't like the vowjs syntax, but it is very popular BDD framework, so maybe I should use it (more) to get sold like a lot of other users. But for now I am going to dismiss vowjs.
I also played with zombie.js a litle bit which is also pretty cool. I also lately saw another cool testing framework which I can't remember the name, but there are luckily enough options to do testing in node.js.
The only thing I don't like is that the integration with IDE is not up to snuff in my opinion. The IDE I had for Java cannot be compared with what I have found for node.js, but I think with a little bit effort I can make a more useful programming environment. I will try and keep you guys informed about this progress.
P.S: But what I do like a lot is the npm package manager. When you compare it to for example maven you just say wow. I still has some minor bugs because it is still a young project. But still npm is very good in my opinion!
I recently inherited a large codebase and am having to read it. The thing is, I've usually been the dev starting a project. As a result, I don't have a lot of experience reading code.
My reaction to having to read a lot of code is, well, umm to rewrite it. But I need to bring myself up to speed quickly and build on top of an existing system.
Do other people have techniques they've learned to absorb a code base? At this point, I'm just reading through the code. I've tried generating UML diagrams using UModel. They're so big they won't print cleanly and when I zoom in, I really do lose the perspective of seeing all the relationships.
How have other people dealt with this problem?
Wow - I literally just finished listening to a podcast on reading code!!!
http://www.pluralsight-training.net/community/blogs/pluralcast/archive/2010/03/01/pluralcast-10-reading-code-with-alan-stevens.aspx?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+pluralcast+%28Pluralcast+by+Pluralsight%29
I would recommend listening to this. One interesting point that was made that I found radical and may be something you could try (I know I'm going to!). Download the entire source code base. Start editing and refactoring the code then...throw that version away!!! I think with all the demands that we have with deadlines that doing this would not even occur to most developers.
I am in a similar position to you in my own work and I have found the following has worked for me:
- Write test cases on existing code. To be able to write the test case you need to be able to understand the cde base.
- If it is available, look at the bug\issues that have been documented through the life cycle of the product and see how they were resolved.
- Try and refactor some of the code - you'll probably break it, but that's fine you can throw it away and start again. By decomposing the code into smaller problems you'll understand it bettter
You don't need to make drastic changes when refactoring though. When your reading the code and you understand something, rename the variable or the method names so the better reflect the problem the are trying to solve.
Oh and if you can, please get a copy of Working Effectively with Legacy Code by Michael C. Feathers - I think you'll find it invaluable in your situation.
Good luck!
This article provides a guideline
Visualization: a visual representation of the application's design.
Design Violations: an understanding of the health of the object
model.
Style Violations: an understanding of the state the code is currently
in.
Business Logic Review: the ability to test the existing source.
Performance Review: where are the bottlenecks in the source code?
Documentation: does the code have adequate documentation for people
to understand what they're working on?
In general I start at the entry point of the code (main function, plugin hook, etc) and work through the basic execution flow. If its a decent code base it should be broken up into decent size chunks, and you can then go through and figure out what each chunk of the code is responsible for. Looking back at when in the execution flow of the system its called.
For the package/module/class exploration I use whatever doxygen will generate once its been run over the sources. It generates some nice class relation diagrams, inheritance hierarchies and file dependencies graphs. The benefit of these is they are each focused on a single class and how it ties it with its neighbors, siblings and parents, so the graphs are usually of manageable size and easy to understand.
As you understand what different, classes, functions and sub-systems do I like to add comments to fill what sounds like obviously missing documentation. This helps you when you re-read through the code the second time.
I would recommend another podcast and resources:
SE-Radion episode on Software Archeology
Is there a workaround for implementing cross cutting concerns without going into aspects and point cuts et al.?
We're in Spring MVC, and working on a business app where it's not feasible to go into AspectJ or Spring's aspect handling due to various reasons.
And some of our controllers have become heavily bloated (too heavily), with tons of out-of-focus code creeping in everywhere.
Everytime I sit down to refactor, I see the same things being done over and over again. Allow me to explain:
Everytime I have to prepare a view, I add a list of countries to it for the UI. (Object added to the ModelAndView). That list is pulled out of a DB into ehCache.
Now, initially it was terrible when I was trying to add the lists INLINE to the mav's everywhere. Instead, I prepared a function which would process every ModelAndView. How? well, with more garbage calls to the function!
And I bought out one trouble for another.
What's a design pattern/trick which can help me out a bit? I'm sick of calling functions to add things to my ModelAndView, and with over 3500 lines of only controller code, I'm going mad finding all the glue points where things have gone missing!
Suggestion are welcome. Cross cutting concerns flavor without AspectJ or Spring native.
Since you are using Java, you may consider moving your code to Scala, since it interacts well with Java, then you can use traits to get the functionality you want.
Unfortunately cross-cutting is a problem with OOP, so changing to functional programming may be a solution, but, I expect that in actuality they are using AOP to implement these mixins, so it would still be AOP, just abstracted out.
The other option is to look at redesigning your application, and make certain that you don't have duplicate code, but a major refactoring is very difficult and fraught with risk.
But, for example, you may end up with your ModelAndView calling several static utility classes to get the data it needs, or do ensure that the user has the correct role, for example.
You may want to look at a book, Refactoring to Patterns (http://www.industriallogic.com/xp/refactoring/) for some ideas.
What are some ways in which I can write code that is easily modified?
The one I have learned from experience is that I almost always need to write one to throw away. That way I have developed a sense of the domain knowledge and program structure required before coding the actual application.
The general guidelines are offcourse
High cohesion, low coupling
Dont repeat yourself
Recognize design patterns and implement them
Dont recognize design patterns where they are not existing or necassary
Use a coding standard, stick to it
Comment everyting that should be commented, when in doubt : comment
Use unit tests
Write comments and tests before implementation, that way you know exactly what you want to do
And when it goes wrong : refactor, refactor, refactor. With good tests you can be sure nothing breaks
And oh yeah:
read this : http://www.pragprog.com/the-pragmatic-programmer
Everything (i think) above and more is in it
I think your emphasis on modifiability is more important than readability. It is not hard to make something easy to read, but the real test of how well it is understood comes when someone else (or you) has to modify it in repsonse to changing requirements.
What I try to do is assume that modifications will be necessary, and if it is not really clear how to do them, leave explicit directions in the code for how to do them.
I assume that I may have to do some educating of the reader of the code to get him or her to know how to modify the code properly. This requires energy on my part, and it requires energy on the part of the person reading the code.
So while I admire the idea of literate programming, that can be easily read and understood, sometimes it is more like math, where the only way to do it is for the reader to buckle down, pay close attention, re-read it a few times, and make sure they understand.
Readability helps a lot: If you do something non-obvious, or you are taking a shortcut, comment. Comments are places where you can go back and refactor if you have time later. Use sensible names for everything, makes it easier to understand what is going on.
Continuous revision will let you move from that first draft to a better one without throwing away (too much) work. Any time you rewrite from scratch you may lose lessons learned. As you code, use refactoring tools to eliminate code representing areas of exploration that are no longer needed, and to make obvious things that were obscure. The first one reduces the amount that you need to maintain; the second reduces the effort per square foot. (Sqft makes about as much sense as lines of code, really.)
Modularize appropriately and enforce encapsulation and separation of logic between your modules. You don't want too many dependencies on any one part of the code or that part becomes inherently harder to understand.
Considering using tried and true methods over cutting edge ones. You give up some functionality for predictability.
Finally, if this is code that people will be using before and after modification, you need(ed) to have an appropriate API insulating your code from theirs. Having a strong API lets you change things behind the scenes without needing to alert all your consumers. I think there's a decent article on Coding Horror about this.
Hang Your Code Out to D.R.Y.
I learned this early when assigned the task of changing the appearance of a web-interface. The code was in C, which I hated, and was compiled to a CGI executable. And, worse, it was built on a library that was abandoned—no updates, no support, and too many man-hours put into its use to change it. On top of the framework was a disorderly web of code, consisting of various form and element builders, custom string implementations, and various other arcane things (for a non-C programmer to commit suicide with).
For each change I made there were several, sometimes many, exceptions to the output HTML. Each one of these exceptions required a small change or improvement in the form builder, thanks to the language there's no inheritance and therefore only functions and structs, and instead of putting the hours in the team instead wrote these exceptions frequently.
In my inexperience I was forced to change the output of each exception, rather than consolidate the changes in an improved form builder. But, trawling through 15,000 lines of code for several hours after ineffective changes would induce code-burn, and a fogginess that took a night's sleep to cure.
Always run your code through the DRY-er.
The easiest way to modify a code is NOT to write code. Write pseudo code not just for algo but how your code should be structured if you are unsure.
Designing while writing code never works...for me :-)
Here is my current experience: I'm working (Java) with a kind of database schema that might often change (fields added/removed, data types modified). My strategy is to parse this schema and to generate the code with apache velocity. The BaseClass generated is never modified by the programmer. Else, a MyClass extends BaseClass is created and the logical components of this class (e.g. toString() ! )are implemented using the 'getters' and the 'setters' of the super class.