Related
Can anyone give me an example of successful non-programmer, 5GL (not that I am sure what they are!), visual, 0 source code or similar tools that business users or analysts can use to create applications?
I don’t believe there are and I would like to be proven wrong.
At the company that I work at, we have developed in-house MVC that we use to develop web applications. It is basically a reduced state-machine written in XML (à la Spring WebFlow) for controller and a simple template based engine for presentation. Some of the benefits:
dynamic nature: no need to recompile to see the changes
reduced “semantic load”: basically, actions in controller know only “If”. Therefore, it is easy to train someone to develop apps.
The current trend in the company (or at least at management level) is to try to produce tools for the platform that require 0 source code, are visual etc. It has a good effect on clients (or at least at management level) since:
they can be convinced that this way they will need no programmers or at least will be able to hire end-of-the-lather programmers that cost much less than typical programmers.
It appears that there is a reduced risk involved, since the tool limits the implementer or user (just don’t use the word programmer!) in what he can do, so there is a less chance that he can introduce error
It appears to simplify the whole problem since there seems to be no programming involved (notoriously complex). Since applications load dynamically, there is less complexity then typically associated with J2EE lifecycle: compile, package, deploy etc.
I am personally skeptic that something like this can be achieved. Solution we have today has a number of problems:
Implementers write JavaScript code to enrich pages (could be solved by developing widgets). Albeit client-side, still a code that can become very complex and result in some difficult bugs.
There is already a visual tool, but implementers prefer editing XML since it is quicker and easier. For comparison, I guess not many use Eclipse Spring WebFlow plug-in to edit flow XML.
There is a very poor reuse in the solution (based on copy-paste of XML). This hampers productivity and some other aspects, like fostering business knowledge.
There have been numerous performance and other issues based on incorrect use of the tools. No matter how reduced the playfield, there is always space for error.
While the platform is probably more productive than Struts, I doubt it is more productive than today’s RAD web frameworks like RoR or Grails.
Verbosity
Historically, there have been numerous failures in this direction. The idea of programs written by non-programmers is old but AFAIK never successful. At certain level, anything but the power of source code becomes irreplaceable.
Today, there is a lot of talk about DSLs, but not as something that non-programmers should write, more like something they could read.
It seems to me that the direction company is taking in this respect is a dead-end. What do you think?
EDIT: It is worth noting (and that's where some of insipiration is coming from) that many big players are experimenting in that direction. See Microsoft Popfly, Google Sites, iRise, many Mashup solutions etc.
Yes, it's a dead end. The problem is simple: no matter how simple you make the expression of a solution, you still have to analyze and understand the problem to be solved. That's about 80-90% of how (most good) programmers spend their time, and it's the part that takes the real skill and thinking. Yes, once you've decided what to do, there's some skill involved in figuring out how to do that (in a programming language of your choice). In most cases, that's a small part of the problem, and the least open to things like schedule slippage, cost overrun or outright failure.
Most serious problems in software projects occur at a much earlier stage, in the part where you're simply trying to figure out what the system should do, what users must/should/may do which things, what problems the system will (and won't) attempt to solve, and so on. Those are the hard problems, and changing the environment to expressing the solution in some way other that source code will do precisely nothing to help any of those difficult problems.
For a more complete treatise on the subject, you might want to read No Silver Bullet - Essence and Accident in Software Engineering, by Frederick Brooks (Included in the 20th Anniversary Edition of The Mythical Man-Month). The entire paper is about essentially this question: how much of the effort involved in software engineering is essential, and how much is an accidental result of the tools, environments, programming languages, etc., that we use. His conclusion was that no technology was available that gave any reasonable hope of improving productivity by as much as one order of magnitude.
Not to question the decision to use 5GLs, etc, but programming is hard.
John Skeet - Programming is Hard
Coding Horror - Programming is Hard
5GLs have been considered a dead-end for a while now.
I'm thinking of the family of products that include Ms Access, Excel, Clarion for DOS, etc. Where you can make applications with 0 source code and no programmers. Not that they are capable of AI quality operations, but they can make very usable applications.
There will always be "real" languages to do the work, but we can drag and drop the workflow.
I'm using Apple's Automator which allows users to chain together "Actions" exposed by the various applications on their systems.
Actions have inputs and/or outputs, some have UI elements and basic logic can be applied to the chain.
The key difference between automator
and other visual environments is that
the actions use existing application
code and don't require any special
installation.
More Info > http://www.macosxautomation.com/automator/
I've used it to "automate" many batch processes and had really great results (surprises me every time). I've got it running builds and backups and whenever i need to process a mess of text files it comes through.
I would love to know whether iHook or Platypus (osx wrapper builders for shell scripts) could let me develop plugins in python ....
There is definitely room for more applications like this and for more support from OSX application developers but the idea is sound.
Until there's major support there aren't many "actions" available, but a quick check on my system just showed me an extra 30 that i didn't know i had.
PS. There was another app for OS-preX called "Filter Tops" which had a much more limited set of plugins.
How about Dabble DB?
Of course, just like MS Access and other non-programmer programming platforms, it has some necessary limitations in order that the user won't get him or herself stuck... as John pointed out programming is hard. But it does give the user a lot of power, and it seems that most applications that non-programmers want to build are database-type applications anyway.
I always think document is really important for a project and a team and should be written regularly and detailedly. It can make things go in parallel without always asking the skilled programmers here and there. But truly I find many developers(even leaders) don't put so much attention to documents and just take them for granted, which makes me feel bad.
So is my attitude to documents right? Are documents are really important? How should I persuade the team leader to put more attention to the documents?
If documents are important, the second question springs up. Who should write the documents? IMO, they should be written by the skilled programmers like the framework creator(if we use our own framework), the important parts of project(like db schema, the whole architecture, etc.) and more.
The benefits are obvious, like helping fresh man, help maintaining and more.
So from my opinion, the skilled programmers(the definition here may be different) should pay more attention to the documents writing than code writing after the infrastructure is done.
Am I right about this point?
Thanks for your sharing about these questions.
You have several kinds of documentation, one of them is your responsibility:
Document each function, class, structure, member as you complete it
Ideally, you do this in a way that permits automatic extraction of source documentation (e.g. Doxygen). Just be sure to do it as you go.
As far as customer documentation goes, my beliefs are:
Every development firm should employ testers
Testers should contribute heavily to the documentation process
I've been with companies that simply will not pay in full for the final product unless it ships with complete and comprehensive documentation. 10% is usually held back just to ensure that the contractor has incentive to deliver all materials.
As far as testers go, they are really your best friend (or should be). They are the people who know how your software works almost as well as you do. And yes, I agree, you should have at least an outline of a programs functionality, this keeps you from going off on 'value add' tangents. It just makes sense to let the testers be the one to fill this in, then have the developers review it for accuracy.
You may even find your self saying "No no no .. it doesn't work that way .. the testers got this wrong ...", then you fire up the app to realize that they got it right :) In that aspect, it's also helpful to the QA process.
I think it is up to each team, but many times, programmers aren't skilled in writing documentation. That is why there are people like Technical Writers. The programmers should be involved in every step since they are the subject matter experts, but the writers should write.
Your group should create a software development process that defines how you go about developing your software products. Part of that process would define the documents to be written, and in my experience, all members of the development team share in the documentation process -- It can (and should) be a learning process.
Your software development process should define other topics as well, such as code reviews, unit testing, configuration management, etc.
There are lots of examples of processes, from very light to very heavy on the web.
The important questions to ask about any potential documentation is what is the goal, intended usage, and expected frequency of usage of the documentation? You mention helping a fresh man, but in practice, is reading the documentation faster and more efficient then getting a walkthrough from another developer? A walkthrough takes time from the other developer, but likely far less time then writing a document would.
Documentation with a strong business case and ROI over alternative options makes sense to create, but there are likely less cases of that then you imagine, and creating documentation without having a clear answer to my initial questions will guarantee you don't get the ROI for it.
You're not wrong, but fact is that documentation doesn't make money. If anything, poor documentation can increase revenue because you've ensured that clients will need your support contract.
Documentation is also a pain because in theory it's supposed to be done before development, but in reality things change so it's really only worth creating/updating after a major version release.
Ideally, the author should be the business analyst, not the developer.
Another way of looking at the documentation is for CYA purposes. If you ever have the misfortune of being in a project, where the leadership does not generate documentation, then blame for the bad code can go to you. Unless you protected your self with documentation.
The person who speaks the language of your client best is the person who should write the documentation, even if they aren't the person who best understands the product. They should confer with the person who best understands the product, but documentation is not about coding ability; it's about communication.
If you are bad at communicating, it doesn't matter how well you know your product, your documentation will be useless.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I have met a lot of people for whom Agile has worked really well, and most of them tend to be managers and architects who plan and delegate the work. However I really haven't found much good developers convinced that Agile is working for them.
Of course you can say if Agile isn't working for you, you aren't doing it right. But whatever remixes of Agile are out there, is it working for you as a Developer? And why? Does anyone else think, within a traditional (or close to) team structure, Agile feels more like a form of micromanagement than self-management?
At my first job, we had daily scrums, wrote automated tests, had automated builds, pair programmed, etc. We had been in the agile groove for several years. And for our efforts, we were rewarded with software that I wouldn't touch with 20ft pole. The quality of our product was atrocious: I'd describe as the piecemeal hacking of 100 amateur developers.
What went wrong:
The company I worked at had a notorious reputation for hiring entry-level developers for the lowest pay ($25-27K/yr was the norm), and frequently we'd outsource work to the lowest offshore bidder. I've heard that agile just doesn't work on inexperienced developers, and I think it showed through the code and our turnover rate.
No documentation of any sort. No functional documentation, no technical documentation, no requirements, no bug tracking. No one ever wrote things down on persistent media: all requirements and bugs were communicated by email, word of mouth, and psychic mindreading.
Lousy testing: our automated tests were invaluable, but QA and UAT testing was a disaster. Since we didn't have any formal requirements documentation, QA users didn't know what new functionality they were testing, so all QA consisted more or less of haphazard end-to-end testing. User acceptance testing was performed in production: we installed the product on our customers servers and reported bugs as they occurred in production.
Crisis-driven development: bugs were handled by using the "OMG WE HAVE TO FIX THIS AND REDEPLOY PRONTO! NOW NOW NOW! NO TIME FOR TESTING JUST FIX IT!" management methodology.
Although we did everything right and really adhered to agile principles by the book, the methodology failed harder than anything else I've ever seen.
In contrast, the company that I work for now uses a waterfall-like methodology, produces a few hundred pages of documentation for each project, has few automated tests but a sizable QA team. Interestingly, the quality of our product is through the roof, and the work environment is orders of magnitude above and beyond the other company.
I know many people have had the opposite experience. As is usually the case, methodologies are not a golden hammer --- you can probably start a successful project no matter what methodology you choose. In my experience on successful and unsuccessful projects, I get the feeling that methodology doesn't matter as much as environment: comfortable, happy developers and sane project managers are all it takes make a project work.
At my company, we made a wholesale switch to agile practices about 4 years ago when a new VP came in. This VP had experienced success with Agile in the past, and decided it was what we needed.
As it turns out, he was right. I was a developer at the time (albeit a somewhat junior one), and I loved the practices. Pair programming really aided knowledge transfer and prevented the formation of knowledge silos. Unit testing, test driven development, and test emphasis in general made for more robust code that wasn't over-engineered. No Big Design Up Front meant that instead of spending 6 months writing requirements documents (by which time the market had passed us by), we were prototyping and delivering real value to customers in a timely matter. Working closely with a customer surrogate (in our case, a technical product manager) greatly shortened cycle feedback time, which helped us deliver what the customer actually wanted.
Our organization had quite a few talented developers, but we were very prone to cowboy coding. A few developers didn't like the new practices ("What do you mean, write tests? I'm a developer!"), but generally everyone loved the changes. Defect rates went down, customer satisfaction rates went up, and our office became very well regarded in our company.
About a year ago I became a manager, and I heavily use Agile practices, incorporating some Lean principles as well (value stream analysis, waste elimination, kanban). Tightening up release cycles has been an ongoing activity, and my team now releases as often as possible (with quality!) - often every two weeks. We have no field reported defects from my team in the past year, and the sales people and product management love the shorter release cycles.
As a developer, Agile really increased my confidence in working with various areas of code (I now feel nervous whenever I'm changing anything in a package that DOESN'T have 100% unit test coverage!), taught me to be a more well-rounded programmer (thinking of test implications, business impacts, etc.), and helped me write simple, self-documenting code. As a manager, Agile and kanban gives me predictability, lower lead times, lower defect rates, and an empowered team. This is not theory, or speculation, or hand waving - our team morale, defect rate, customer satisfaction, and balance sheet have proven that Agile can do wonderful things for an organization.
To comment on the Principles of the Agile Manifesto from my experience at a site that tried it.
Our highest priority is to satisfy the
customer through early and continuous
delivery of valuable software.
This was a double-edged sword for my last site -- valuable was taken to mean 100% perfect and bug-free.
Welcome changing requirements, even
late in development. Agile processes
harness change for the customer's
competitive advantage.
I still communicate with that site and just today, their rock-hard deadline date, they were given a requirement change. That was just the way things were there; it's almost as if they wanted failure.
Deliver working software frequently,
from a couple of weeks to a couple of
months, with a preference to the
shorter timescale.
The norm for many years was basically build and deploy daily, hourly, near real-time...
Business people and developers must
work together daily throughout the
project.
Some of the meetings/reviews with respect to this were hilarious. We were reprimanded for not working with the people (because they asked us not to because they were already working 9-10 hour days) and then for bothering them because they were busy.
Build projects around motivated
individuals. Give them the
environment and support they need,
and trust them to get the job done.
Ahh, here's our problem... We had top-of-the-line PCs but the business side wasn't supportive. The positive morale essentially got beaten out of you after about a year or so... This also negates your micromanagement concern (if implemented correctly).
The most efficient and effective
method of conveying information to
and within a development team is
face-to-face conversation.
This worked out well. Personally I prefer email because I hate taking notes.
Working software is the primary
measure of progress.
No doubt here.
Agile processes promote sustainable
development. The sponsors,
developers, and users should be able
to maintain a constant pace
indefinitely.
I agree with this 100%; the problem with the last business team I worked with was the expectation of 30-hour days, 10-day weeks, and 400-day years was not a pace I agreed with.
Continuous attention to technical
excellence and good design enhances
agility.
This is where some developer morale & education was needed.
Simplicity--the art of maximizing the
amount of work not done--is
essential.
I love this one and it's long been one of my goals. However, there was a "if you're not typing, you're not working" attitude that was tough to overcome.
The best architectures, requirements,
and designs emerge from
self-organizing teams.
I agree with this about 90% -- my only caveat is that they must be well-educated and well-informed teams.
At regular intervals, the team
reflects on how to become more
effective, then tunes and adjusts its
behavior accordingly.
We just failed here and it likely caused a lot of other problems we had. The business side was really good at saying "you need to do what we say needs to be done."
To wrap it up, if you're working somewhere where everyone is informed and on board with an Agile methodology, it should be a great place to work. When the goal is great software, momentum alone will carry any project.
Agile has worked awesomely for me as a Developer in my current environment. Here are some practices and why it seems to work so well:
Pair programming - This prevents anyone from feeling an individual ownership of the code. Pairs of developers tend to make better code than one person's "mad science" code that seems to happen if one person writes a bunch of code in isolation. This also allows for someone else to be brought in if someone goes away and that feature or enhancement has to get done while the person is away. Sometimes, one developer may think something will be great but if no one else can understand the code, it is useless to have unless it is perfect and futureproof which isn't likely.
Scrum - This creates both an accountability and communication so that each person knows what the other is doing. If someone wants to know how the sprint is going, just show up at the stand up. The standup is really simple in that it is just 3 questions: What did I do yesterday, what I am doing today and what would prevent me from getting that done?
Test-driven development - A variation on this is practiced where I work in that we generally try to have tests for most of the plumbing code we are writing to customize a CMS in the big project we are doing. This mindset can be tricky to get into though it does get easier as one practices it more.
YAGNI - The "You Aren't Gonna Need It" principle that can really be hard if you've been a perceptive programmer that generally puts in 101 things as a "Well, I might need this someday..." mindset. Another way to put this is a "Keep It Simple, Stupid" idea.
Sprints - The idea here just seems to prevent a sense of being overwhelmed as we are just working for 2 weeks on this or that, and don't look too far forward as it may well change.
Demos - Showing off what we have done, getting feedback on what is good and what isn't is crucial for getting things better and having a mindset that we want "continuous improvement" in the project and what is good enough today, won't be good enough tomorrow and get better at what we do.
IPM, Retrospectives - The ability to look back at what was done in retrospectives is useful for venting frustrations, celebrating things working well and finding ways to address pain points. IPM is where we determine our future for the next sprint in terms of what will be the goals and how long do we think various things will take by using a couple of different estimation tools, one for points on "epics" as we call them and the other for hours on an individual task or card which is part of a story that is something between the epic and a small piece of work.
Storywall and user stories - Now this low tech tool since it is just a few whiteboards, with dividers and post its provides some structure to things. We have lanes for each of the epics and various columns for states of work: Backlog, in development, on dev, or on test. There are also places for the task backlog, blocked cards, questions, standards and practices and a few other things that may be useful for managers to see to get an overview on the current status if they want more of a bigger picture than what they would get at standup.
Broken windows/technical debt/tasks - These are similar in some respects and are useful as a way to illustrate that quality matters,i.e. we don't want broken windows that can be easily explained in non-technical terms by either using a house in a neighbourhood or the New York Subway sytem as starting points. Technical debt being something that doesn't immediately add business value that is sometimes an important thing to use to prevent some problems as there may be problems with a particular architecture and so part of a sprint may be spent doing a re-arch that has to be communicated so that if there is a sprint with little to demo this is why.
I don't know if the idea of a "self-organizing" or "self-managing" team is part of Agile, but it has been a bit of a challenge for me to have enough faith and trust in my co-workers that things will work out fine. The professionals that are the rest of my team know what has to be done, are reasonable, honest people with integrity to just get the work done and not be jerks about getting things done. There aren't egos or bad attitudes which really do help foster building a team.
Agile hasn't worked for me, the main reason being that the systems I usually develop need a well-defined and well-thought architecture, which is hardly realisable by an agile approach. Agile approaches tend to write as little code as necessary to pass the applicable tests, and thus to grow the system organically. This can be nice from many perspectives, but it wreaks havoc from the architectural viewpoint.
From my personal experience, Agile methodology tends to create a huge technical debt in the long term, and while it might save you (as a business owner/management) a couple of bucks short term, in the long term it will come back and bite you. Whatever you do not fix now will eventually cost you many hours of work to fix at a much higher cost than it would have cost you to invest some more hours into the original problem.
Agile is always great from the point of view of beginners and management, but I do not know one experienced programmer who actually loves it. The whole point of Agile is to save development money for a company, it has nothing to do with actual product quality. In fact most of the methodology promotes bad code done quick over well-engineered code. The difference is that a few years down the road, you have to do the whole work all over again whereas the well-engineered code can last decades without corrections. Good programmers do not need Agile methodology most of the time.
We have a business logic library written 22 years ago here by a single team of 3 programmers using waterfall methodology, and that code hasn't needed a single correction since. Because it was tought properly, was well-engineered, and the three programmers took their time and were careful with their implementation. Agile methodology would instead ask of those three to do the strict minimum to make sure some ill-defined tests passed, and then wait until the next time you hit a wall to add some more duct tape code. It's a ridiculous way to work and goes against every engineer fiber in my body.
To this day I refuse to work in an Agile environment, because frankly I do not need it, and I do not need an employer who thinks I do need it.
Agile is not a methodology, it is a subset of methodologies that have a common set of goals, and more often then not those methodologies have wildly varying results based on team makeup, corporate culture, and implementation.
Off the top of my head, purely developer agile practices would include pair programming, TDD, user stories over specs, the assumption that all code is going to be refactored several times (although this is part of TDD) and code reviews more then anything. Things that impact us are daily standups, being engaged with users regularly and directly, postmortem introspections, and very tight development cycles.
I'm a developer and a manager at the same time, so I either have special insight or my opinion is totally invalid. ;)
I will say that Agile means a lot of things. It's actually a whole family of methodologies and guidelines at this point.
Exposing yourself to all these interesting ideas is really the thing. As a manager, it's very hard for me to decree that a whole team suddenly adopt a whole methodology, but if they see me constantly trying to improve every aspect of my game, I think they appreciate that. And hopefully, if they like what they see, they follow my example.
I've managed to slowly implement a bunch of things from Scrum without (hopefully) coming off as a tool. Burn down reports, stand-up meetings, and story cards on the whiteboard have really made us better at shipping software. For instance, on any project tasks are constantly being done ahead of schedule or late. In a really big project, it can become very difficult to tell what that's doing to your ship date. With burn down reports, I can tell what a slip does to our ship date, and if we need to just start cutting features in order to meet a deadline.
That's more of a management thing, but the other devs here care about it because it might mean they get to keep their jobs or avoid a death march. :)
But it's not just management stuff. There's tons in Agile about best practices for source control, unit testing, etc. Just good solid best practices. As an industry, we are pretty terrible about mentoring, so it's good that this information is out there.
From the developers perspective I think it works well. In my point of view agile techniques have in common that the loop between defining the task, working on the task and getting feedback from that task is a very small one as compared to a non-agile approaches.
Take TDD as an example: Code the test, red bar, code the functionality, green bar, refactor, red bar, fix, green bar.
From the managers perspective this faster feedback loop is also true: Daily meeting one, status green, daily meeting two, status yellow, countermeasures / re-assign ressources, daily meeting three, status green.
Immediate feedback and knowing where you are heading gives a feeling of safety.
In the so called 'traditional team', Agile development would increase the visibility of individual developers to management. That would probably allow managers and architects to plan their work better. Ofcourse that could be interpreted as micromanagement.
But from an organizational perspective, if it produces results, who cares.
I guess what makes an "agile" project agile, is the methodology: "Design for today not for tomorrow".
For any not life-critical software systems this is a way to keep programmers coding in stead of discussing ages about design. Please note that design is not scrapped, it is just done in smaller and therefore more overseeable chunks.
All other techniques that are associated with agile, like pair programming, are more borrowed ideas that could also be used effectively in any other methodology.
Now, does this technique 'work'? Yes! If applied correctly, the technique promotes that the software product will be ready for shipping at any time to react to competition.
On the other hand, because programmers are feeling they are coding more, they are generally happier. And they are less irritated by writing specs because this phase is inherently always small.
But again, if you know exactly what your product is going to be and especially if it is life-critical like the space shuttle, agile development is not what you want.
Nearly every management is aware of "Agile" by now: It's this thing, you know? Alone by your initial question I would assume that something is really going wrong. I really recommend you reading a book like Practices of an Agile Developer (as the title suggests - it's about what's in for you).
Some managers read a book and then will know what agile is all about. They are telling you what to do and everything is fine, isn't it?
If you look around, there are a lot of developers (in Agile companies) who can't tell you within a second what the purpose of a stand-up is - and that's an issue. If you (and maybe even nobody else) don't know the why the StandUp won't make things better.
Take a look at time tracking (and time estimation) - there are some managers who think it's about measuring how much work you do: Hey, you have a 40h contract but the time tracking tool says that you have only be working for 38h this week! That's not how it was meant to be used.
The only thing you can do about that: you need to learn what agile methods are out there. Mediocre managers will pick the ones they find interesting. Good managers will grasp the why and not only choose the methods for their direct benefit - but also those which will make the team more happy / efficient / teamish (Team vs Workgroup).
P.S. Something you really need to take care of: In agile there is no place for slackers. Everybody has to do stuff on their own. You have to put personal interest into the success of the product. If you don't do things on your own, somebody will tell you what to do (and then there's micromanagement).
Has Agile really worked? "Yes."
Before there was "Agile Programming" there were equivalent largely unrecognized methodologies. I thought these were called incremental prototyping but apparently this has been split into that and evolutionary prototyping.
I suspect that many or most of the successful systems were so constructed. Just because the methodology grew a new name doesn't mean that it suddenly appeared.
It's just that Waterfall and other broken management techniques that got all the press.
I say Agile works.
I say it's the only thing that ever worked.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
I know that for me I first got started following the waterfall method of project management and along with that I went with the predictive approach to software design. In this I mean we had huge packets of documentation, UML, database schemas, data dictionaries, workflows, activity diagrams, etc.
Having worked in software for over 10 years now I find it to be much more realistic to approach software design from a Reactive approach. I frequently follow a scrum approach to project management and with that very little heavy documentation is ever generated. We have very little workflow specification (though they still have there use). This is a much more dynamic approach to software creation. Of course along with it comes frequent refactoring as time goes on as we find out new features over time that had we planned for up front would have changed things dramatically.
The big difference for us is that the first approach takes longer, seems to fail more frequently in a software construction world, and isn't nearly as flexible. The second approach provides more flexibility, makes us aware of failure faster (so we can course correct faster), and provides some form of functionality at the end of every iteration.
Knowing both sides from experience, I still find many people that LOVE the waterfall approach over the agile approach for software development. I don't get it.
question: Why would someone use waterfall over some form of agile with all of the research backing agile? What are strong arguments for using waterfall over agile?
When I started programming (with COBOL no less), waterfall was the "new" approach. Today, I'd tell you that I use a waterfallish agile methodology. For larger systems, I find a waterfall type start works best. Not creating huge documents (that's a waste of time IMO) but rather to take some steps like creating a UI prototype and/or use cases to get our heads around the business problem at hand. Once we are comfortable we have the problem scoped and we have a solid understanding, we move into an agile development mode.
To answer your question though, I think the big reason waterfall sticks around is many people don't like change. It's scary to change and moving from waterfall to agile is a big change.
I think that part of the reason why people still often cling to waterfall is that it gives the illusion of control. In a waterfall, you can do enough up front work to put together a beautiful schedule that nicely addresses every contingency that you can think of, and then give a detailed roadmap for the future to anyone on the business side who asks when feature X will be available.
The problem is that you can almost never follow that plan to the letter, and you are almost always late/dropping features. However from the upfront, it looks very controlled and manageable.
I'm a big Agile fan, but what I've always struggled with is the long range roadmap/forecasting that is often asked for by the sales and marketing folks. I think that the waterfall's illusion of certainty is very comforting to managers and business folks.
My boss tells me to.
I suspect many people have no choice and old bosses don't learn new tricks.
Not taking sides, but pretty much any research would be unscientific at best.
You say (emphasis is mine)
question: Why would someone use waterfall over some form of agile with all of the research backing agile? What are strong arguments for using waterfall over agile?
but don't link to any studies.
It's one of those things that are known to be extremely difficult to actually test. You can't have two identical teams work on the same project at the same time, because there's no such thing as two identical teams. You can't have the same team complete the same task twice in a row using two different methodologies without the first pass tainting the second. I've never heard of anyone designing an experimental (or even statistical) study that can convincingly argue for any software development methodology. I'd be interested to see one though, if you have a link.
Short of real evidence, it boils down to personal preference. What are the strong arguments for chocolate over vanilla?
I'll play devil's advocate and state that agile is flawed is nearly as many ways as the waterfall method is. I'm not one of those that love the waterfall method, but I don't love agile either.
My experience with agile hasn't been very positive. To be fair, I used it in a corporate environment, which paid lip service to "agile" while still expecting our manager to produce long term milestones and deliverables upfront.
However, I found that agile (scrum in particular) methodologies often disguise major problems with design. While waterfall gives managers the illusion of control, agile seems to do the same for development teams. I've seen teams where bringing up any issue that aren't in the current sprint/iteraton is frowned upon, with the expectation that it'll be handled "in time". It only requires a few major design decisions to be ignored for the project to go belly up in future, while current iterations go smoothly and project looks to be on track.
You can argue that the team's at fault for not understanding the spirit of agile, but I'd like to see better methodologies that incorporate the best parts of agile.
One of the premises of (at least) XP is that change is cheap. The waterfall model was built on the principles that change, any change, is costly. The assumption in the waterfall model is that once software has been written, changing it is more expensive than investing the time up front to come to a "complete" understanding of the problem.
Experience seems to indicate that it is very hard to come to a complete understanding of the problem and that if some precautions are taken (e.g. Unit Testing) change can become a lot cheaper. Therefore if you encounter a problem where some of the agile premises don't hold true other approaches might become feasible again. In between Waterfall and Agile there is at least Spiral development which is - sort of - what we practice.
You need to be preditive enough to deliver the goods. You need to be reactive enough to deal with the issues.
I was once stuck with six months to complete a project estimated to take a year, and based on past experience experience would take two. So I spent three months researching methodolgies. We finished on time (in three months), using the appropriate parts of a waterfall process.
A few points that made the methodoly work:
- Create an use standards, update them when needed.
- Build libraries: Do it once, do it well, fix it without breaking existing code.
- Do just enough documentation.
- Version control everything you can.
- Break things down; a method should either manage work or do work.
- Increase cohesion, decrease coupling, reuse.
- Buy or build the tools you need.
- Track your issues and progress.
Another project I was breifly involved was a six month project. I didn't get involved until a year and a half after it started. The development lead had been hired at an extreme markup as he was leaving a career with a pension plan. At the start of the project he asked the project manager, "Do you want me to do it right or be reactive?" Can you guess the answer? The week I was involved same feature was implemented on Monday, Wednesday, and Friday. Guess what happened Tuesday and Thursday?
The strength in Agile is its emphasis on just enough, just in time. The strength in the waterfall methodoly is that it covers all the things you need to think about. I've yet to work on a project that did or should have done all the steps. I have worked on many projects that did steps which should have been done on a corporate basis.
The title says it all. (Actually: proactive vs reactive). Why chose the reactive way and give up control unless you don't have to? Waterfall is not the only alternative, you can have any kind of development process what you refine when you like. Control is the key.
It's a spectrum btw, the waterfall on one end and the totally reactive, zero documentation methods on the other end. If you work in the consultant industry for powerful (and usually indecisive) clients, you have to resort to reactive methods. If you develop shrinkwrap software you can plan ahead and manage knowledge. Some projects also require tons of specifications and rules, where the code and fix approach just don't cut it. For me software engineering is primarily about knowledge management and design, coding comes second.
P.s. there is no such a thing as agile and fixed price. Not in the classical way they usually sell the method. See http://martinfowler.com/bliki/FixedPrice.html
If you exactly know the requirements that never chagen, if you know how long each step will take and if you know all the resources are avaliable at the time needed you can do waterfall and it will work. But in deed these kind of projects are quite rare and I think I will never be part of it.
When designing systems to be used by end users, agile often works well because the requirements are likely to be incorrect and a large part of the process is getting feedback from users in the form of a usable version.
However, when creating software that interfaces with other software often the requirements can be worked out in very clearly. In this case it is often more productive to ensure that you have a very clear and accurate specification, unit tests In this model you can also generate fairly good work estimates and there would be a great deal more cost to use the agile model.
retroactive behaviour
If you have a team of a few dozen people that have over the course of a decade, refined the waterfall strategy to the point that it works well for them, who are you to come in and say, "You're doing it wrong..."? Really, if it is working for them, why change things? Yes this is merely flipping the question around but I think it may be a valid point.
In my team we've found that with maintenance projects (which is the bulk of what we do) where we're tweaking or replacing like with like there isn't always as much need for user input beyond perhaps some UI prototypes.
In that case, particularly given that there are commercial deals involved the waterfall approach at a macro level can fit well. Even then we still like incremental / agile approaches at the implementation level.
It is worth noting that most of our clients are large lumbering organisations in love with their paperwork, so that adds even more impetus for us to at least appear traditional to them.
The documentation generated during the waterfall process allows for a lot of CYA. You can point fingers when a project goes off the rails. Very few executives are going to be OK with "oh well, I guess that project got away from us! Well, at least we found out early, no harm no foul!"
Also, design docs can automatically generate test plans, which is useful for QA.
It's pretty common when bidding for a contract that one of the iron-clad conditions is that you follow their "process" which on inspection is waterfall.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
A couple of years ago the media was rife with all sorts of articles on
how the idea of code reuse was a simple way to improve productivity
and code quality.
From the blogs and sites I check on a regular basis it seems as though
the idea of "code reuse" has gone out of fashion. Perhaps the 'code
reuse' advocates have all joined the SOA crowd instead? :-)
Interestingly enough, when you search for 'code reuse' in Google the
second result is titled:
"Internal Code Reuse Considered Dangerous"!
To me the idea of code reuse is just common sense, after all look at
the success of the apache commons project!
What I want to know is:
Do you or your company try and reuse code?
If so how and at what level, i.e. low level api, components or
shared business logic? How do you or your company reuse code?
Does it work?
Discuss?
I am fully aware that there are many open source libs available and that anyone who has used .NET or the Java has reused code in some form. That is common sense!
I was referring more to code reuse within an organizations rather than across a community via a shared lib etc.
I originally asked;
Do you or your company try and reuse code?
If so how and at what level, i.e. low level api, components or shared business logic? How do you or your company reuse code?
From where I sit I see very few example of companies trying to reuse code internally?
If you have a piece of code which could potentially be shared across a medium size organization how would you go about informing other members of the company that this lib/api/etc existed and could be of benefit?
The title of the article you are referring to is misleading, and is actually a very good read. Code reuse is very beneficial, but there are downsides with everything. Basically, if I remember correctly, the gist of the article is that you are sealing the code in a black box and not revisiting it, so as the original developers leave you lose the knowledge. While I see the point, I don't necessarily agree with it - at least not to a "sky is falling" regard.
We actually group code reuse into more than just reusable classes, we look at the entire enterprise. Things that are more like framework enhancement or address cross-cutting concerns are put into a development framework that all of our applications use (think things like pre- and post-validation, logging, etc.). We also have business logic that is applicable to more than one application, so those sort of things get moved to a BAL core that is accessible anywhere.
I think that the important thing is not to promote things for reuse if they are not going to really be reused. They should be well documented, so that new developers can have a resource to help them come up to speed, as well. Chances are, if the knowledge isn't shared, the code will eventually be reinvented somewhere else and will lead to duplication if you are not rigorous in documentation and knowledge sharing.
We reuse code - in fact, our developers specifically write code that can be reused in other projects. This has paid off quite nicely - we're able to start new projects quickly, and we iteratively harden our core libraries.
But one can't just write code and expect it to be re-used; code reuse requires communication among team members and other users so people know what code is available, and how to use it.
The following things are needed for code reuse to work effectively:
The code or library itself
Demand for the code across multiple projects or efforts
Communication of the code's features/capabilities
Instructions on how to use the code
A commitment to maintaining and improving the code over time
Code reuse is essential. I find that it also forces me to generalize as much as possible, also making code more adaptable to varying situations. Ideally, almost every lower level library you write should be able to adapt to a new set of requirements for a different application.
I think code reuse is being done through open source projects for the most part. Anything that can be reused or extended is being done via libraries. Java has an amazing number of open source libraries available for doing a large number of things. Compare that to C++, and how early on everything would have to be implemented from scratch using MFC or the Win32 API.
We reuse code.
On a small scale we try to avoid code duplication as much as posible. And we have a complete library with a lot of frequently used code.
Normally code is developed for one application. And if it is generic enough, it is promoted to the library. This works excelent.
The idea of code reuse is no longer a novel idea...hence the apparent lack of interest. But it is still very much a good idea. The entire .NET framework and the Java API are good examples of code reuse in action.
We have grown accustomed to developing OO libraries of code for our projects and reusing them in other projects. Its a part of the natural life cycle of an idea. It is hotly debated for a while and then everyone accepts and there is no reason for further discussion.
Of course we reuse code.
There are a near infinite amount of packages, libraries and shared objects available for all languages, with whole communities of developers behing them supporting and updating.
I think the lack of "media attention" is due to the fact that everyone is doing it, so it's no longer worth writing about. I don't hear as many people raising awareness of Object-Oriented Programming and Unit Testing as I used to either. Everyone is already aware of these concepts (whether they use them or not).
Level of media attention to an issue has little to do with its importance, whether we're talking software development or politics! It's important to avoid wasting development effort by reinventing (or re-maintaining!) the wheel, but this is so well-known by now that an editor probably isn't going to get excited by another article on the subject.
Rather than looking at the number of current articles and blog posts as a measure of importance (or urgency) look at the concepts and buzz-phrases that have become classics or entered the jargon (another form of reuse!) For example, Google for uses of the DRY acronym for good discussion on the many forms of redundancy that can be eliminated in software and development processes.
There's also a role for mature judgment regarding costs of reuse vs. where the benefits are achieved. Some writers advocate waiting to worry about reuse until a second or third use actually emerges, rather than spending effort to generalize bit of code the first time it is written.
My personal view, based on the practise in my company:
Do you or your company try and reuse code?
Obviously, if we have another piece of code that already fits our needs we will reuse it. We don't go out of our way to use square pegs in round holes though.
If so how and at what level, i.e. low level api, components or shared business logic? How do you or your company reuse code?
At every level. It is written into our coding standards that developers should always assume their code will be reused - even if in reality that is highly unlikely. See below
If your OO model is good, your API probably reflects your business domain, so reusable classes probably equates to reusable business logic without additional effort.
For actual reuse, one key point is knowing what code is already available. We resolve this by having everything documented in a central location. We just need a little discipline to ensure that the documentation is up-to-date and searchable in a meaningful way.
Does it work?
Yes, but not because of the potential or actual reuse! In reality, beyond a few core libraries and UI components, there isn't a large amount of reuse.
In my personal opinion, the real value is in making the code reusable. In doing so, aside from a hopefully cleaner API, the code will (a) be documented sufficiently for another developer to use it without trawling the source code, and (b) it will also be replaceable. These points are a great benefit to on-going software maintenance.
Do you or your company try and reuse code? If so how and at what
level, i.e. low level api, components or shared business logic? How do
you or your company reuse code?
I used to work in a codebase with uber code reuse, but it was difficult to maintain because the reused code was unstable. It was prone to design changes and deprecation in ways that cascaded to everything using it. Before that I worked in a codebase with no code reuse where the seniors actually encouraged copying and pasting as a way to reuse even application-specific code, so I got to see the two extremities and I have to say that one isn't necessarily much better than the other when taken to the extremes.
And I used to be an uber bottom-up kind of programmer. You ask me to build something specific and I end up building generalized tools. Then using those tools, I build more complex generalized tools, then start building DIP abstractions to express the design requirements for the lower-level tools, then I build even more complex tools and repeat, and at some point I start writing code that actually does what you want me to do. And as counter-productive as that sounded, I was pretty fast at it and could ship complex products in ways that really surprised people.
Problem was the maintenance over the months, years! After I built layers and layers of these generalized libraries and reused the hell out of them, each one wanted to serve a much greater purpose than what you asked me to do. Each layer wanted to solve the world's hunger needs. So each one was very ambitious: a math library that wants to be amazing and solve the world's hunger needs. Then something built on top of the math library like a geometry library that wants to be amazing and solve the world's hunger needs. You know something's wrong when you're trying to ship a product but your mind is mulling over how well your uber-generalized geometry library works for rendering and modeling when you're supposed to be working on animation because the animation code you're working on needs a few new geometry functions.
Balancing Everyone's Needs
I found in designing these uber-generalized libraries that I had to become obsessed with the needs of every single team member, and I had to learn how raytracing worked, how fluids dynamics worked, how the mesh engine worked, how inverse kinematics worked, how character animation worked, etc. etc. etc. I had to learn how to do pretty much everyone's job on the team because I was balancing all of their specific needs in the design of these uber generalized libraries I left behind while walking a tightrope balancing act of design compromises from all the code reuse (trying to make things better for Bob working on raytracing who is using one of the libraries but without hurting John too much who is working on physics who is also using it but without complicating the design of the library too much to make them both happy).
It got to a point where I was trying to parametrize bounding boxes with policy classes so that they could be stored either as center and half-size as one person wanted or min/max extents as someone else wanted, and the implementation was getting convoluted really fast trying to frantically keep up with everyone's needs.
Design By Committee
And because each layer was trying to serve such a wide range of needs (much wider than we actually needed), they found many reasons to require design changes, sometimes by committee-requested designs (which are usually kind of gross). And then those design changes would cascade upwards and affect all the higher-level code using it, and maintenance of such code started to become a real PITA.
I think you can potentially share more code in a like-minded team. Ours wasn't like-minded at all. These are not real names but I'd have Bill here who is a high-level GUI programmer and scripter who creates nice user-end designs but questionable code with lots of hacks, but it tends to be okay for that type of code. I got Bob here who is an old timer who has been programming since the punch card era who likes to write 10,000 line functions with gotos in them and still doesn't get the point of object-oriented programming. I got Joe here who is like a mathematical wizard but writes code no one else can understand and always make suggestions which are mathematically aligned but not necessarily so efficient from a computational standpoint. Then I got Mike here who is in outer space who wants us to port the software to iPhones and thinks we should all follow Apple's conventions and engineering standards.
Trying to satisfy everyone's needs here while coming up with a decent design was, probably in retrospect, impossible. And in everyone trying to share each other's code, I think we became counter-productive. Each person was competent in an area but trying to come up with designs and standards which everyone is happy with just lead to all kinds of instability and slowed everyone down.
Trade-Offs
So these days I've found the balance is to avoid code reuse for the lowest-level things. I use a top-down approach from the mid-level, perhaps (something not too far divorced from what you asked me to do), and build some independent library there which I can still do in a short amount of time, but the library doesn't intend to produce mini-libs that try to solve the world's hunger needs. Usually such libraries are a little more narrow in purpose than the lower-level ones (ex: a physics library as opposed to a generalized geometry-intersection library).
YMMV, but if there's anything I've learned over the years in the hardest ways possible, it's that there might be a balancing act and a point where we might want to deliberately avoid code reuse in a team setting at some granular level, abandoning some generality for the lowest-level code in favor of decoupling, having malleable code we can better shape to serve more specific rather than generalized needs, and so forth -- maybe even just letting everyone have a little more freedom to do things their own way. But of course all of this is with the aim of still producing a very reusable, generalized library, but the difference is that the library might not decompose into the teeniest generalized libraries, because I found that crossing a certain threshold and trying to make too many teeny, generalized libraries starts to actually become an extremely counter-productive endeavor in the long term -- not in the short term, but in the long run and broad scheme of things.
If you have a piece of code which could potentially be shared across a
medium size organization how would you go about informing other
members of the company that this lib/api/etc existed and could be of
benefit?
I actually am more reluctant these days and find it more forgivable if colleagues do some redundant work because I would want to make sure that code does something fairly useful and non-trivial and is also really well-tested and designed before I try to share it with people and accumulate a bunch of dependencies to it. The design should have very, very few reasons to require any changes from that point onwards if I share it with the rest of the team.
Otherwise it could cause more grief than it actually saves.
I used to be so intolerant of redundancy (in code or efforts) because it appeared to translate to a product that was very buggy and explosive in memory use. But I zoomed in too much on redundancy as the key problem, when really the real problem was poor quality, hastily-written code, and a lack of solid testing. Well-tested, reliable, efficient code wouldn't suffer that problem to nearly as great of a degree even if some people duplicate, say, some math functions here and there.
One of the common sense things to look at and remember that I didn't at the time is how we don't mind some redundancy when we use a very solid third party library. Chances are that you guys use a third party library or two that has some redundant work with what your team is doing. But we don't mind in those cases because the third party library is great and well-tested. I recommend applying that same mindset to your own internal code. The goal should be to create something awesome and well-tested, not to fuss over a little bit of redundancy here and there as I mistakenly did long ago.
So these days I've shifted my intolerance towards a lack of testing instead. Instead of getting upset over redundant efforts, I find it much more productive to get upset over other people's lack of unit and integration testing! :-D
While I think code reuse is valuable, I can see where this sentiment is rooted. I've worked on a lot of projects where much extra care was taken to create re-usable code that was then never reused. Of course reuse is much preferable to duplicate code, but I have seen a lot of very extenisve object models created with the goal of using the objects across the enterprise in multiple projects (kind of the way the same service in SOA can be used in different apps) but have never seen the objects actually used more than once. Maybe I just haven't been part of organizations taking good advantage of the principle of reuse.
The two software projects I've worked on have both been long term development. One is about 10 years old, the other has been around for over 30 years, rewritten in a couple versions of Fortran along the way. Both make extensive reuse of code, but both rely very little on external tools or code libraries. DRY is a big mantra on the newer project, which is in C++ and lends itself more easily to doing that in practice.
Maybe the better question is when do we NOT reuse code these days? We are either in a state on building using someone elses observed "best practices" or prediscovered "design patterns" or just actually building on legacy code, libraries, or copying.
It seems the degree to which code A is reused to make code B is often based around how much the ideas in code A taken to code B are abstracted into design patterns/idioms/books/fleeting thoughts/actual code/libraries. The hard part is in applying all those good ideas to your actual code.
Non-technical types get overzealous about the reuse thing. They don't understand why everything can't be copy-pasted. They don't understand why the greemelfarm needs a special adapter to communicate the same information that it used to to the old system to the new system, and that, unfortunately we can't change either due to a bazillion other reasons.
I think techies have been reusing from day 1 in the same way musicians have been reusing from day 1. Its an ongoing organic evolution and sythesis that will keep ongoing.
Code reuse is an extremely important issue - where code is not reused, projects take longer and are harder for new team members to get into.
However, writing reusable code takes longer.
Personally, I try to write all my code in a reusable way, this takes longer, but it results in the fact that most of my code has become official infrastructures in my organization and that new projects based on these infrastructures take significantly less time.
The danger in reusing code, is if the reused code is not written as an infrastructure - in a general and encapsulated manner with as few as possible assumptions and as much as possible documentation and unit testing, that the code can end up doing unexpected things.
Also, if bugs are found and fixed, or features added, these changes are rarely returned to the source code, resulting in different versions of the reused code, that no one knows of or understands.
The solution is:
1. To design and write the code with not only one project in mind, but to think of future requirements and try to make the design flexible enough to cover them with minimal code change.
2. To enclose the code within libraries that are to be used as-is and not modified within using projects.
3. To allow users to view and modify the code of of the library withing its solution (not within the using project's solution).
4. To design future projects to be based on the existing infrastructures, making changes to the infrastructures as necessary.
5. To charge maintaining the infrastructure to all projects, thus keeping the infrastructure funded.
Maven has solved code reuse. I'm completely serious.