Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
I'm very positive towards agile development, and have worked on agile projects on and off for about 13 years. But I have a concern that I've never really been able to address. It doesn't always seem to manifest, but it has bitten me a few times.
Agile seems to be in some sense a 'greedy algorithm'. Start with the highest value story, optimise the system to precisely fulfil that story, and repeat.
Actual greedy algorithms are prone to suffering from converging to locally optimal solutions, while missing a globally optimal solution.
Has this been people's experience?
Is it actually a problem?
If so, what techniques do you use to avoid such local optima and yet remain agile?
Actual greedy algorithms are prone to suffering from converging to locally optimal solutions, while missing a globally optimal solution.
This holds true if EPIC technical User Story and guideline is not established, along with the normal business EPIC user story.
Has this been people's experience?
At times yes, it has been my experience. One instance was when the user stories we worked on were broken down too much, and the solution was to broaden them to get a more global outlook at our designs. And at times it was different enterprise scrum teams in the same projectt, conflicting with different technical framework uses and approaches.
Is it actually a problem?
It is only a problem, if you ignore the technical EPIC user story or guideline.
If so, what techniques do you use to avoid such local optima and yet remain agile?
Here is one Agile approach to solving this:
During Agile Release planning, instead of just coming up with a Business EPIC User Story, also come up with a Technical EPIC User Story. The Technical EPIC User story would have the product vision from a technical stand point, in terms of technical architecture, application framework, quality standards, and global design considerations etc. These could be broken down into smaller technical user stories, and have a Scrum Team which works on getting those user stories working. An example of a user story could be: "As a Technical Project Manager, I want the whole enterprise project using A, B, C framework, and coding as per X,Y,Z coding standards, so that there is uniformity in project development work.
If you don't want to form a scrum team separately for this, then just keep them as reminder cards next to backlogs for development teams to use as guidelines.
As a testing guideline, we used to have successful integration testing as a done criteria for each backlog. A global test was conducted in an integration environment, on all working software deployed from all enterprise teams, to deem it shippable. So right from inception to end of the backlog, the theme is set for global working software and not just local working software.
Finally, Agile development involves keeping a constant eye on quality, and one of the quality issues could be bad design or a too localized design. As and when this is discovered, it should be redesigned within that backlog itself, and followed going forward for other Backlogs.
I've been on a project which has had this problem, and has not dealt with it effectively.
The local quality of the code - over the scale of a package, say - was not bad. But there were problems at larger scales; things like duplication of logic (but not code) between packages, use of batch recomputation jobs where we should be using event-driven approaches, splitting the system into separate services at the wrong place, etc.
None of these problems could be fixed by refactoring a single class or package. As a result, they never happened in the normal course of events. We did refactorings at smaller scale - when adding a feature, we'd refactor in that area before starting, and again after we finished (as well as making some effort to write good code as we were going). But that never led to refactoring larger, architectural concerns.
We were all conscious of the problems, we just didn't have anything in our process that let us fix them.
One notable victory we did have was where there was duplication between two distantly related module. Essentially, there was code to render a web page showing the results of some set of calculations, and also a background job to generate reports doing similar calculations. The calculation code was shared, but the code to set up the calculations was not; one was driven by a user's view preferences, whereas the other was driven by a configured reporting job. We had a feature to implement that would have involved adding a new aspect to the calculations, which would have meant adding more items to both kinds of configuration, and then adding business logic to both sets of calculation setup code. We managed to get the product manager (our customer proxy) to agree to budget enough time for the work that we could refactor to unite the ideas of user view preferences and configured reporting job, so throwing away one of the sides of the duplication, then implement the feature. This took longer than just implementing it twice, but the product manager was wise enough to realise that this would let us implement future features spanning both pages and reports more quickly.
The mechanism in the process by which we did this was writing stories for the job of refactoring. Essentially, something like "As a product manager, i want pages and reports to use common calculation setup code, so that i can get features added more quickly". This is absolutely not a proper kind of story, but it fitted in the system, and it did the job.
I think that if the running of this project had been a bit healthier, then there would have been a steady stream of stories like this. We would acknowledge that we had a lot of architectural debt, and that work to pay it off had value, and allocate a fixed fraction of our time to it, perhaps about 20% (which would really mean one pair at a time). We could then have generated features/epics, stories, and tasks just as we did for customer-oriented work. These would originate from the team themself, rather than the product managers.
Sadly, there wasn't quite enough communication and trust between the development and product management sides that this was feasible; we could say to product that we had a problem, it was important, and that it would take so long to fix, and they couldn't know if that was true or not. As such, they were generally unwilling to schedule time to do it. The sad thing was that everyone was in agreement that there were problems and it would be good to fix them, we just had an impasse over actually doing it.
in my experirence, if you´re working a project context with fixed time/requirements then yes, most of the times Agile leads to local optima.
But my point is that in a complex endeavour, the requirements, the team itself and even the goals change. Agile is also about embracing changes.
Then, paradoxically, this greedy strategy arrises as a reasonable option for global optimization when dealing with moving targets.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
Obviously the differences in the impact to teams, customers, ROI, etc. of applying the two approaches is huge and is the subject of many books and endless discussions and conferences.
But as I think more about it, I am having a hard time finding any difference between the two that doesn't ultimately map to a single root difference which is the frequency of release.
Waterfall spends time on a design, then writing the code, then testing and finally releasing. But Agile does exactly the same set of steps - its just that each one is smaller.
A key piece of the Agile approach is about learning from each release and using that to let the larger design emerge instead of trying to predict it at the beginning.
But Waterfall does this too. Its just that instead of learning every 3 or 4 weeks, the Waterfall team only learns every 6 or 9 months. But the Waterfall design still emerges. That is, waterfall release 2 will reflect what was learned in release 1. So the process is not different, its just that it executes at a different speed.
Agile focuses on close customer collaboration. But Waterfall does this too. Its just that since waterfall has a longer iteration time, an enumerated list of requirements in the form of a contract is more needed to keep everyone on the same page over the long period of time. But again, this is just an artifact of frequency. The higher the frequency of delivery, the lower the need for a contract.
Are there any other primitive differences that I am missing - or is it really just frequency?
Waterfall:
you design the product
you build it
you test it
you document it
you release it when you have developed all the requirements
Agile:
you design the most valuable feature (or user story) first
you test it (TDD ;))
you built it
you document it
you repeat the process with the next most valuable feature
you can potentially release it at any time
(after each feature you completed or after the time boxed period (usually called sprint or iteration))
The difference is pretty clear to me.
With Agile, you can adapt what to build by delivering small chunk of the software frequently. You can stop when you have enough.
Faster feedback - at all scales, not just releases, is certainly one common factor in many agile practices. But I don't really think it's the primary difference between agile & waterfall. For example:
Waterfall teams tend to be built around group of narrow specialisations. The analysts/architects/designers/coders/testers tend to be separate groups of people working alone. Agile teams work together.
Waterfall processes depend on lots of documentation and handovers. Agile teams are oriented around working code/products.
I'd disagree that waterfall focusses on customer collaboration. There is a single point of contact, with a small group of the overall development team, often only at the start of the process. Agile is built around continual collaboration throughout the whole development process. Very different.
Waterfall processes are built around the idea that you can fully define the product & architecture before development begins. Agile processes are built around the idea that you discover the product/architecture as you go.
Waterfall spends time on a design, then writing the code, then testing and finally releasing. But Agile does exactly the same set of steps - its just that each one is smaller.
Agile is not a single entity, but an umbrella for many varying methodologies.
In at least some of them, as others have noted, these "phases" overlap much more and are in somewhat different normal order.
In fact, in XP, the order is more or less:
test (TDD/test first)
code
design (refactoring)
repeat and eventually release
which sort of inverts most of the sequence.
And the test, code and design is done at a finer grade than the release.
A key piece of the Agile approach is about learning from each release and using that to let the larger design emerge instead of trying to predict it at the beginning.
But Waterfall does this too. Its just that instead of learning every 3 or 4 weeks, the Waterfall team only learns every 6 or 9 months. But the Waterfall design still emerges. That is, waterfall release 2 will reflect what was learned in release 1. So the process is not different, its just that it executes at a different speed.
This depends heavily on practice. As described in DOD-STD-2167A, (Section 4.1.1) the waterfall model does indeed allow for the phases of the development process to overlap and iterate (in short, to be somewhat agile). But most actual practice missed that, and there was no learning until the end of the project.
Agile focuses on close customer collaboration. But Waterfall does this too. Its just that since waterfall has a longer iteration time, an enumerated list of requirements in the form of a contract is more needed to keep everyone on the same page over the long period of time. But again, this is just an artifact of frequency. The higher the frequency of delivery, the lower the need for a contract.
Again practice-dependent. I don't see in the spec referenced above much mention of the customer responsibilities at all, let alone continuously.
As Jerry Coffin noted in a comment, Waterfall is indeed a strawman used to argue in favor of Agile (as indeed I'm using it now...), but it's worth looking at this document and thinking about what it implies and how it's been applied.
What this spec does show is an intense focus on contracts, delivery and maintenance of plans and documents, and adhering to plan. And I believe that did carry over into practice.
The contrast with agile is not the timeboxing, but a change in values.
As The Agile Manifesto proclaims:
We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more.
Curiously, this values statement says nothing about frequency of delivery (though the following "Principles" section does). It does however shift the value system away from plans, documents and contracts and back where it belongs, on actually delivering working software.
Frequent release is a mechanism for fulfilling these values.
If you worked in "mini-waterfalls" it would indeed be a bit more agile than the strawman BDUF waterfall. But frequency of delivery is certainly not the whole story.
One difference is transparency: whether people outside the dev team, during the dev cycle, have any visibility into the process, the progress, the obstacles, and what the end result will look like.
Waterfall does not imply transparency. Often (though not necessarily) it's implemented as "programmers go into their cave and emerge n weeks/months later with a 'finished' product". The business experts write the specs up-front, and that may be the end of their involvement -- they might no longer be available when the programmers have questions during implementation. The programmers might not show any deliverables to anyone until the end of the cycle.
Agile requires transparency -- it's part of the basic fabric. People outside the team will (or at least can) see what the team is doing. (If not, the team is lying about being Agile.) This might be the daily stand-up meetings of Scrum, or the Big Visible Charts and information radiators, or the demo at the end of the iteration. It might be XP's requirement that the Customer make all the business decisions, instead of leaving the programmers to scratch their heads and blindly pick an option when the requirements aren't clear. It might be the fact that the testers -- and the Customer -- are considered part of the team, not separate teams.
You certainly could run Waterfall with transparency, and in a well-run Waterfall shop, you probably would. But with Agile, it's a given.
Mark,
As you pointed out, both approaches share "good things" that should be in every good project. For instance take these two: early feedback and transparency. While it is true that Agile has an structure that encourages this, these two "good things" can (and should) be in any waterfall project, too.
However, I tend to (respectfully!) disagree with the idea that release frequency is the only difference. One substantial difference is the following:
Waterfall spends time on a design,
then writing the code, then testing
and finally releasing. But Agile does
exactly the same set of steps - its
just that each one is smaller.
I don't think so.
In Agile, you attempt to do all these things concurrently, with a multidisciplinary team.
I say "attempt" because is not something that can be easily done... but at least trying helps.
On the traditional waterfall, on the contrary, you expect to have separate teams (research/analysis, QA, design, marketing, etc.) and hand offs between them. You mix disciplines and form a special team only in exceptional cases, or when you need to do exploratory research or risk analysis in a complex project.
Just my two cents...
I really like this question.
I have worked maintenance with bad examples of massive waterfall projects. The deliverables for the initial developers were several sets of documents, and one codebase. Once the high-level design document was completed, it was delivered, and never again updated. Ditto SRS, Detail design etc. There is documentation, all of which is unreliable and suspect.
With Agile, there is code. The long-since-gone developers thought it was self-documenting, because they were current with the project when it was written. (Have you ever proofread your own memos? They always make sense to the author.) I will try to discern the architecture from looking at 1500-2000 modules. Likewise trying to figure out the high-level design. I will take copious notes. Maybe even binders full. Looking at the "self-documenting" code will tell me what is being done (in that module), but not why. Oh yeah, the staff who collaborated with the developers got promoted (or scared) and are no longer available.
Bad agile is not better than bad waterfall. In fact, bad agile may be worse than bad waterfall. Is good agile better than good waterfall?
The manifesto says nothing about documentation. The real danger here is that "agile" means to many developers and clients a justification of a quick cheap heroic model. Do you think the customer enjoyed reading the three thick three-ring-binders of high-level design? We all heard in Computer Science 100 that the majority of the cost of software is maintenance, not development. Am I incorrect in thinking that the maintenance aspect is totally ignored in this discussion?
The difference may be that modern clients cannot afford to not specify "agile" because they fear being thought backward.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
I have been interested in agile methods of late and have found a lot of prescriptions and minute descriptions of a lot of practices.
Still, I remember my best projects as run-to-completion spikes followed by some debugging and minimal testing before going live.
I have been asking myself, did Flickr use agile methods? Does Facebook practice TDD? Was Gmail made in 25 minute spans followed by 5 minutes of daydream?
In other words, before I listen further to all the preaching and jump into the manuals, what evidence do I get that this is the way to be successful in a successful project in a successful company?
Of course, I am asking this because I want to read the answers, not because I want to dismiss an argument.
A related question is, how many non-Agile (Waterfall, "Big Design Up Front", etc) projects are successful? In my experience, not many. In fact, I just rolled off a two-phase project in which the first phase was traditional Waterfall and failed pretty significantly, but the second phase was iterative in nature and yielded substantially better results (on time, far fewer defects, end result was closer to client's actual needs than the original spec).
I've been doing Agile development for a few years now and, overall, have found it to be superior to the alternative. A few things I've noticed:
Agile != "no process". Agile is about having only as much process as you need and continually refining that process.
Agile requires discipline. You not only have to have a process, you have to follow it.
Agile won't turn a failing project into a success. It can help you identify that the project is failing sooner rather than later, and help you figure out why it's failing. It's about shortening the feedback loop so that you have a chance to get back on course before it's too late.
Microsoft Research recently posted an article in which they empirically evaluate some Agile methods. It's well worth a read and might provide some of the information you're looking for.
Here's a successful project of mine: http://www.sky.com
Went live after a few months.
Delivered new functionality to the CMS and servers behind the site weekly - with deployments typically every week or so.
All done with all the extreme programming disciplines.
Weekly demos to the customer to go with the weekly iterations.
Here's another agile project (also done strictly with XP), also a big success: http://showbiz.sky.com/
I've also worked on two other successful XP projects:
Banking A system for cleaning up and distributing fixed income data across investment bank sites in NY, London, Paris and Tokyo. I believe the whole project only had one production incident over the course of a few years.
Mobile Data A system for configuring mobile phones and PDAs for mobile networks and handset manufacturers. We built the core product incrementally over a number of years and co-ordinated the work over three sites across the world. All done using extreme programming. Customers were some of the largest companies in the mobile business. Our apps provided global support for some of these clients.
I really wouldn't go back to the old way of doing things - and neither would the customers that sponsored the projects I've mentioned above.
In most of the big companies (IBM for instance), the methodology is not always the same, Agile or Rational or Waterfall. That depends in a lot of the history of the projects and the experience of the current People and Project Managers.
If you plan to develop on something is always good to check on all the sides before deciding what suits best for your plan.
So the short answer is: It depends.
My product (the Sophos Email Appliance) is developed using agile methods. Industrial Extreme Programming, as espoused by Joshua Kerievsky, was used for the first several years of development. Recently I have started to move the team more towards Kanban, visualizing work flow and using pull-based scheduling instead of time-boxed iterations.
In other words, before I listen further to all the preaching and jump into the manuals, what evidence do I get that this is the way to be successful in a successful project in a successful company?
There is empirical evidence that most IT projects are not successful (where success means on time, on budget and fully functional here). Given this evidence, it seems reasonable to wonder if a deterministic approach (the waterfall) is well suited for software projects.
"The definition of insanity is doing the same thing over and over and expecting different results." --Albert EinsteinRita Mae Brown1
If a deterministic process produces failures over and over, we are likely not applying the right process for software development projects and Agile methods are an alternative. The theory behind these methods is that most software projects are not deterministic, they are creative (like in art) and complex (as defined by Ralph Stacey) projects and we can't predict everything. So, instead of trying to predict everything and then fighting against change, we should use an adaptive process. And this is what Agile methods are about.
Now, using an Agile method will never guarantee systematic success (and someone claiming the inverse is a liar) but they'll give you better control over the risks. And, if your project has to fail, it will at least fail fast.
Update: 1 Actually, this quotation seems to be misattributed to Albert Einstein. The earliest known occurrence, and probable origin, cites to Rita Mae Brown.
I believe Doublefine just produced Brutal Legend using Scrum.
From what I understand StackOverflow is a successful website built with agile practices and TDD.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
I'm trying to figure out how to solve some challenges that my team is facing while attempting to apply agile. The one that is causing the biggest headache at the moment is due to the dual role nature of projects that come into the business.
Basically, we have a number of software that we produce and deploy for various markets. This software is planned and scoped in quarterly release cycles. At the same time we have large contracts come through that take anywhere from 1-3 months to complete. The problem comes from the fact that management wants to work on the incoming contracts first and foremost and all normally scheduled release work is brushed aside to get the next contract out the door.
We're trying to scope the releases to shorter than 3 months so that a contract would have to wait that long for work to begin.
Has anyone dealt with a scenario like this while trying to apply agile? What are some ideas/approaches to working on release scoped/planned work and keeping management happy that high priority contracts are being delivered in a timely manner?
The only way I can see is an internal market.
Assign a $ value to the next release of your 'real' product and then you can fairly apportion effort to that vs the incoming contract.
Of course the value of the 'real' product depends on management but at least it pushes the problem onto them in a rational way.
Instead of looking at your situation as multiple shorter projects that are interleaved into one longer project, you might instead think of it as a single larger project. The small projects then become interrupts or the equivalent of scope changes, which are things that all large projects need learn to manage anyway.
As with interrupts and scope changes, you will need to address schedule impacts, the effect of "context switching" overhead on your staff, etc -- and possibly consider dropping features or cutting back in other ways in order to make your next scheduled delivery date.
If management wants the new work to be done first, while the mainline project is put on hold, then it seems to me that's what you should give them. Why drag your feet for 30 or 45 days before starting the new project? From the perspective of a single larger project, that's certainly not very agile. You could instead get a faster start, and then communicate the resulting impact.
Over the long term, you may find certain staff members are slowed down more by periodic course changes than others. In those cases, you might consider making semi-permanent assignments, so they can continue what they were working on, even in the event of interrupts. Similar arrangements are typical in larger, interrupt-driven shops.
Even in an agile workplace there is some kind of "resource planning" at management level. As long as there is some predictability on when the contracts come in, the allocation of people to teams and between teams can be decided before the start of each iteration.
If an unexpected event occurs and it is necessary to terminate a sprint, or re-plan it mid-iteration then that's what you have to do.
Agile methodologies are supposed to help you to "embrace change" and make sure that the highest value requirements are delivered first. They don't change the fact that there is always more work to do than people to do it, but they do provide a framework for managing the chaos that this will cause if people aren't realistic about priorities, actual staffing levels and work rates (or "velocity").
Agile doesn't mean that there won't be difficult conversations, but if it's done well then the conversations should mostly happen in time to take some kind of corrective action.
I'm assuming that there is some kind of officially sanctioned agile process in place. I don't believe that agile methodologies (e.g. Scrum) can be made to work under the radar, because:
Agile methodologies are about self-managing teams. If your management doesn't accept the teams' right to self-manage then there is going to be a power struggle.
Agile methodologies are about having high quality communication both within a team and between the team and it's stakeholders. It doesn't work if the process is hidden from the stakeholders.
From the comments above, your process seems to be in pretty good shape. You have identified a genuine business problem and you are having a constructive dialog with your management team.
If you haven't got management buy-in, doing Agile is very difficult.
By the sounds of it, the management don't have a problem at the moment. They drop a contract on you, you do it, the quarterly release slips but they get the nice contract money.
Is your team large enough that you could conceivably split it into two teams: one focused on the internal releases, one focused on contract work or perhaps two teams that swap over responsibilities after each release so they each get to spend some time on green field and some time on BAU projects.
On a general Agile methodology note, you'd be better off with Kanban than Scrum because it sounds like if you tried to plan iterations, you'd end up with 90% of the work in the unexpected 'contract came in' column.
Who is driving the quarterly releases on your product: customer requests or what you would like to do? Like mgb said, what profit is the business making from them?
I think it's first important to define by what you mean to "apply agile." There are a lot of different parts of agile and I would try and start small with the pieces that you can do. For example, do you have a continuous build running? Have you developed a product backlog?
Working on multiple projects is difficult to begin with but without management buy-in (as Wysawyg mentioned), it will be difficult to become more agile. You need to show the benefits of agile development in terms of cost savings to management. Have you determined why you want to become agile? How will it help? Once you have shown why, then just start doing some of the pieces that you can and after you start seeing some improvements, talk to management about the bigger pieces.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
I know that for me I first got started following the waterfall method of project management and along with that I went with the predictive approach to software design. In this I mean we had huge packets of documentation, UML, database schemas, data dictionaries, workflows, activity diagrams, etc.
Having worked in software for over 10 years now I find it to be much more realistic to approach software design from a Reactive approach. I frequently follow a scrum approach to project management and with that very little heavy documentation is ever generated. We have very little workflow specification (though they still have there use). This is a much more dynamic approach to software creation. Of course along with it comes frequent refactoring as time goes on as we find out new features over time that had we planned for up front would have changed things dramatically.
The big difference for us is that the first approach takes longer, seems to fail more frequently in a software construction world, and isn't nearly as flexible. The second approach provides more flexibility, makes us aware of failure faster (so we can course correct faster), and provides some form of functionality at the end of every iteration.
Knowing both sides from experience, I still find many people that LOVE the waterfall approach over the agile approach for software development. I don't get it.
question: Why would someone use waterfall over some form of agile with all of the research backing agile? What are strong arguments for using waterfall over agile?
When I started programming (with COBOL no less), waterfall was the "new" approach. Today, I'd tell you that I use a waterfallish agile methodology. For larger systems, I find a waterfall type start works best. Not creating huge documents (that's a waste of time IMO) but rather to take some steps like creating a UI prototype and/or use cases to get our heads around the business problem at hand. Once we are comfortable we have the problem scoped and we have a solid understanding, we move into an agile development mode.
To answer your question though, I think the big reason waterfall sticks around is many people don't like change. It's scary to change and moving from waterfall to agile is a big change.
I think that part of the reason why people still often cling to waterfall is that it gives the illusion of control. In a waterfall, you can do enough up front work to put together a beautiful schedule that nicely addresses every contingency that you can think of, and then give a detailed roadmap for the future to anyone on the business side who asks when feature X will be available.
The problem is that you can almost never follow that plan to the letter, and you are almost always late/dropping features. However from the upfront, it looks very controlled and manageable.
I'm a big Agile fan, but what I've always struggled with is the long range roadmap/forecasting that is often asked for by the sales and marketing folks. I think that the waterfall's illusion of certainty is very comforting to managers and business folks.
My boss tells me to.
I suspect many people have no choice and old bosses don't learn new tricks.
Not taking sides, but pretty much any research would be unscientific at best.
You say (emphasis is mine)
question: Why would someone use waterfall over some form of agile with all of the research backing agile? What are strong arguments for using waterfall over agile?
but don't link to any studies.
It's one of those things that are known to be extremely difficult to actually test. You can't have two identical teams work on the same project at the same time, because there's no such thing as two identical teams. You can't have the same team complete the same task twice in a row using two different methodologies without the first pass tainting the second. I've never heard of anyone designing an experimental (or even statistical) study that can convincingly argue for any software development methodology. I'd be interested to see one though, if you have a link.
Short of real evidence, it boils down to personal preference. What are the strong arguments for chocolate over vanilla?
I'll play devil's advocate and state that agile is flawed is nearly as many ways as the waterfall method is. I'm not one of those that love the waterfall method, but I don't love agile either.
My experience with agile hasn't been very positive. To be fair, I used it in a corporate environment, which paid lip service to "agile" while still expecting our manager to produce long term milestones and deliverables upfront.
However, I found that agile (scrum in particular) methodologies often disguise major problems with design. While waterfall gives managers the illusion of control, agile seems to do the same for development teams. I've seen teams where bringing up any issue that aren't in the current sprint/iteraton is frowned upon, with the expectation that it'll be handled "in time". It only requires a few major design decisions to be ignored for the project to go belly up in future, while current iterations go smoothly and project looks to be on track.
You can argue that the team's at fault for not understanding the spirit of agile, but I'd like to see better methodologies that incorporate the best parts of agile.
One of the premises of (at least) XP is that change is cheap. The waterfall model was built on the principles that change, any change, is costly. The assumption in the waterfall model is that once software has been written, changing it is more expensive than investing the time up front to come to a "complete" understanding of the problem.
Experience seems to indicate that it is very hard to come to a complete understanding of the problem and that if some precautions are taken (e.g. Unit Testing) change can become a lot cheaper. Therefore if you encounter a problem where some of the agile premises don't hold true other approaches might become feasible again. In between Waterfall and Agile there is at least Spiral development which is - sort of - what we practice.
You need to be preditive enough to deliver the goods. You need to be reactive enough to deal with the issues.
I was once stuck with six months to complete a project estimated to take a year, and based on past experience experience would take two. So I spent three months researching methodolgies. We finished on time (in three months), using the appropriate parts of a waterfall process.
A few points that made the methodoly work:
- Create an use standards, update them when needed.
- Build libraries: Do it once, do it well, fix it without breaking existing code.
- Do just enough documentation.
- Version control everything you can.
- Break things down; a method should either manage work or do work.
- Increase cohesion, decrease coupling, reuse.
- Buy or build the tools you need.
- Track your issues and progress.
Another project I was breifly involved was a six month project. I didn't get involved until a year and a half after it started. The development lead had been hired at an extreme markup as he was leaving a career with a pension plan. At the start of the project he asked the project manager, "Do you want me to do it right or be reactive?" Can you guess the answer? The week I was involved same feature was implemented on Monday, Wednesday, and Friday. Guess what happened Tuesday and Thursday?
The strength in Agile is its emphasis on just enough, just in time. The strength in the waterfall methodoly is that it covers all the things you need to think about. I've yet to work on a project that did or should have done all the steps. I have worked on many projects that did steps which should have been done on a corporate basis.
The title says it all. (Actually: proactive vs reactive). Why chose the reactive way and give up control unless you don't have to? Waterfall is not the only alternative, you can have any kind of development process what you refine when you like. Control is the key.
It's a spectrum btw, the waterfall on one end and the totally reactive, zero documentation methods on the other end. If you work in the consultant industry for powerful (and usually indecisive) clients, you have to resort to reactive methods. If you develop shrinkwrap software you can plan ahead and manage knowledge. Some projects also require tons of specifications and rules, where the code and fix approach just don't cut it. For me software engineering is primarily about knowledge management and design, coding comes second.
P.s. there is no such a thing as agile and fixed price. Not in the classical way they usually sell the method. See http://martinfowler.com/bliki/FixedPrice.html
If you exactly know the requirements that never chagen, if you know how long each step will take and if you know all the resources are avaliable at the time needed you can do waterfall and it will work. But in deed these kind of projects are quite rare and I think I will never be part of it.
When designing systems to be used by end users, agile often works well because the requirements are likely to be incorrect and a large part of the process is getting feedback from users in the form of a usable version.
However, when creating software that interfaces with other software often the requirements can be worked out in very clearly. In this case it is often more productive to ensure that you have a very clear and accurate specification, unit tests In this model you can also generate fairly good work estimates and there would be a great deal more cost to use the agile model.
retroactive behaviour
If you have a team of a few dozen people that have over the course of a decade, refined the waterfall strategy to the point that it works well for them, who are you to come in and say, "You're doing it wrong..."? Really, if it is working for them, why change things? Yes this is merely flipping the question around but I think it may be a valid point.
In my team we've found that with maintenance projects (which is the bulk of what we do) where we're tweaking or replacing like with like there isn't always as much need for user input beyond perhaps some UI prototypes.
In that case, particularly given that there are commercial deals involved the waterfall approach at a macro level can fit well. Even then we still like incremental / agile approaches at the implementation level.
It is worth noting that most of our clients are large lumbering organisations in love with their paperwork, so that adds even more impetus for us to at least appear traditional to them.
The documentation generated during the waterfall process allows for a lot of CYA. You can point fingers when a project goes off the rails. Very few executives are going to be OK with "oh well, I guess that project got away from us! Well, at least we found out early, no harm no foul!"
Also, design docs can automatically generate test plans, which is useful for QA.
It's pretty common when bidding for a contract that one of the iron-clad conditions is that you follow their "process" which on inspection is waterfall.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
For those of you who have implemented Scrum in your organizations, what were your biggest obstacles and if you did overcome them, how?
Background: In 2006 I contracted with a large company which had adopted Scrum cold turkey just months before I arrived. The company hoped Agile/Scrum would save their huge enterprise software product. Of the hundred or more programmers there, I worked closely with a team of about a dozen for a year, observing and participating in their Agile experiment.
Summary: I believe Agile helped more than it hurt. By the end of the year, the team could consistently estimate and produce features, whereas previously their productivity was rather erratic.
Implementation: Since this was a large organization and a large product, the project ran as a "scrum of scrums." There was one scrum master for about every 15-20 developers and these teams were often divided into smaller, closely working scrums of about 6-8 people for an iteration. Teams were largely independent, could adjust their own iteration frequency (1 month down to 1 week) and were given lots of flexibility to implement agile as they saw best. The company regularly brought in Agile coaches (such as Object Mentor) to help train the scrum masters, teams, and management.
Obstacles: Plenty. Some of them related to Agile, some not. In no particular order, here are some lessons learned:
The product backlog was revised way too often in the beginning. Eventually, the team and management took several days to go over all the features, estimate them, and prioritize them. It was a big hit, but it helped tremendously. Lesson learned: get your product backlog in order early and keep it maintained. Product owners must have a clear idea of what they want.
We lost time experimenting and dealing with fads and hype. When you start, you have no way of knowing if you're doing things correctly. There's temptation to constantly fiddle with the agile process taking the focus away from the product. Lesson learned: having an experienced Agile coach does help reduce this learning curve. There should always be someone pushing back on any experimentation. Limit the number of "spikes".
A good scrum master is invaluable. Certainly in the beginning, it's a full-time position.
It takes time. It took several months before the team started to be comfortable with the process.
Pick your battles. Some programmers will be understandably skeptical and others will outright dislike and flight the change. Allow for some flexibility. For example, enforce the use of a product backlog and iteration schedule, but don't require everyone use note cards. Be particularly sensitive to introducing tools and programming methodologies such as pair programming or test first development.
Finally, keep communication open and manage expectations.
Good luck!
While working as a Delphi developer a few years ago, I managed to get Scrum adopted by my development team for a time.
The whole process worked very well for us - having the team estimate prioritized tasks on a backlog gave us meaningful timeframes to target, and the whole "Managements job is to remove impediments" was great.
The biggest problem was that the process was always perceived - and referred to - as "Bevan's good idea".
While the team appreciated the value we gained, and were happy to continue with Scrum, the Team didn't take the scrum methodology on board as their own. After a while, I got tired of "pushing" and we "fell out" of following the Scrum approach.
Lesson: Make sure the team takes Scrum on board and owns the approach.
We do mostly scrum projects at the customer site. Hardest part in my experience is finding a good product owner in the customer organization:
Too many people think they should be the product owner,
The product owner has a hard time following the pace of the team
Product owner has a hard time getting all the detailed information the team needs
Moving items down the product backlog to add something with a higher priority is difficult
etc.
Training internal teams to use scrums is doable, bringing in your own scrum master is doable, but a good product owner should be part of the client organization. It's harder to train this external person.
Having a proxy product owner, who works together with the customer product owner does help a lot.
I moved from a company that adopted Agile to the tee to another company which follows the traditional methodologies.
Perhaps the biggest difference I have seen is that the second company struggles to prioritize. There is so much work on each person's plate that they fail to deliver on time. IMO, Agile brings about some transparency to the situation and lets the team as a whole prioritize.
A scrum master in the Agile world would take care of fire-fighting and be the voice of the (sprint) team. In fact, in the first company (where we had a separate scrum master and program manager), the scrum master would fight it out with the program manager when the latter makes false promises to the management. Meaning, the scrum master knows how much a team can produce/deliver after a few sprints, which helps her nail down on the predictability of a team.
I also noticed that the R&D resources have a sense of accomplishment at the end of each cycle, and are looking forward to the next one. But then, a good project manager could get this done in traditional scenarios as well.
The biggest issue, as already stated, that I too have experienced is the lack of buy in. It is very difficult to get people to truly become vested in the process.
The other issue, which is also one that directly contributes to the above issue, and also in a large part one of the founding causes of Agile is the lack of management to stick to the outlines of the manifesto of Agile.
In Scrum, Lean, or whatever version of Agile you are working with one cannot break from the manifesto points. If a process is being used to break away from those priorities then most likely the management is screwing up and the buy in will fall apart. The manifesto MUST be followed:
Manifesto:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
Some scenarios might be when a gantt chart appears from one of the above processes for whatever reason. Gantt charts can be useful, but if all of a sudden a developer is reviewing a gantt chart with management, the last point is broken. Responding to change has slowed because encouragement of the plan is being favored over change. Instead a board with stickies should be used, simplify what is on the board with only the current working items and back burner items. This makes changes easy. Once anything is solidified in a "tool" it slows responding to changes. Sure, management needs to record and track things in some ways, but pushing that onto development only slows the responding to change, and pushing tools onto developers (unless they want them for development and can utilize them appropriately) messes up the first point, of the individuals over tools and process.
In another way, don't stop development for the purpose of writing comprehensive documentation. Unless you only have a single developer, then someone should take the documentation load autonomously from the development role. Pushing these things together drastically slows development and for periods of time, can shut down any effort to actually get working software.
The last point, is to always, ALWAYS stay in contact in some way with the customer or prospective customer. Talk to them regularly about what they want. Talk to them daily and show them as much as you can of UI, or even data flow work. Anything that they would understand they should see. Talk to them, educate them about the architecture and ideas going into the application and never forget that you are building the application for them.
Summary:
Biggest issue is buy in. Second is management sticking to the manifesto guidelines.
If you can mitigate these two risks, you should be good to go. Anything else is cakewalk after getting buy in and getting management to understand that they'll need to be truly strong, and non-micro management managers. Specifically, managers might even need to become leads, or fill a different style of role.
...hope I didn't stray off point too much. :)
I have been running Scrum in several projects. The biggest problem, as I see it, is that not everybody in the organization is into the process. Everybody needs to be committed. Not only the team of developers. Often the managers are the persons that initialize the process and expect that things will change to the better without them doing anything.
My suggestion is that you run a workshop with the whole organization so everybody knows how the process works. Not only the developers. It's essential that you have a person that is really into the process. A person that can answer questions the team and organization have. A mentor.
Being agile is about welcoming change. You should not let the process gets in the way of sense. Do things that works for your organization, but you should try out the whole process before throwing something out.
We implemented Agile (set of SCRUM - management and XP - engineering practice) in an environment that was waterfall with large projects in an environment that was heavily integrated. The waterfall police were everywhere. As you can imagine, many projects failed. Having done Agile at a previous employer, we received permission to trial agile for the project.
Internal to the team, we used the Agile practice. Externally, we wrapped the agile practices with waterfall processes meaning primarily reporting. Thus, we looked from the outside like a waterfall project. However, there was a big difference, internally we were using agile and consequently we delivered, on time, within budget with high quality.
The critical success factors were embedded coaches (Iteration Manager Coach, Dev Lead Coach, Test Lead Coach and a Solution Analyst Coach). Securing commitment from dependent system in advance (required that we look ahead to identify depend systems and the work required from those systems) was a must in a heavily integrated environment. Prior to starting, we immersed the technical and business members of the team in an agile boot camp. This ensured that the key players (product owner and technical team) knew there roles and could execute effectively. Finally, the wrapping of the project with waterfall reporting enabled us to tie into all the existing reporting structure in the enterprise.
The net result is that the company is now moving waterfall projects to agile. This is all possible only because we have been able to deliver high quality software at a sustainable pace.
Where I work has been using Scrum for a while now but it seems to have gone through a few phases. In terms of obstacles, one part is to prevent putting in too much change at once and just introduce things slowly,e.g. put in a daily standup one week, a couple weeks later put in a story board, a couple weeks later bring in pair programming. This allows for the various tweaks that will happen to work and if the changes improve things then this can help build up some good momentum. Another point is to make sure that if there should be changes in how something is done that the person being corrected isn't belittled or mocked. At times this may mean that you interrupt someone or that you bring in a "Can we get back to basics?" or something similar to try to put things back on track rather than just yelling at someone or doing something else that is counterproductive.
Bringing in consultants was one of the best things done around here, IMO. Now, these guys came in to help evolve how development was done here. Bringing in pair programming, TDD, concepts like broken windows, organizing project folders, and bringing in mocking for tests, were all excellent additions that while we may have gotten there on our own, it may have taken a long time which wouldn't work out so well.