There’s much positive to be said about the benefits of Agile software development, and the shift of software development teams is one sign that many feel this way.
However, I think there are some limits to Agile, and this leads teams to fall into certain traps over and over. Indeed, the Wikipedia page highlights a common criticism of Agile, namely:
Lack of overall product design
A goal of agile software development is to focus more on producing working software and less on documentation. This is in contrast to waterfall models where the process is often highly controlled and minor changes to the system require significant revision of supporting documentation. However, this does not justify completely doing without any analysis or design at all. Failure to pay attention to design can cause a team to proceed rapidly at first but then to have significant rework required as they attempt to scale up the system. One of the key features of agile software development is that it is iterative. When done correctly design emerges as the system is developed and commonalities and opportunities for re-use are discovered.
Now, it’s not the case that teams either do design or not. But I have seen that there are a number of specific traps bigger that Agile teams fall into that arise from lack of design. These traps arise from making tactical or limited decisions outside of a larger framework or structure, which isn’t surprising since Agile followers are guided by the principles that the “best architectures, requirements, and designs emerge from self-organizing teams” and “working software is the primary measure of progress”. Unfortunately what I’ve seen that lead to is:
- Poor middleware/database system decisions: with this trap, you get teams making a decision on deploying a specific middleware package or database system that will support the software development they are doing. However, while that may be great for the Dev, it may not be great for the Ops. If you have enough teams making tactical decisions, you may end up with a more complex system than is necessary and greater technical debt that you want. Once you get enough data into your database systems, trying to reverse the decision may result not result not only in new development and testing, but a small (or not so small) migration project as well.
- Poor infrastructure decisions: likewise, with this Agile trap I have seen teams using IaaS pick different images to deploy their software onto. Like the database problem, developers may choose one operating system (e.g. Windows) over another (e.g. Debian) because they are comfortable with the former, even if the production environment is more of the latter. The result can be your organization ending up with multiple VMs with many different operating systems to support and thereby increasing the operational costs of the system.
- Poor application framework decisions: I see less of this one, although it can happen where teams pick an assortment of application frameworks to use in creating their software, and as with middleware and infrastructure, this will drive up the support effort down the road.
Given these traps, I think the way to avoid them is to inject some specific design phases into the overall software development lifecycle. One way to do that is to revisit a software development lifecycle (see diagram below) used by practitioners at IBM and documented in places like this IBM redbook. It has a waterfall quality about it, but it doesn’t have to be limited to waterfall type projects. It can be highly iterative.
The lifecycle process is shown here (taken from the redbook):
The part of the lifecycle in the large box is iterative and not all that different from an agile sprint. But here you take time to explicitly make design / architecture decisions before building software. You continue to iterate while making important design decisions before building.
Now, before you start your iterative software development lifecycle, you should need to make specific architectural decisions. You should make these specific decisions in the solution outline and macro design phase. For smaller teams, these two phases may blend into one. But it is here in solution outline and macro design where you make decisions that are fundamental to the overall solution.
For example, in solution outline you could make overall architectural decisions about the application architecture, the choice of infrastructure technology, what application frameworks are the target for the team. These overall architectural decisions guide the dev, test and ops teams in the future. You may also decide to park some of these decisions in order to do further discovery. Macro design could be next, where each of the dev teams make certain design decisions about their technology choices before they proceed with their iterations. As they are building and deploying code, they can run into issues where they need to make further design decisions, either due to problems that arise or technology choices that have to finally be made: and this is where the micro design phase is useful. Micro design decisions could be quickly made, or they may require spikes and proof of concepts before proceeding. Or there could be no decisions to be made at all. The main thing is more design checkpoints are built into the development lifecycle, which can result in less complexity, less maintainability costs, and less technical debt down the road. What you lose in velocity you can make up in overall lower TCO (total cost of ownership).
There is a risks to this type of approach as well. For example, if the project gets hung up with trying to make design decisions instead of gather requirements and making working software. The key stakeholders need to be aware of this and push on the design teams to prevent that from happening. If anything, it can help the key stakeholders better understand the risks before getting too far down the road in terms of developing software. Overall I think the risks are worth it if it helps you avoid these common agile traps.
Smart People I Know 