Requirements Envisioning: An Agile Core Practice

1. When should you do initial agile requirements envisioning?
2. Why should you do some initial agile requirements envisioning?
3. What should you model initially?
4. What modeling tools should you use?
5. How much initial requirements envisioning do you actually need to do?
6. Why don’t you want the details up front?
7. When does it make sense to do a lot of requirements envisioning?
8. Are people actually doing this?

1. When Should You Do Initial Agile Requirements Modeling?

Agile Model Driven Development (AMDD), see Figure 1, explicitly includes an initial requirements envisioning effort during Iteration 0 of an agile initiative (what some processes might call the Warm-UP, Inception phase, or Initiation phase). Initial requirements envisioning is particularly important for scaling agile software development techniques to larger, more complex, or globally distributed development (GDD) efforts. The goal is to understand the requirements at a high-level, it isn’t to create a detailed requirements specification early in the lifecycle, a traditional practice referred to as “Big Requirements Up Front” (BRUF) which proves to be very risky in practice (traditional theory is that BRUF is a “best practice “, but experience shows otherwise).

Figure 1. The AMDD lifecycle (click to enlarge).

2. Why Should You Do Some Initial Agile Requirements Envisioning?

Some people will tell you that you don’t need to do any initial requirements modeling at all. However, my experience is that doing some initial requirements modeling in an agile manner offers several benefits:

1. You can answer fundamental business questions.Like it or not, people are always going to ask you what the vision is for you’re going to build (what’s the scope), how long do you think it’s going to take (the schedule), and how much is it going to cost (the expected budget). You often don’t need to answer these questions in detail but you will need to convince the people who are funding and supporting your team that you understand the fundamental business issues that your team is going to address.
2. Improved productivity. You can identify and think through some of the critical business issues facing your initiative.
3. Reduced business risk. Your team gains the advantage of having a guiding business vision without the disadvantages associated with BRUF. In the Unified Process the primary goal of the Inception Phase is to drive to scope concurrence with your stakeholders, thereby reducing business risk, and the way that you do this is through initial requirements envisioning. Not only do you identify the scope of the effort, you also help to make the relevant stakeholders aware that there is a range of needs being met by your system, not just theirs, and that they are going to have to compromise sometimes.
4. Scaling agile software development. Your initial requirements model will be a key work product in any “agile at scale” efforts because it provides the business direction required by your overall architecture team for their initial architectural envisioning efforts (typically done in parallel with initial requirements modeling) and by sub-teams to define and guide their efforts within the overall initiative.

3. What Should You Model Initially?

For the first release of a system you need to take several days to identify some high-level requirements as well as the scope of the release (what you think the system should do). The goal is to get a good gut feel what the initiative is all about. For your initial requirements model my experience is that you need some form of:

• Usage model
• Domain model
• User interface model(s)

3.1 Usage Model

Usage models enable you to explore how users will work with your system, which is absolutely critical if you’re to understand what they need. Your usage model may be a collection of features for a Feature Driven Development (FDD) team, or a collection of user stories for an Scrum team. I generally prefer either user stories or use cases although my actual choice in practice will be driven by the software process (XP, FDD, …) the team is following.

At the beginning of an initiative a list of use case names might be sufficient although usually you’ll need to go into a bit more detail. Figure 2 depicts the level of detail that I would consider going to for a high-level use case at this time. It indicates the name, in this case Enroll in Seminar, and the basic logic which the use case covers. I’d likely write this sort of information on an index card or sticky note (as Figure 7 shows, paper-based modeling like this is common on Agile teams). I might even decide to capture it in an electronic tool such as a word processor or spreadsheet, although at this point in time such a tool would likely hamper my modeling efforts more than they would help.

Figure 2. Enroll in seminar as a high-level use case.

An interesting thing to note about the use case of Figure 2 is that it doesn’t capture a lot of detail. For example, it only has a high-level overview of the logic of when things work out well (the basic course of action) but it doesn’t capture alternate courses such as what happens when the student isn’t eligible to enroll, or there isn’t room for them in the seminar, or it doesn’t fit into the schedule, or they can’t pay the fees, and so on. As Figure 1 indicates, the details can be captured on a just-in-time (JIT) basis via model storming. Right now the goal is to gain a high-level understanding of the scope, not write documentation, the the level where we’re comfortable with our understanding of what needs to be built.

Corresponding user stories to the use case of Figure 2 would be:

• As a student I search for seminars so I may enroll them
• As a student I enroll in a seminar so that I may take it
• As a student I pay fees so I may be enrolled at school

Although they’re relatively useless for developers, you’ll often find that you’ll want to create a UML use case diagram to overview the usage requirements for what you’re build. Figure 3 presents an example of such a diagram drawn on a whiteboard, although you’ll often find that you need to use an electronic drawing tool such as Visio or even a CASE tool such as Rational System Architect (RSA) to “pretty up” your diagram for presentations to senior management and important stakeholders such as the “gold owner” who is paying for the initiative. Having said that, I’ve often used digital photos of whiteboard sketches, such as the one in Figure 3, in management presentations. They’re adults, they understand that you have more important things to do (such as build working software) instead of transcribing information into more complex tools.

Figure 3. Whiteboard sketch of a UML Use-Case Diagram.

3.2 Domain Model

Part of your initial modeling efforts, particularly for a business application, will likely include the development of a conceptual domain model. This model should be very slim, capturing the main business entities and the relationships between them. As you see in Figure 4, your model doesn’t need to be complete, it just needs to cover enough information to make you comfortable with the primary business concepts. I would draw a model such as this on a whiteboard to start out with, and would likely keep it on the whiteboard throughout the lifecycle. My experience is that a slim domain model such as this is a valuable asset to the team, one that should be very easy to view (you should simply have to look up from your desk and view the shared whiteboards in your work room), to create (whiteboards are very inclusive), and to modify.

Figure 4. The initial domain model.

A domain model identifies fundamental business entity types and the relationships between then. Domain models may be depicted as a collection of Class Responsibility Collaborator (CRC) cards, a slim UML class diagram, or even a slim data model . At this point in time your domain model will contain just enough information to understand “the landscape” of your business entities. Figure 4applies a UML-based data modeling notation (you can use any notation that you like when agile data modeling, I prefer UML). Your initial domain model will be used to help guide both the physical data model as well as the class design.

3.3 User Interface Model(s)

The user interface (UI) is the portion of software with which a user directly interacts. For most stakeholders the UI is the system so it is crucial to the initial success of your team to ensure that you understand what stakeholders expect of the UI. One way to explore the requirements for the UI is to create what are called essential UI prototypes, also known as an abstract prototypes or paper prototypes. Essential UI prototypes represent user interface requirements in a technology independent manner, just as essential use case models do for behavioral requirements. Figure 5 depicts an essential UI prototype to enroll students in seminars. The Student name sticky note is a container that includes four active elements: First name, Surname, Middle, and Title . The other sticky note represents a list of the seminars a student has taken or is currently enrolled in. Notice how each sticky note has a name that describes its purpose, but not how it is implemented. You can look at the sticky note and immediately know how it is used.

Figure 5. Essential UI prototype for enrolling in seminars.

Although essential UI prototyping is a great technique, many stakeholders find them too abstract and instead prefer screen sketches and not surprisingly working software. Figure 6 represents the screen sketch version of Figure 5. In this case there are two screens instead of one for cohesion issues – I prefer screens that fulfill a single purpose, in this case capturing basic student information and enrolling a student in seminars respectively. This is arguably a design issue (there’s a fine line between requirements/analysis and design which you’ll cross all the time). The sketches provide a final level of detail than the paper prototypes do, it is much easier to get a feel for how the screen will be implemented from than it is from Figure 5, although the sketch isn’t as flexible because it’s hard to shift widgets from one part of the diagram to another whereas with the paper it is very easy.

Figure 6. Screen sketches.

A related model is a UI flow diagram, often called UI storyboards, as you see in Figure 7. However, UI flow diagramming is arguably a UI architecture issue which is addressed by your parallel initial architecture envisioning efforts. The bottom line is that the categorization of the model doesn’t matter, what’s important is that you do it if and when it makes sense.Figure 7. A UI flow diagram.

Some teams may even require the development of a concrete user interface prototype, as you see in Figure 8. You’ll often do this for at least for a critical subset of the UI, to explore usability issues at the beginning of the initiative, but not for the entire system (that would be BRUF).
Figure 8. Concrete UI prototype (HTML page).

4. What Modeling Tools Should You Use?

Agile modelers will use the simplest tool which will get the job done. As you can see in Figure 3 and Figure 6 I’ve found whiteboard diagrams typically suffice for initial requirements models. Your goal is to think through the critical business issues at this point in the lifecycle, not create pretty diagrams. There’s nothing wrong with using more sophisticated drawing tools, as you can see in Figure 4, just ensure that the investment you’re putting into the effort is worth it. It all depends on the situation that you’re working in. Furthermore, paper-based tools are also quite common. For example I would be tempted to initially capture the use case of Figure 2 on an index card initially and only later capture the information using an electronic tool if appropriate.

5. How Much Initial Requirements Envisioning Should You Do?

The quick answer is a bit more than what the “extreme agilists” will tell you but a lot less than what the professional modelers believe. Remember, at this point your goal is the understand the scope and vision, not to document in detail people’s speculations about what they want. It’s important to recognize that you don’t need the details, you just need enough information to answer the business questions that you’re being asked. If a list of bullet points is good enough for you to make a reasonable guess as to the relative effort of implementing a use case then only capture that information for now. The details, if you ever need them, can be gathered on a just in time (JIT) basis via model storming during the initiative or during iteration modeling. Trying to capture the details behind requirements early on is called ” Big Requirements Up Front (BRUF) “. BRUF can seem like a good idea in theory, but in practice proves to be quite risky because of the high-levels of wastage which it motivates.

There are several potential end products of your initial requirements modeling efforts:

1. Initial requirements stack. The initial “prioritized requirements stack” that Agilists has to come from somewhere, and that somewhere is your initial requirements modeling effort.
2. Initial vision. To reduce your overall business risk you should consider developing an initial vision. A good vision can be as simple as a few bullet points or a paragraph and shouldn’t be any more than a page. Traditional organizations, particularly those that made the mistake of instantiating RUP in a heavy manner, or the advice of the PMI in a heavy manner, will often invest far too much time in vision statements. A good rule of thumb is that if your vision is longer than a single page you’ve put far too much effort into writing it.
3. Overview diagrams. Because you’ll likely need to give presentations to key stakeholders overviewing the initiative you’ll likely want to create a couple of scope diagrams which describe the business. UML use case diagrams or traditional business process models are usually pretty good at this. If you spend more than a couple of hours sketching these diagrams on a whiteboard and then transcribing them into a tool to pretty them up you’ve gone too far.

Figure 9 depicts the value of modeling, showing that models reach their point of maximal value when they are barely good enough, and for the vast majority of initiatives when it comes to initial requirements envisioning this is typically after several days or a week or so. Many people are initially shocked by this claim, but when you think back to all the teams that you’ve ever been on, how many of them, if you were allowed to, could you have identified the initial scope within a week or two by putting a few people who understood the domain together in a room with a lot of whiteboards? I’ve asked this question of several thousand people now and only been told about a handful of initiatives where this wasn’t true. Many people had stories around not being able to get people to agree, an indication to me that you should either rethink doing the initiative completely or attempt something with a smaller scope for now, not being able to get access to stakeholders (a logistical problem that you could choose to overcome if the effort was actually important enough), or they simply didn’t know what they wanted (clearly an indication to not do it right now). Those were all excuses, but they weren’t good reasons to increase your risk by doing too much initial modeling.

Figure 9. The value of modeling.

6. Why You Don’t Need, Nor Want, Details Up Front

There are several reasons why eliciting detailed requirements at the beginning of a development initiativs proves to be an incredibly poor idea in practice:

1. It results in significant wastage. It’s been shown that when teams are actually able to deliver software into production when taking a BRUF approach that on average 45% of the functionality delivered is never used. 45% is an incredible amount of wastage in my books. Sadly, a BRUF approach is often taken to obtain the information required for an “accurate” estimate at the beginning of the initiative, something the financial people require in a misguided attempt to reduce financial risk. Considering the resulting wastage, it’s pretty clear that this approach appears to increase the financial risk, not decrease it.
2. It decreases the chance that you’ll detect that you’re building the wrong thing. A detailed requirements specification can provide you with a false sense of security that you understand stakeholder needs. This in turn demotivates you from exploring and rethinking the requirements as the initiative progresses, reducing the likelihood that you’ll recognize and address any deviations from what your stakeholders actually want.
3. People aren’t good at defining up front what they want. This is something that we’ve known for decades, it motivated spiral/RAD movement of the 1980s as well as prototyping practices within the UI community.
4. It motivates poor decision making. The beginning of the initiative is when you know the least about the problem domain and your stakeholders know the least about the solution domain (you haven’t produced any working software yet to show them). Yet the traditional dogma is to put all of your limited and very likely to change understanding of the requirements down on paper so that you can start making important scheduling and budgeting decisions. Then, later in the initiative we’re surprised that the decisions that we made early on, particularly our estimates, prove to be rather poor. Remember the GIGO maxim: Garbage in, garbage out.
5. It increases communication risk. We’ve also known for decades, at least the psychology community has know, that providing documentation to someone is the poorest way to communicate information to them. There are significantly better strategies for people to communicate with one another, and in the case of requirements elicitation the AM practice of Active Stakeholder Participationwill dramatically reduce communication risk.
6. You can always document the requirements once they’ve stabilized. If there is a need to write requirements documentation it is more efficient to document as late in the lifecycle as possible once they’ve stabilized. This doesn’t necessarily imply that you do the documentation at the end, a requirements specification can and should be a living document that changes over time. Furthermore, most agilists prefer to capture detailed requirements information in customer acceptance tests, one form of executable specifications.

Many traditionalists will tell you that you need to model everything up front in detail. They mistakenly believe this for several reasons:

1. That’s what they know. Big modeling up front (BMUF) is what they’ve been trained to do, that’s what they’ve been doing throughout their entire IT careers, and they’ve been told that working otherwise is ineffective. The traditional dogma is to do BMUF, and when you’ve been subjected to this sort of dogma for several decades it takes several years to break.
2. They’re specialists. Many people have chosen to specialize in modeling activities and as a result that’s what they do because that’s their only viable options. Specialists have a tendency to not only perform their specialized activities (no surprise there) whether it’s needed or not they will also have what appears on the surface to be a coherent reason for why it’s an important thing to do. Unfortunately their reasoning usually focuses on the perceived benefits without addressing the actual risks.
3. Traditional project management ideas motivate poor modeling strategies. Traditionalists are often forced to create detailed, up front estimates and plans in a misguided attempt to reduce risk (this strategy inevitably increases risk because it motivates a host of questionable development activities).
4. They underestimate their own scheduling skills. Traditionalists are often convinced that they can’t get access to stakeholders throughout the initiative, a self-fulfilling prophecy because traditionalists have trained stakeholders over the years that they’re only needed during the requirements and user acceptance testing phases. In practice, gaining access to stakeholders often proves quite easy.
5. They’ve given up hope. Many traditionalists, perhaps the majority, don’t really believe that BRUF is a good idea but are forced into it by everyone else in the organization. It’s interesting that when you get all of the major decision makers together in a single room how quickly people will come to the conclusion that none of them really want to do BRUF but that they thought that everyone else needed to work this way.

7. When Does it Make Sense to do a lot of Requirements Envisioning?

There are several reasons why you my need to spend more time in requirements envisioning than what I described earlier. These reasons are:

1. You’re working in an unknown problem space. The problem space that you’re addressing may be completely new to your organization, and as a result you may need to do more initial requirements envisioning to better understand that space. In such situations you may find that you need to create a user interface prototype early in the agile lifecycle, an iterative effort which can take weeks and sometimes even months. However, recognize that your understanding of the problem space will still need to evolve throughout.
2. You’re working on a commercial product. It can be very difficult to obtain regular access to your end users, which increases the risk. One way to deal with that risk is to hold focus groups with potential customers early in the lifecycle to attempt to better understand their needs. Due to the expense of holding focus groups, remember that software development is an economic activity, you are motivated to hold as few as possible. This in turn motivates you to do a bit more requirements modeling up front than you normally would. However, that doesn’t mean that you only do this up front. A wise product owner/manager will keep in touch with key end users/customers (or potential customers) and get feedback from them throughout the initiative.
3. Your governance framework is serial. Many IT governance frameworks, although they claim to be process agnostic, include serial milestones which make it very difficult for software development teams to truly benefit from agile approaches. The good news is that it is possible to adopt a lean governance which enables agile approaches. The bad news is that the governance people often seem to have a command-and-control cultural mindset, making the collaborative approach promoted by lean governance to be an alien concept to them. Sadly, appropriate IT governance appears to be elusive in practice.
4. You’re doing systems engineering. On systems engineering initiatives the constraints of working with hardware, particularly when you’re also developing the hardware, motivate you to do more requirements exploration up front because if you get hardware requirements wrong it can be spectacularly expensive to fix the problem later in the lifecycle. I highly suggest that you read Real-Time Agility if you are working in this space.
5. Your contract demands it. Many IT initiatives involve outsourcing work to another organization. Although it is possible to take an agile approach to outsourcing, the unfortunate reality is that many contract procurement people still haven’t gotten their heads around agile and still assume a serial approach to development. The implication is that the contracts which they write often require a BRUF approach, even though the contract may also specify that the development team take an agile approach to development. The best advice that I can give in these situations is to do the best that you can and to work closely with your contracting people so that they understand the implications of what they’re putting in their contracts.
6. Your organizational culture promotes it. Many people still work in organizations with a serial culture, and one result of that is that development teams will be required to do far too much modeling up front. The solution is to help change the culture, something that is very easy to say but difficult to do.

Recognize that it is very rare to need to do significant amounts of requirements modeling early in the agile lifecycle, that these reasons are likely applicable less than 10% of the time. Also, remember that the risks associated with BRUF don’t go away in these situations — what you’re doing is accepting some risks to decrease others. IT is always a collection of trade-offs.

8. Are People Actually Doing This?

Yes! In my Agile Adoption Rate survey in March 2007 one of the questions that I asked was how effective initial agile requirements modeling was on agile teams. The results is summarized in the histogram of Figure 10. As you can see, Agilists do in fact model in practice. 78 percent of respondents who said that their organizations were doing Agile indicated that those teams were also doing initial high-level agile requirements modeling, and that 85% of those teams found the effort worthwhile (they gave it a rating of 3 or more out of 5). Similarly, the DDJ 2008 Modeling and Documentation survey found that sketching was the most common primary approach to modeling (see Figure 11). The 2013 Ambysoft Agile Project Initiation Survey, which surveyed the agile community directly, found that over 90% of respondents indicated that their agile teams did some sort of up front requirements modeling.

Figure 10. Adoption Rates of Modeling on Agile Teams.

Figure 11. Primary approaches to modeling.

In summary, do just enough initial high-level requirements modeling at the beginning of an Agile software development initiative. You don’t need a lot of detail, nor do you need a lot of documentation, but you do need an initial understanding of the scope. By building an initial vision of what you’re going to build, and by addressing fundamental business questions surrounding costs and benefits, you’ll quickly put your team on firm ground.