Strategic Challenges

In some ways, an agile transformation is like any other organization transformation: there are challenges pertaining to education, protecting early adopters, advocating and achieving consensus on many issues, coordination, and motivation and managing progress. In other ways, an agile transformation has its own unique kinds of challenges: As an IT transformation, it is deeply steeped on technical issues; and the place where the “rubber meets the road” – software development teams – is notoriously difficult to measure and manage. Thus, while agile transformation is challenged by all of the normal snags and trials of an org change, it is also hindered by the impenetrability of software engineering.

The most common areas of greatest challenge are described below.

Senior Executive Leadership

In Executive Leadership Required we explain how important it is for IT senior executive management - most prominently the CIO (or whoever leads the IT function) - to own and drive the transformation, playing an ongoing active role in planning and execution. Unfortunately, the reality if often one or more of these situations:

• The CIO does not participate sufficiently in discussions about transformation issues and transformation planning.
• IT senior leadership does not actively lend support to transformation efforts, such as (making sure that there are) cross-functional workshops and discussions about aspects of their function that involve planning and performance measurement - i.e. aligning incentives so that functions are motivated to cooperate.
• Executives do not understand org change or change leadership and therefore do not assume the required critical leadership roles.
• Some CIOs “leave it up to their staff.

The CIO must be willing to issue mandates and follow progress closely, and meet personally with those who lead the agile transformation. The transformation must be set up as a strategic program, under the CIO. Otherwise it will not have the authority that it needs to work directly with the CIO's staff and set up cross organizational teams.

Politics, Organization Structure and Incentives Internal to IT

As we discuss here, most challenges in an agile transformation are management level. This seems a paradox because agile is about how software teams operate - programmers. But it turns out that in a large organization, those teams rely on many enterprise level functions, and if those functions do not work in a way that is compatible with agile, the teams are stuck: in that situation, it is only management that can resolve the impediments, and that often requires a great deal of restructuring of functions, as well as learning new ways of working at both the management levels and the team levels. Hence, politics plays a large role in an agile transformation, because it always does whenever you contemplate any kind of organizational restructuring, or any kind of redefinition of how things should work between departments. When you enter that realm, you enter the realm of private agenda, distrust, and power politics.

Politics External to IT

Strategic system development initiatives that are important to parts of the business can override IT process improvement initiatives that are underway. In order for IT to transform how it delivers, it needs to first slow things down while teams learn new skills and new management level systems are developed. Large business initiatives are usually unwilling to slow down, even for a limited period.

Large business initiatives also often have complex governance groups that oversee requirements and planning, and such groups are slow moving and inaccessible and therefore present challenges for agile methods, which rely on easy access and rapid response.

On top of these challenges, business programs often negotiate COTS technology components when the program is established, and such technology choices often present tremendous challenges for agile processes because agile processes rely heavily on automated and continuous testing, and COTS components are very difficult to test in this manner. Any agile transformation that is underway in the presence of strategic business system initiatives is therefore highly vulnerable, and managing this situation can be highly political for the CIO.

Relationship Between IT and Business Functions; Agile Program Management

Large business initiatives, or even small programs that have business autonomy in how they engage IT, often initiate projects in a non-agile manner by having a lengthy requirements effort performed, and then later ask that the project be "done agile". Agile methods require that requirements are addressed incrementally as development is done, and performing a requirements effort up front undermines the agile process. In order to head off up front requirements efforts, IT must work with the various business areas that have authority to initiate projects and get them to understand the buy into the agile process. If the organization has a portfolio function, this can often be addressed through that function.

Other challenges in the IT/business relationship include:

• Clients insisting on traditional (document) deliverables or process. (This requires org change, educating the client, pilot approach etc.)
• Failing to have the right discussions early on.
• Treating software development as an “IT thing”. (If IT is supposed to support the business and help create business value, the business should assume co-ownership with solution delivery.)
• If IT has a reputation of being unresponsive or uncooperative, then this reputation will be a severe impediment and must be changed.
• If there is a history of blame between IT and business functions, then it will be difficult for IT to form the collaborative relationship with business that is needed. Both the business and IT need to understand each other’s challenges and the reasons for their respective priorities.

Measurement

Waterfall projects are measured based on whether they are on time (on plan) and on budget, with the requirements for a release fixed firmly. What tends to happen in practice is that the time and budget get extended, but the requirements remain fixed. Agile projects usually fix the time and the budget but allow the requirements to be somewhat flexible, so that a release might not complete everything that was planned, but useful capability still gets deployed on time. That requires a mindset change in terms of what "success" means for a project. It also presents a challenge in that the traditional metrics of time and budget are less useful because, since requirements can change, being on time and on budget does not tell one whether a capability has been developed. Therefore, a different approach is needed for measuring agile projects and determining which are "doing ok" and which are not. Various approaches can be used, but upward reporting tends to shift to a focus on issues that have been escalated, and portfolio planning shifts to a focus on work streams instead of individual projects. Decisions about work stream health are then made when the portfolio of projects is reviewed and are compared based on their spend rate and what capabilities have been delivered over time.

Infrastructure

Modern agile methods rely heavily on cloud infrastructure to enable the rapid (automated) provisioning of test environments on demand. Government data centers usually do not provide this capability, or if they do, it is not operated properly, often as a result of rigid contracts with data center operators – contracts that are often negotiated at a high level within the government by people who are not familiar with how cloud systems are used. Private and public companies rarely have such constraints because they are usually free to make more tactical and more local decisions with less emphasis on standardization and less pressure to create very large long term multi-agency contracts.

Access to Skills

Agile methods rely heavily on specialized skills such as test automation, application level security programming knowledge, build automation, and virtualization. These skills are very much in demand in the marketplace, and so the organization is competing for these skills. An inability to attract or grow these skills will greatly impede agile process and cause most agile projects to perform very poorly – possibly worse than they would perform without agile.

Distributed Workforce

Highly skilled people have choices and they can select jobs that are not only lucrative but that are also convenient. If the organization is located in a place that is not considered to be desirable to today's tech worker, it will be at a disadvantage. Further, large organizations are inevitably distributed, with offices located far apart. Agile methods rely heavily on face-to-face collaboration. Electronic collaboration can work very well (see Distributed Scrum: Agile Project Management with Outsourced Development Teams, by Jeff Sutherland et. al.), but it needs to be set up effectively, and that is often difficult because of organizational restrictions about using social networking tools that store data outside the organization's network. Effective software collaboration is often graphical, via a whiteboard, and to implement that electronically requires the use of computer screen sharing, clear phone connections, and facilitators who are experienced with remote collaboration.

Perhaps the biggest challenge with a distributed team is getting it to behave as a team. Anonymity breeds bad behavior, and when people do not see each other regularly they feel anonymous. It is therefore important to ensure that there is frequent personal contact among the team members, even if it is electronic. A possible solution is to establish small complete co-located teams that communicate with other complete teams in other regions, rather than splitting up teams geographically.

Regional Culture

Regional culture matters. Asian cultures (India, Vietnam, Thailand, Philippines, China, Japan, Malaysia, Singapore) are receptive to collaboration, but are particularly sensitive to hierarchy, status and titles. Respect (for hierarchy) is very important. This must be taken into account when promoting agile methods, which advocate open door policies and egalitarianism as well as the right to speak up about any issue. The cultural preference for hierarchy also challenges agile values for self organization, treating “fast failure” as learning, and independent decisionmaking. Even the phrase “servant leader” has a confusing or negative connotation in some Asian cultures, in which the term “servant” connotes someone of lesser status. Many organizations in Asian cultures also have entrenched practices for employee performance appraisal that are counter to agile philosophy, such as imposing performance bell curves, creating a zero sum game for employee success. Too much organizational process around controlling and managing employees will also impede an agile transformation by inhibiting cooperation, collaboration, openness, and employee initiative.

Northern Europe presents an interesting contrast. For example, in Holland, a project manager has to earn the respect of his/her team! He or she starts out as the least respected person!

Organizational Culture

An organization’s culture includes things like:

• Are new ideas heard or pushed down? (“We don’t do things that way here.”)
• Is individual initiative rewarded or reprimanded?
• Are people empowered and given a chance to self organize, or must one always wait for direction?
• It work organized hierarchically, or organically?
• Is a courageous experiment that did not quite succeed treated as a failure or as a learning experience?
• Is speaking the truth accepted and rewarded, or punished?
• How well is change of any kind accepted?
• Is it permissible to ask questions of those at higher levels of the hierarchy?

One might assume that for agile to work, all of these questions should be answered in the affirmative. However, things are not so black and white. At a project level, it would certainly be true, but at an organization level, one cannot simply throw out all command and control and hope that self organization will take over and drive profitability. Rather, applying agile at an organization level is a nuanced and thoughtful process, and must be informed by other ideas spheres including Lean Manufacturing, Systems Thinking, and Adaptive Management. Still, what works for one organization might not work for another; and in any case, it is not always best to introduce radical change overnight. In short, agile provides ideas for how to loosen the rigidity of an organization, but visualizing the right path forward is a great challenge for which there is no repeatable recipe.

Contracts and Procurement

Non-agile organizations typically utilize vendors using fixed price, fixed duration contracts that specify very detailed deliverables. That entire procurement methodology is incompatible with agile methods, and that is why organizations that adopt agile also change their contracting methods, moving to contracts that refrain from defining deliverables but instead define skills and the ways in which a team will work with the organization. Contracts must be structured so that they foster collaboration and win-win partnership, and there are many mechanisms for achieving this, such as by making the contract renewable, and by using a standard contracts that can be used for many different vendors, thereby enabling the organization to substitute one vendor for another if “things are not working out”. Achieving this requires educating the procurement office on the types of contracts that are required, and instituting new contract models.

Security and Compliance

Private industry takes a value and risk based approach to security and reliability, which is compatible with agile. Government agencies take a compliance based approach, and that presents substantial challenges for agile processes. Compliance based risk management focuses on reviews, and usually reviews of documents. Documents take time to prepare and review, introducing a substantial latency into the development process between when a project has completed and when it can be deployed. Agile processes generally assume that any portion of the software can be changed up to the last minute and the software will be deployed as soon as it has run through its automated tests - usually the next day! For example, organizations such as Amazon deploy continuously - many times a day. In a compliance centric environment, it would take time to update the documents and then review them again, and this would add weeks or even months, thereby undermining the agile processes in which rapid change is considered to be normal.

The solution to this is to take a risk based approach, and to validate the development and deployment processes rather than trying to validate the actual software itself: if the process is trusted to faithfully implement all of the required controls, and verify that they are all in place, then the process can be trusted to deploy to production. Many US government agencies and DOD are moving in this direction. Achieving this requires changing how risk is managed and how development testing and deployment are done. These changes are very large and involve policy changes.