Enterprise DevOps: Too big to fail?

Reprint Dec 2104 from infocus.emc.com.

For many enterprise IT shops, the “too big to fail” theory from the financial services sector is akin to how the often fragile application delivery pipeline is both perceived and maintained.  The “too big to fail” theory asserts that certain financial institutions are so large and so interconnected that their failure would be disastrous to the economy, and they therefore must be supported by government when they face difficulty. (Andrew Ross Sorkin)  Complex multi-department approval flows, regular business escalations, and developer death marches are all indicators that, culturally, failure along the application delivery pipeline is considered “disastrous to the [business]”and therefore, the Software Development LifeCycle process in place must be maintained at all cost.  This sentiment leaves most enterprises fostering dysfunction to maintain the status quo rather than challenging the system and collaborating to create corrective solutions.  This paradigm is very different than what DevOps suggests, which is namely continuous improvement or change.    Why is this so difficult, you ask?

Rachel Shannon-Solomon captured many of the reasons in a CIO Journal article titled “DevOps Is Great for Startups, but for Enterprises It Won’t Work –Yet”.  In that article, she outlined 4 key deterrents to the adoption of devops practices at the enterprise, namely:

  • Siloed structures and organizational change
  • Tendency to build incrementally or cobble together piecemeal solutions
  • Cultural implications
  • Measuring ROI

To Rachel’s list, I would add technical debt in the portfolio and complex SDLC processes that reflect the structure of the organization that the processes are based (aka. Conway’s Law).  But are these impediments too much to overcome for an enterprise?  Like many before me, I would argue “no”.  Enterprises can and do change however there are unique challenges and often larger hurdles to overcome due to the magnitude of the change.

In the Enterprise, more so than smaller organizations, successful change requires both a top-down and bottoms-up approach.  Top-down typically comes from the executive or senior leadership levels of the organization.  It defines the strategy, context, standards, and metrics that will govern the change process, establish boundaries for the change initiative, and provide feedback needed to validate and/or adapt the implemented changes.  In agile circles, this is often referred to as the WHAT.  Without this level of sponsorship and rigor, change initiatives are often derailed by resource shortages, scope creep, and/or conflict and competition between groups and/or departments.  At a minimum, this top-down structure provides the escalation and decision making framework for change.  In my experience, this is most often what is missing in the enterprise since it requires significant political capital and a commitment to the organization over self.  This is what Rachel was referring to as cultural implications.  My favorite example of this occurred at Amazon in early 2002 when Jeff Bezos issued a mandate to expose data and functionality through externalizable api’s.  In his announcement to Amazon, Jeff stated

  • All teams will henceforth expose their data and functionality through service interfaces.
  • Teams must communicate with each other through these interfaces.
  • There will be no other form of inter-process communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.
  • It doesn’t matter what technology they use.
  • All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.

His mandate closed with “Anyone who doesn’t do this will be fired.  Thank you; have a nice day!” (http://apievangelist.com/)

This change wasn’t small. It likely delayed work-in-progess and required the development of new frameworks to support future development.  But Bezos, as other leaders, recognized that to achieve the vision — change was required; and, it wasn’t optional.

What I specifically like about this example is that it does a very good job at illustrating the WHAT.  This leads me to the bottoms-up requirement of devops in the enterprise, otherwise known as the HOW.  In Bezos’ mandate, he did not tell teams how to implement the change.  He simply set expectations and standards and left the details to each team.  When implementing DevOps, it is important to let those closest to and most impacted by the problem define how they intend to self-correct.  (This is why it is so critical to have cross-functional teams designing and developing the solutions so as to incorporate input from multiple perspectives along the delivery pipeline.)  By using a bottoms-up approach, enterprises shift ownership of the solution onto those responsible for enacting it thereby creating de facto champions that can help drive adoption.

Changing the big Enterprise is no easy or quick task.  It will take time and commitment from individuals and groups at all levels of the organization.  DevOps in the Enterprise is possible; and as tools and practices mature, the ongoing transformation gets easier and easier.  When failure isn’t an option, who will bail you out?