Reading time: 10 minutes
Audience: Speaking to DevOps practitioners and the people that manage their work.
I don’t care what work management framework your team or company claims to use, making software will still be hard. Supporting those software teams is even harder. Good frameworks can definitely help you manage and deliver your work, but no system in the world can make a difference if people don’t follow the program.
This article dives into some of the biggest culprits that make it difficult for DevOps teams to stay organized with their workloads. This has nothing to do with any specific framework or tool, but instead emphasizes the impact of the realities DevOps teams face.
The "why" isn't always evident.
“What’s in it for me?”
If the customer and/or the engineer feels that the process is in the way of getting things done, instead of adding value, it will likely be a never ending challenge to ensure it’s being followed. This can happen as a result of processes being designed in a vacuum, or requiring too much heavy handedness.
Redundancies are bad things.
I hate to state the obvious here. I’ve seen so many cases where engineers are asked to input the same data or information in multiple places. Some engineers will be good sports about it and put up with the bureaucracy, but it’s not a good cultural dynamic.
“Do it or else” doesn’t work in the long run.
It can be difficult to get engineers to do something they don’t intellectually accept as necessary. This is especially true as engineers become more experienced and senior. It’s easier to micro-manage a junior Help Desk engineer into granular tracking than a seasoned engineer who has migrated data centers into the cloud. However, if there are major consequences to not following the rules, the short term result will be compliance, but the long term outcome is usually excessive turn over.
Tracking the work gets sacrificed by doing the work.
Fire fighting is disruptive.
DevOps is hard. It’s messy. What seems like a good idea today often becomes a future regret. You get pulled in many directions. You are probably someone’s hero and someone’s scapegoat on the same days. If you try to implement a process but fire fighting is your reality, sticking to protocols becomes difficult. Even if the chaos isn’t perpetual but you have a rhythm such as periods of quiet time interrupted by hectic scrambling, this is detrimental to consistency. A program inconsistently followed not only isn’t really adding value, but has a high probability of being abandoned.
It’s easier to just get it done when it’s urgent.
It could be there’s a tendency to begin work on a customer’s request and they routinely neglect to log a ticket. Most teams will hit critical mass at some point and try to impose a policy like, “no ticket, no work.” Even so, if the underlying problems that cause so many requests to be urgent aren’t addressed, the volume and urgency will override protocol and you can find yourself defaulting into “just get it done” mode. The more you compromise on following the process, the less chance it has of surviving.
It’s hard to plan work that’s trail blazing.
Complexity is another reason tracking work gets neglected. This isn’t exclusive to DevOps, but manifests often because norms like working with open source tools that are on 0.x releases and utilizing public registries are facets of wild west development. It’s difficult to give an estimate on something you can’t predict.
You're the foundation of many.
It’s usually a one to many relationship.
Sometimes, there is no DevOps team. It could be a single individual on loan to different Development teams. Maybe you have a developer moonlighting as the DevOps resource, or an Operations engineering trying to make the leap. Most commonly, you are a shared services team that supports developers with unrelated priorities (the fun of figuring out who to disappoint is usually on you). If you’re a shared resource, it’s likely not possible to be in every meeting and conversation. Decisions get made, dates get agreed to, and hot potatoes will be coming your way.
A major contributing factor is that most organizations have this model:
Business Teams -> Product Owners -> Development & QA Teams -> DevOps/SRE/Operations
You know what this means? You’re in the critical path while being the most levels removed from the decision makers. This leaves you relying on developers to give you adequate notice of inbound work, and we all know developers are known for their awesome communication skills. You’re usually the last to know when something is needed. Multiply this scenario by the X number of teams you support, and this is how fire drills become normal.
Very few understand your world.
Product Owners usually have software backgrounds.
This isn’t necessarily malicious. I have found working with many clients that the biggest issue is a lack of recognizing dependencies. A Product Owner may have no idea that implementing a change in their app has implications on the infrastructure, and their developers may not have any idea either. It’s not uncommon they may have faulty assumptions about how the infrastructure works, just because “it’s in the cloud.” If they don’t have you in the conversation, or don’t articulate well enough what it is they are trying to do, then no one in the room will be in a position to ask the right questions. This lack of understanding also means prioritizing investment into better infrastructure engineering is difficult. Have too many hacked solutions and you’ll have even more fires, which will further hinder your ability to deliver.
PMOs usually service software teams.
It makes sense. You solve for the majority of use cases and worry about exceptions when they come up. If software development represents the largest chunk of the puzzle to solve, DevOps is the exception to be dealt with later. Modern Agile frameworks were built with software development as their focus, so it’s relatively easy to find PMs and processes that work for those teams. There are tons of online resources on this. You can find lots of qualified candidates, coaches, and so on. DevOps is a different beast. I’ve seen a lot of organizations struggle to find PMs that can hang with the chaos DevOps teams deal with, let alone add value.
You can't automate your way to a better state.
I’ve constantly been impressed by the creative ways engineers find to programmatically solve every day hassles. The more things can be exposed via an API, the more cool things we can glue together. Unfortunately, a lot of the elements that make a work management framework effective calls for human intervention.
You can’t automate work intake.
In order for work to be ingested into a tool and tracked, worked on, and delivered, it needs to first be conceptualized. The problems themselves need to be recognized. Then there needs to be requirements, design, iterations on design. This can be as informal as a conversation, but it’s helpful to visualize your work in familiar abstractions like a user story, a task in a plan, or some blurb about the desired end state. These all come from people, so the degrees of variation are endless. There are templates you can copy and reuse for these activities, but you can’t automate the process that brings them into being. There’s also a never ending evolution of how to do things, so your templates will become stale before long.
You can’t automate status reporting.
You can have a report be sent out by a tool to a distribution list or a Slack channel that shows the status of a customer’s request, or a breakdown of a project’s completion percentage, blockers, risks, and so on. You still need people to input that data in the tool. Knowing what the report should say and how it should be said is a fine art. It takes deep contextual understanding of the current dynamics going on within the affected teams and the project, as well as the internal culture and politics of the company. Simply providing data isn’t enough, your data NEEDS context, or you risk confusion and unnecessary drama.
You can’t automate communication.
If any framework you implement has any chance of adding value, you must have solid communication throughout the org. A tool cannot produce a high quality meeting where everyone is engaged and walks away with what they need to do their jobs. The quality of articulation is also subject to interpretation. Imagine a simple scenario where you need a developer to unblock you on your task. This usually takes some back and forth, deconstructing the problem, some debate on the proper approach to rectify, and finally an agreement on next steps. This is the messy stuff. The “soft skills.”
So what can we do about it?
Bring in a consultant to help. C’mon, you knew it was coming. We’re a consulting service, after all. Seriously though, bringing in outside expertise is the fastest path to get where you want to go. You also need the right people to deliver, which brings us to our first point.
Get the right talent.
This is a great place to start. Most of these issues can be alleviated by having a good Project Manager. If you don’t have one, get one. We can help. It pains me to say this, but so many PMs don’t add a lot of value. Sometimes it’s not totally their fault. For example, it’s a tall order to ask a PM who has experience with app development to make the leap to infrastructure engineering. Not understanding the work, at least conceptually, is a huge handicap. Having an effective PM that understands infrastructure can make all the difference in the world.
What should I look for in a consultant?
Get someone that understands the infrastructure world and avoid extremists. Our approach at Blue Pisces aims at one thing: help the engineers deliver. It’s really that simple. If it doesn’t help the engineer do their work, then it isn’t providing optimal value to the business. Agile frameworks weren’t made to be beholden to, they were made to improve our productivity.
Should I expect to start from scratch?
If a consultant tells you that, it better be after an assessment and accompanied by other options. Before we declare bankruptcy on your current state of affairs, know that sometimes all you need are small adjustments to right the ship instead of radical change.
I’ve met with people who are ready to rip everything out and start from scratch, and those who have strict rules about not changing certain things. We don’t go into any engagement with a pre-determined solution. We assess and then provide options with pros and cons.
A consultant should help you define the right goals for the right reasons.
You might have a wish list, but a consultant should challenge you to answer why. Why do you want to be Agile, why do you want to track better, why do you want to plan more strategically, why do you want fancy metrics? It’s helpful to decompose these goals down to the problems you are trying to solve. It’s also really useful to identify your limitations. For example, you might not have the support of every team in your org, or some on your team may be resistant to change. Only when these things are clear can the right solutions be crafted.
- A good framework helps DevOps engineers get their work done.
- Organic buy-in is infinitely preferable to relying on policy enforcement.
- Fires and complexity make it hard to track and plan consistently.
- Popular org structures tend to produce communication bottlenecks that are felt by DevOps teams.
- Product Owners and PMOs probably don’t have a lot of infrastructure experience.
- Automation can’t help with the “soft” problems that need to be solved.
- Hire a good PM and/or a consultant to help if you don’t have the expertise in-house.