Jeff Keyes, VP of Product at Plutora, offers 10 metrics to give you a foundation to measure DevOps that will help your business to flourish and allow you to operate with an optimised DevOps architecture.
With the DevOps market size expected to reach US$12.9 billion by 2025, it is well worth understanding how to make best use out of this evolving approach. DevOps essentially links the development and operational sides of software delivery, however, it is often considered tricky to benchmark its success. But don’t worry, there are some performance indicators – known as metrics – that every DevOps team can use to gauge the effectiveness of their approach.
Here are what can be considered the top 10, with each of these metrics important enough to affect the success of the developer-operations relationship implied by the very concept of DevOps.
- Application availability
Users need to have uninterrupted access to your application 24/7, unless you’ve warned them in advance that scheduled maintenance is due. Measuring availability is a simple metric to boost resources towards maintaining an application. After deployment takes place, an application’s availability can drop. The DevOps team should be able to follow logs and rollback changes to its last most available version, however, businesses that deploy their applications thousands of times on a daily basis will argue this is harder than it sounds.
To measure availability, an orchestration tool like Kubernetes or Jenkins can be useful. You can also look into application performance managers (APMs) to smoothly handle the logs and rollback points.
- Traffic and application usage
The next focus is monitoring an application’s traffic and usage. If your application experiences high demand, it could struggle under the pressure. An intuitive log analyser can notify developers whenever that breaking point is near.
Usage statistics are also useful as you release new updates over time. A dip in usage can suggest that you implemented a change that’s unappealing to the end-user. A competent DevOps team will infuse measures and controls to quickly correct dips in usage caused by new features. For example, you can introduce new changes using feature flags, which allow you to gradually release features to groups of users. This helps your feature to be rolled out as an option slowly enough to gain some acceptance among users without a drop in usage.
- Number of tickets
Users provide feedback by submitting tickets to your support team, and changes can be implemented between coding and tests in a DevOps pipeline. Logistics around this DevOps metric should be kept simple – you don’t want users submitting tickets about a poor ticketing system. Third-party tools commonly used to track tickets and their life cycles are the smartest option. This might cost more, but less time and effort should also be considered.
Your DevOps team can focus on keeping the system fresh by submitting internal tickets. However, they’re naturally of lower priority than externally generated ones. One way to keep on top of this is with a graph of tickets vs. time as it helps to focus on the trend and not just the tickets themselves.
- Commit count
Commits are changes sent to the main source code file using a version control system (VCS), such as GitHub. The more commits your team is recording, the more productive they are generally. However, a commit is only truly useful when senior developers have approved it, and therefore approved commits matter much more.
Any VCS should allow you to count the total commits completed in any given period. This can be a source of inspiration, as typically developers with more commits inspire the rest of their team to speed up. Low commits can also reveal why your application versioning is slow, enabling managers to identify and support the developers with the lowest commits.
- Number of tests conducted
Looking deeper than the overall commit count, the number of tests completed on a single commit is also vital. If you’re having to do more than one test on a single possible change, there could be a problem with the process. Conducting tests manually is a common cause. While thorough, relying on a human tester creates opportunities for mistakes when running manual experiments on your commits and builds before deployments.
Alternatively, using automated tests spread across as many containers as possible in production environments lessens the amount of time you’ll have to wait for a session. It also increases the number of concurrent tests done (which count as one rather than multiple).
- Rate of deployment
A deployment is when a new version of your application is created, and the speed these are produced is an overall indication of your team’s productivity. However, increasing successful deployments relies on a good system managing every step prior to deployments. You could have a thousand commits awaiting tests, but if you’re not using the best testing methods then you can still have the best developers but deploy very little.
While a company the size of Amazon will have DevOps teams working to make improvements around the clock, most smaller enterprises don’t deploy as frequently. But the deployments that they do, when done daily, reflect the work going on behind closed doors. For management, dips in this metric are clues for the ops side of DevOps to improve processes that lead to deployments.
- Speed of deployment
Speed of deployment measures the time it takes to complete a single deployment. Each application’s deployment time will vary based on the final size of the file. The more lines of code, assets and dependencies, the longer an app will typically take to deploy.
One thing to consider is whether it would be better to commit once per day and build just as much, or whether it would make more sense to apply multiple tiny changes over the course of the day? The latter option allows quicker deployments, saves the compute resources required at any given time (i.e. costs) and does so safely enough that if required you can return to the last state where changes were made correctly.
- Deployment rollbacks and fails frequency
Deployments are sometimes undone via rollbacks, for reasons including missing the scope of an app’s requirements, executive directions and bugs that could have escaped testing. All continuous integration / continuous deployment (CI/CD) tools log successfully completed deployments as well as those that hit a problem and stop before completion. You want as few rollbacks as possible, so if lots are happening you’ll benefit from more manual checks before builds.
The fails frequency of your cycle is also important. It looks at not only the final deployment, but every step to get there. A well-defined feedback process leads to better coding and commits, which then leads to fewer rollbacks after deployment.
- Version lead time
Version lead time measures the time it takes from a ticket being received and being resolved. Perhaps one of the most accurate depictions of your DevOps processes’ productivity; version lead time takes a step back to watch everything in action. Project managers typically use this metric to estimate task deadlines when apportioning resources to projects.
It’s important here to consider how tasks vary in difficulty. It helps to include the relative weight of a change in the reports gathered to present lead times, as well as how many developers were involved in the change. Logically, the more hands you put to work on a change, the less time it should take.
- Rate of response to tickets
When optimising for speed, you get lots of insight by monitoring the rate of response to tickets. The previous nine metrics gain momentum from the results you get from this step. Regardless of how difficult a change suggested by a ticket seems, the ticketing process ensures enough workers are assigned to it. This immediately cuts the expected lead time and increases the number of commits possible.
One way to build a great response rate to tickets lies in having a value stream management (VSM) tool. These connect information across all your applications in the departments along your pipeline, increasing visibility. The background information your developers have when creating and mending features is increased, and ticket issues reach the deploy stage quicker and with fewer chances of being rolled back.
This list of metrics isn’t exhaustive, however, these 10 give you a foundation to measure DevOps that will help your business to flourish. Once these metrics are in place and data is being revealed, the key objective is to make small changes to different areas to improve the pipeline, so that each metric slowly improves too. Ultimately, having a clear objective known by all team members for each of the DevOps metrics sets you in the right direction towards an optimised DevOps architecture.
