Skip to content

Delivery metrics#

Why do we need delivery metrics?#

Turning specifications into actual products is a core activity of engineering teams, and it is therefore key to optimize the related processes. We take a data-driven approach to measuring our efficiency so that we have an objective view on our delivery performances and avoid relying on feelings, as it is extremely hard to have an objective view on how delivery is going. Those metrics allow us to identify bottlenecks in our delivery cycles and bias in our predictions, so that we can improve to become more efficient and reliable and so that we can see progress!

There can be many reasons for some delivery metrics not to be as expected. The interpretation of those metrics should be discussed within the teams and in regard to their specific context to understand what the issues could be and how things could be improved. Some generic root causes are suggested in this document as hints, but retrospectives within the team are usually the best way to identify issues and ways to improve.

Our delivery metrics#

This section lists common and recommended delivery metrics we are using. Each team can have additional metrics depending on their needs.

Lead time metrics#

The lifecycle of a task is split into several steps, usually represented by columns on a board on JIRA, GitHub, etc. For instance: In Progress, Ready for Review, Ready for QA, etc. Measuring the time tasks spend in each column provides great insights on our delivery pipeline:

  • If tasks remain too long being developed (typically more than 2 to 3 days), it can be a sign that we can break down work further in smaller steps, or that the developer experience could be improved with more automations.
  • If developers have to wait too long for a review (typically more than 24 hours), it is an area for improvement to shorten the iteration cycle and reduce context switching.
  • If the QA/test time is becoming the main bottleneck, it could mean the team would benefit from better test automation, or change its approach to QA.
  • If tasks remain for a long time being completed but not released, it could mean the team would benefit from a streamlined release process.

Tracking the lead time in each status allows team to understand where time is spent between picking up a task and delivering it. Therefore, it allows team to identify opportunities to improve their delivery time.

Measuring lead time per status also allows to generate alerts or notification, if a specific task is stuck for longer than expected, so that teams can focus quickly on stuck tasks to unlock them.

Steps back in task lifecycle#

Implementation tasks are expected to move forward from being implemented, to being reviewed, then validated and finally delivered. While it is expected that some tasks may not pass reviews or tests and go backward, this should remain a limited behavior. For instance, if tasks consistently fail the QA, review or testing steps, it is a sign that quality standards are not enforced and respected enough at implementation time and that further qualitu considerations should be taken when developing tasks: spending more time testing, learning how to test properly, having more automated tests and code checks, etc.

Goal completion rate#

Being as predictable as possible is key for our teams, as many other teams are dependent on Product & Engineering progress: Marketing needs to know release dates to prepare campaigns, Support wants to communicate bug fix dates to impacted customers, etc. We also know that perfect predictability is not a realistic standard in Tech, as complexity and number of unknowns is often quite high, especially on large timeframes. Engineering teams should learn and organize to be as predictable as possible on short term commitments. Our "short term commitments process" is Weekly (or sprint) goals. At each end of a cycle, teams provide an update on goal completions and should track their completion rate over time. While each team and context is different, a general target would be to reach on average 80% goal completion rate. This accounts for uncertainty while setting ambitious standards for ourselves.

The goal completion rate should allow teams to counter-balance their bias: teams that consistently achieve close to 100% may benefit from taking a bit more risks and having more ambitious goals, while teams with low completion rate may need to be less optimistic in their predictions.

Number of alerts and/or escalations#

Engineering teams are usually the last escalation level of support escalation processes. Teams managing services also handle operational alerts. Those activities are expected to some extent, but they should not represent a major or significant part of the team's workload. If it is the case, it can be sign of too many quality defects in which the team should invest to reduce operational work in the future, or inefficient escalation processes that should be taken care of to better empower support teams without having to rely on engineering teams.

Metrics for those topics can be counting number of escalations per day/week/month and number of alerts received per day/week/month.

Our implementation of delivery metric dashboards and alerts#

The delivery metrics described above are automatically computed based on data gathered from our different tools (JIRA, GitHub, Slack). This data can be explored through Metabase and standard dashboards are available with our default metrics.

Internal documentation is available here.