The 8 key metrics product leaders can no longer ignore
Our top 8 key software metrics for product leaders:
- Cycle time & new feature cadence
- Usability over key flows
- User engagement with new features
- Time to resolution
- Product/application crash frequency
- Time investment profile
- Defect removal efficiency
Why do software metrics matter for PMs?
The Consortium for Information and Software Quality (CISQ) recently teamed up with Synopsis to calculate the financial cost of poor software quality in 2022. Their findings?
All in all, poor software quality led to a loss of $2.41 trillion in 2022(that’s up significantly from the already astronomically high loss of $1.31 trillion back in 2020)
To add to this, since the start of 2023, around 300 tech companies have laid off almost 100,000 workers - many of whom are in the product management and quality assurance disciplines. What does this mean for software quality in 2023? Expect that $2.41 trillion number to increase substantially.
So, how can product leaders stack the deck back in their favor this year and get software quality under control and back to working as a driver of growth? With concrete, measurable goals that signal success or failure without second guess.
As the purse strings tighten across many tech organizations this year, more and more CEOs and revenue leaders will be looking to roadmap and benchmark software development and quality with reliable metrics. As a product leader, while you may not be as directly responsible for software metrics as engineering leaders in your organization might be, it’ll still be on you to contribute to improving these numbers and know exactly what any changes in them imply for your team’s output and the quality of your product in general.
To add to the laundry list of reasons to double down on quality in 2023, the advancement of cloud technology and micro-services and their incorporation into the average company’s tech stack has brought about a fast-paced development schedule for software teams. However, this has also created a more complex environment where more preventable bugs slip through the cracks as developers work to deploy and manage software across different platforms and devices, all of which carry their own associated risks when it comes to quality. As things are, many developers are already feeling tired, burnt out, and more ready than ever to abandon ship. Add to this an already strained relationship typical to the product/engineering paradigm, and you see why taking on some amount of ownership of these metrics can help you understand and guide your engineers’ output far more effectively.
Furthermore,76% of consumers now say they’d cease doing business with a company after just one negative customer experience due to economic pressures like inflation. By the time frustrated users are airing your company out on Twitter for a delayed feature or broken experience, statistically, it’s already too late to salvage the relationship - and in the current climate, the blame for such churn will increasingly fall at product’s feet.
Add all this up, and you can see why forward-thinking product leaders are tripling down on product quality. In such an environment, setting clear standards for what constitutes an acceptable product experience and optimizing your workflow to address only the most mission-critical issues in your application is no longer a nice-to-have. Your developers don’t have time to crawl through every log and study every stack trace anymore, and you certainly don’t have time to wade through every bug report that comes across your desk or hunt and peck for every growth opportunity in your product analytics software. You need a fast and efficient way to know what’s breaking, if/why it matters, and what you need to do to fix it, period.
So... how do you make software quality tangible across your organization?
Well, PlayerZero can certainly help! PlayerZero helps product teams understand which incidents are having the greatest impact on their customers — in real time so you can take control of every incident.
Learn more about how we help product managers measure the impact of incidents, identify affected customers, get the right engineering context to their technical teams and get a fix out 2x as fast today:
The 8 most crucial software metrics to watch in 2023
Below are the 8 software metrics you should be owning and analyzing each and every week over the next year.
1. Active users
There is no single metric in 2023 that’s more important than the number of active users in your platform. While active users are an output of all disciplines of a business (marketing to bring them in as leads, sales to close the loop and make them users, customer success to keep them happy), their tendency to remain active users over time will correlate directly to the quality of experience delivered by your product.
Consumer products have long keyed in on this metric as one of importance, but more recently B2B SaaS has realized the immense value of understanding & directly quantifying the number of active users in their software - especially for those that are built around a primarily self-service model. Try reaching out to the sales & customer success leaders at your organization to get a pulse on what these numbers look like and where any weak points in your product are that are leading to churn.
2. Cycle time & new feature cadence
The frequency at which you’re able to introduce significant new features over a given time period is crucial and will only be more-so over the course of the next year. In today's fast-paced world, users demand a consistent flow of updates and enhancements to maintain their interest. When analyzing how your team performs in this regard, it’s crucial to figure out how your rate of innovation compares to your competitors.
No matter who you are, whether you’re an incumbent or a bootstrapped startup, if you can’t keep in time with the pace of innovation in the broader market, you’re guaranteed to fall behind. Pro tip - this metric needs to be balanced with metric #4 (more on that later).
3. Usability over key flows
Look, the reality is that not all errors actually matter - controversial, I know. But anyone who’s done this job long enough knows without a doubt that not every error in your application actually amounts to a worthwhile incident in Jira.
Whether you’re using a traditional engineering monitoring platform, an observability platform like Datadog, a product analytics platform, or some combination of the three to determine what the actual impact of software errors is, you’re undoubtedly spending more time than you need to and missing out on vital context.
PlayerZero flows changes all that. Flows give you the ability to analyze and optimize software quality along the pathways that actually matter in your product - onboarding, checkout, or any other path that directly equates to conversions gained or lost. Think of it as the connecting layer between observability/monitoring and analytics. Flows allow you to quickly identify any issues that may be hindering your users’ progress, and make actionable decisions about how to improve their product experience. Now you can solely focus on the errors that directly prevent important user outcomes and improve your bottom line, all while saving time for yourself and your team.
4. User engagement with new features
New features are only valuable if your users actually use them. As the caveat to metric #2 (new feature cadence), it’s crucial not only that your team is introducing valuable new features regularly, but that those features are actually adding value for your users. If you find that user engagement with new features is routinely low and tends to stay low over time, it’s time to reevaluate your roadmap.
5. Time to resolution
Every minute that your product is down or otherwise unusable is a potential lost user. While the balance between releasing quickly and releasing without bugs is a difficult one to manage, as each user becomes more valuable to organizations due to economic headwinds, product quality won’t be spared from uncomfortable conversations around lost business.
That said, it’s impossible to deploy and manage a 100% perfect product in today’s landscape. Perfection simply isn’t possible in a competitive environment in which everyone is competing to see who can release the newest, shiniest feature set. So, how do you measure how well you and your team handle critical product failures? By measuring the average time to resolution. Look for this number to be low and decreasing over time to show positive progress.
6. Product/application crash frequency
Crashes are code red in the software quality world. Any error significant enough to make your product unusable is guaranteed to cause some amount of customer attrition, therefore, it’s absolutely crucial to minimize the number of times your product or application crashes in each update. Your goal will be to get this number as close to zero as humanly possible. Pro tip - plan big releases that introduce risk to your application for off days (for us, that’s Sunday nights) to avoid potentially crashing your app during peak traffic periods.
7. Time investment profile
This metric is aimed at answering a surprisingly complex question: across the surface area of your company, where is it that your engineers & product managers spend their time?
Tracking your team’s time investment profile is crucial for engineering leaders in a contracting economy because it allows you to better allocate your resources and prioritize projects more effectively. In a contracting economy, resources like time and budget are limited, and competition for funding and support between departments can be intense. You need to have a clear understanding of how your team is spending their time (even more-so for cross-functional product teams) and on what initiatives in order to make informed decisions about where to allocate resources/trim the fat. Pro tip -if you’re like most companies, you’ll be surprised to see just how much time your team is spending each week crawling through logs and reproducing bugs.
8. Defect removal efficiency (DRE)
Defect removal efficiency quantifies how efficient your team is at eliminating errors before a release to prod. Here’s the formula for DRE:
DRE = (# of defects found internally/ # of defects found internally + # of defects found externally) × 100.
As you can see, the denominator in this equation consists of both the number of defects found internally, and those found externally - meaning that this metric should be as close to 100 as possible (ie, zero externally caught bugs). Solid performance in this metric indicates that your new features/updates/launches are generally bug-free before they reach production.
Conclusion
In conclusion, software metrics are crucial for product leaders to measure the quality and efficiency of their product and to make a case to their superiors for their performance and that of their teams. In this fast-paced world, staying ahead of the competition and delivering a consistent flow of updates is more important than ever. But, too many companies have used this as a license to release buggy software - leading to high churn rates industry-wide. By analyzing these metrics each week and continually improving them, you can make software quality tangible across your organization and remain competitive in the ever-evolving technology landscape.