You may ask yourself why you need a production debugger. There are already different ways out there to debug production errors: log files, APMs, error monitors. There are even those system dump files people have been using since the 60s. Well, all those tools may take you some of the way to solve production errors, but none of them give you the code-level visibility needed to perform a root cause analysis to really understand what caused a bug so you can fix it quickly. Yes, those tools do have their place, but a production bugger is a critical member of your enterprise tool stack. This ability of a production debugger to help you fix Production bugs more quickly can have a profound effect on your business. Let’s examine how.
Effectiveness of developer salaries
You pay your developers to develop the capabilities that will drive your business forward. The faster they can develop, the higher your business will soar and adapt to ever-changing customer needs. The problem is that developers spend much of their time debugging rather than building that next killer feature.
What if I told you that, on average, you’re paying EVERY developer on your payroll $10,750 per year to fix production errors!
Let’s do some 6th-grade arithmetic.
Some studies have shown that 43% of application developers waste 25% of their time debugging Production errors.
Let’s assume that the average software developer salary is US$ 100K per year (sources such as US News, Glassdoor, or career sites, indicate that’s a reasonable approximation for our ballpark estimate).
The math is pretty simple.
If you have 100 developers on your payroll, the money you’re spending for them to debug Production errors is:
That’s over a million dollars a year (!) of developer salaries spent on debugging Production errors (and the $10,750 a year per developer that I mentioned)
From the perspective of time, that equates to every developer on your payroll spending 1.3 months of every year fixing production errors. Think how many great features and how much value all those developers could bring to your customers if they could spend that time doing what they love most, developing code.
A production debugger can reduce debugging time by up to 80%, but let’s be more conservative and say it only reduces debugging time by 50%.
Do I need to run the numbers for you again?
Attract more customers and reduce churn
How many times have you tried a new product or service, experienced a problem, and just left? Yes, we’ve all been there. The fact is, production errors cost you customers. If a new visitor to your website experiences a bug, they’re gone. If a customer experiences bugs in your service, eventually, renewal becomes questionable. So if you have a bug in production, time is of the essence. You need to fix it before anyone notices. Even our conservative estimate of 50% reduced time to fix production errors can mean the difference between acquiring a new customer or keeping your customers and losing them.
Fewer support calls
If a new visitor to your website experiences a bug, they are not likely to call your support services. We just agreed that they would simply leave. But a customer who has already paid for a subscription will certainly burden your support system with a new ticket. They paid; they want everything to work perfectly. So if something goes wrong (and whatever you do, something WILL go wrong) in production, time is of the essence. If you can detect the glitch and fix it BEFORE your customers notice, that support ticket triggering a chain of activity can be avoided.
Less Dev/QA friction
Here’s a scenario that happens in almost every development organization.
QA detects a showstopper on the Staging environment just before a release candidate is deployed to Production. This candidate was supposed to fix a critical bug found earlier in Production. Sirens sound, and lights flash. The whole development organization is mobilized because this release HAS to go out ASAP.
QA starts filling out a bug report, describing the scenario, and collecting logs. Once the bug report is ready, it’s passed on to one of the developers. And then, there’s the old, “Hey, it works on my machine.”
Dev and QA the engage in a dialog to understand why the bug reproduces on Staging, but not on the development environment. After some back and forth between dev and QA (which can take days), the developer gets enough information to reproduce the bug and can finally get to work.
If QA could produce the perfect bug report that enables the developer to reproduce the error with no effort, not only would it hasten the fix, but dev and QA would also be best friends. The way Ozcode Production Debugger does this is by providing a single shareable link to a full-fidelity recording of the error with code-level debug information. The developer uses that link to access the Ozcode debugger window and time-travel through the execution flow of the error with radical observability into the code. In fact, the developer doesn’t even have to reproduce the error; he has the recording of the original error as it occurred on Staging through a single shareable link.
Compliance with privacy regulations
You do business internationally and have customers from all over the world. Great. But this also means that you must comply with a variety of privacy regulations. When it comes to debugging, these regulations are an impediment. You can’t provide your developers with all the debug information they need because different user data (specifically PII – Personally Identifiable Information) is interspersed on your Production environment. So you start looking for solutions like reproducing the bug in a development environment or using fake data. Still, any path you choose to overcome privacy regulations only serves to lengthen the time it will take to solve the bug.
A Production Debugger lets you remove this impediment at the source. Ozcode Production Debugger lets you protect PII by defining the data types that should be hidden and redacting it before they get to the developer. You can specify anything from identifier names to regex patterns, and even complete classes and namespaces. This ability strikes the optimal balance between the need to provide your developers with as much information as possible and the requirement to hide PII from them. While debugging without seeing redacted data is not ideal, this is as good as it gets when you must comply with privacy regulations.
The bottom line
In the end, it boils down to paving the smoothest and quickest path to resolving production issues. The less time your developers spend fixing bugs in Production, the more time they have to serve your business and your customers by crafting more awesome code. Your whole development organization will be happier – from the developers building great capabilities, to QA testing them through to your DevOps teams who are deploying the code to production. Your customers will be happier because they will receive all this great new functionality more quickly and skip glitches that do get through to production (because you manage to fix them before they notice). Your business stakeholders will also be happier because, ultimately, these are the things that affect your bottom line.