4 Awesome Tools for .NET Debugging in Production

Advanced .NET Debugging - Ozcode
Debugging .NET systems in Production is hard, but there are many tools out there to help you. Learn about the top tools to fix Production bugs fast.

Modern IDEs provide very sophisticated tools for .NET debugging. We take it for granted that we can hit F5 to start a debug session, and F10/F11 to step over or into lines of code, fully expecting the IDE to display all variables in the vicinity. The truth is that amazing technology lies behind these capabilities, and in Development, we have everything at our fingertips. Not so for Production.

In Production, our ability to debug is hampered in many ways.

Reproducing the error: It’s usually anywhere between hard and impossible to reproduce a Production error.

Log files: We rely heavily on logs, but log files don’t cut it. You never have the right log entry where you want it, so you have to guess the solution, add more logs, build, deploy, rinse, repeat until you get what you want. And even then, logs are widely dispersed among different files, so it’s hard to find the relevant ones and put them together.

Breakpoints: Putting breakpoints in your production code is problematic. It causes your server to stop executing, which means anyone trying to access your application would not get a response.

Source code version: Matching your symbols and source code version to Production is not trivial. You’re usually several builds ahead on your development machine, not to mention that in Production, the code is optimized, so even if you could reproduce the right source code version, you might not be able to stop on breakpoints or see many of the local variables.

Microservices and serverless: With these technologies, not only do you have to grapple with the above, but there are added complications. With microservices, you have to debug across several processes, and in serverless architectures the offending code or scenario that you’re trying to debug is running one moment and gone the next.

Personally Identifiable Information: Production systems often contain sensitive information such as credit card numbers, social security numbers, and a host of other personal information that is heavily guarded by strict privacy laws. You’re supposed to debug Production systems without seeing any of this data.

Access to Production systems: There are different scenarios in which you don’t even have access to your Production systems. Consider, for example, that your customer reports a bug, but it happens on their desktop or on-premises infrastructure to which you have no access.

And yet, Production bugs happen in many shapes and forms, and being able to debug them is critical to your business. Take the classic example of the shopping cart functionality in an eCommerce website. Things can get much more subtle than that, and even small UI glitches can send your customers over to your competitors’ websites.

And yet, all is not lost. Debugging in Production is possible.

This blog is based on a webinar I recently gave where I showed some great .NET debugging tools and techniques you can use on your Production systems.

Scroll down to the end to view the full webinar recording.

The types of bugs you can encounter

There are several types of bugs that hit you in Production. Here are some of them:

  • Failed Requests
    Your client app sends a request to the server, but then something goes wrong, and you don’t get a response. Imagine a customer wanting to check if you have an item in stock and not getting a response.
  • Crashes
    Your whole IIS server or desktop application just disappears. We’ve all seen this – one moment it’s there, then, suddenly, it’s gone.
  • Logical bugs
    These may not throw an exception, but something won’t add up in the back-end logic. A value will be miscalculated, or the wrong action (or no action) will be performed. This can be manifested in any number of ways.
  • Hangs
    In the case of desktop applications, it’s very clear when something hangs. Your application won’t respond. Nothing moves. Your window is frozen. It’s frustrating but clear. But there are more subtle cases. For example, in an ASP.NET application server, if a request hangs instead of returning a response, the client will eventually time out and continue according to its handling of that situation. However, the server hang will cause more and more request threads to get stuck. You start seeing requests that, on the face of it, function perfectly, but the server is so flooded with hung threads that it can’t respond or responds slowly. Seemingly, a performance issue, but rooted somewhere completely different from where it is manifested. Eventually, you might restart the server, which will release all those hung threads, but the problem will only rear its ugly head again sometime soon – one of those annoying issues that are  really hard to reproduce.
  • Memory issues
    There is a variety of memory issues you may encounter. If your application keeps consuing memory, it will eventually crash with an OutOfMemoryException, but it can get much more subtle than that. If your garbage collector doesn’t clean up running objects from memory, they may still be executing code causing wrong behavior, and the garbage collector will be working overtime and affecting performance.
  • Performance issues
    Hangs and memory issues often manifest as poor performance, but so will slow network or database requests or faulty caching functionality.  

.NET Production Debugging tools of the trade

While .NET Debugging is not as straightforward in Production as it is in Development, there are many tools at your disposal.

In this post, I’ll go over the following:

  • Windows Event Viewer
  • Dump files
  • Dedicated Debuggers (Visual Studio’s Debugger, dnSpy, WinDbg)
  • Production Debugger (Ozcode)

Wait, there’s more. 
There are, of course, more tools you can use, but to keep the length of this post sane, I won’t be covering:

  • Decompilers
  • Log analyzers
  • Performance Counters
  • ETW Events
  • Memory & Performance Profilers (dotTrace, PerfView, ANTS)
  • Application Performance Monitoring (APM) tools (Application Insights)
  • Error monitoring tools (Raygun, Application Insights)

In going through these tools, I’ll be using my reference ASP.NET Core 3.1 application, a simple price calculator that takes an input price, and calculates the output price which may be in a different currency and include a coupon code and/or an additional discount. Not much more than your standard “Hello World” with a simple UI.

Of course, I used my best coding skills to plant some neat bugs in this application.

Let’s do some debugging.

Ozcode Production Debugger - START FOR FREE

.NET Debugging with Windows Event Viewer

Event Viewer shows a log of different system and application messages such as errors, warnings, and information. It comes built into the Windows OS and has been around since the release of Windows NT in 1993, so it’s readily available and easy to use. Let’s see how Windows Event Viewer can be used to get to the bottom of an application crash or a failed request.

When trying to convert $200 from “USD” to “EUR,” a request fails with a 500 Internal Server Error.
You don’t want that happening in your shopping cart now, do you?

Let’s Remote Desktop to the machine on which my server is running. ASP.NET logs all failed requests so I can pop up Windows Event Viewer to see what happened. Under Windows Logs | Applications, I can find the corresponding error log. The beauty of this is that it doesn’t require any setup. It’s just built into my Windows, and I can easily open it and access a lot of information.

We can see that the failed request path (basically, the request’s URL) was “/Price/Calculator/Calculate”, and below that, the stack trace shows that the value “eur” was not found. That already tells me a lot. There’s a mismatch between the currencies my UI allows and the ones my server expects. But where is that defined in my server?

So, here’s a neat trick.

I can just copy the call stack, and if I have JetBrains Rider or Visual Studio with ReSharper open on the corresponding project, it gets magically pasted into the Stacktrace window in my IDE. Then, with a few clicks up the call stack, it’s easy to find that the offending “EUR” value in my UI should have been “EURO”, and that’s what caused the exception.

Watch it in action.

The things you can do with dump files

When the call stack and exception details you can get from the Event Viewer don’t provide you with enough information, dump files may come to the rescue. These files are huge because they hold the entire memory heap of a process that has crashed, but they can provide critical information needed to understand what happened. There are different ways to create a dump file, and I recommend using ProcDump, a small but powerful Windows Sysinternals tool that you can download. First, make sure you restart IIS and check that your application is running. Then, to create your dump file, run this in your console:
procdump64.exe -ma -e [your-process-name]
-ma: get the full memory dump needed to debug .NET Processes
-e: to monitor unhandled exceptions (which is what has happened if your application crashed)if your application
Now reproduce the crash, and ProcDump will create the dump file, which you can find in its installation folder. There are different ways to analyze dump files. You could use WinDbg again, but that’s kind of complicated for several reasons I won’t go into right now. I recommend copying the dump file to your development machine and analyzing it in Visual Studio, which is open on the same project that created the executable. When you open a dump file in Visual Studio, at first, you just get some general information such as the process name, its architecture, loaded modules, etc.

Under the Actions menu, you have different options for debugging.

Select Debug with Managed Only, and …

BOOM!

Not only do you get the exception details, but also the exact line of code that threw the exception. You can view local variables in scope, see the different threads, and even travel up the call stack and see the local variables for each call. This is a great debugging experience similar to what you get in Development.

But here’s the catch.

To get this great Dev-like debugging experience, you need to have the same source code that created your production executable loaded into Visual Studio. In practice, by the time you’re debugging an exception in Production, your codebase is going to be several builds ahead. To recreate the exact build, you need to go back to the exact Git version used, with all the same build parameters, etc. If a single character is off, this method won’t work. But there’s another problem. Production code is usually optimized, and this can really degrade your debugging experience. The JiT compiler inlines local variables, so you can’t map them to the source code in your Visual Studio project.

No locals. Bummer!

But the JiT compiler also changes code, so you may not even see the line of execution that threw the exception. (Sigh!)

Ozcode Production Debugger - Start for FREE

Not your usual .NET Debugger

If I shout, “debugger,” the first association that will come to your mind will probably be Visual Studio or JetBrains Rider. Well, I’d like to introduce you to another neat .NET debugger called dnSpy. The great thing about this neat little tool is that it lets you debug your code as it’s running live in your production environment while giving that IDE experience. Let’s see how it works. Believe me when I tell you my reference app has a logical bug. When you try to convert $100 to GBP, it gives you GBP 127. OK, you don’t believe me? Just watch.
This is wrong. The conversion should be the other way around, so either I need to adjust my exchange rates or the formula that calculates the final price. Let’s debug this using dnSpy. I’ll run dnSpy on the server machine running my worker process. Under Debug | Attach to process, I select w3wp.exe Then, in the Modules tab at the bottom, I can find my dll, “AdvancedDebuggingTechniques.dll,” and browse the project files. The beauty with this tool is that I don’t have to match anything to my source code. It works right on production code by decompiling it and presenting it to me in human-readable form. Then, I can place breakpoints in the code and debug it just like I’m used to doing in my favorite IDE. So, I’ll put a breakpoint in the location I suspect is causing the bug, and try to reproduce it (which is uncommonly easy in my contrived reference app.) But here’s where the beauty fades into ugliness. Since I’ve put a breakpoint in my live production code, when I reproduce the bug, my program stops executing (for me and anyone else accessing it). This is not viable for many production systems. Nevertheless, I can now see the code at my breakpoint and start using F10 and F11 to step over and into my code until I find the offending line that needs fixing. Watch.

All other .Net debugging tools rolled into one

The last tool I’d like to show you in this post is Ozcode Production Debugger. It provides the capabilities of the other tools, but doesn’t require any correlation to your source code, and doesn’t interfere with the running of your application in production.

After connecting Ozcode’s agent to your running application, exceptions will just magically appear on the Ozcode dashboard. The screenshot below shows the two exceptions I described earlier in this post (an SSL exception, and the ArgumentException with the illegal “eur” value that caused a crash. All you need to do is click Debug to debug them.

Once you drill down into an exception, you get a lot of information:

  • The line of code that threw the exceptions (i.e., where the application crashed)
  • You can travel back through all the methods of the call stack.
  • Examine locals variables at every stage…all the way up the call stack

So far, we have all the information you can get from dnSpy or from a dump file – but without really having to strain yourself to get it.

But it gets better.

You can also see the latest thousand log entries and even focus on the ones generated by the exception context.

And when the culprit is an HTTP request, you can see the incoming request – along with its headers, request body, as well as the outgoing request, query params, and all.

Let’s see it in action.

Pros and cons side by side

So, let’s put all of this together and compare these four fine tools. The real beauty is that you can pick and choose the best tool for the situation, and even use them together. None of them interfere with each other. The table below shows how each of these tools deals with failed requests and crashes:

Event Viewer

Dumps

dnSpy

Ozcode

Shows exception information

Yes

Yes

Yes

Yes

Stops server activity

No

No

Yes

No

Shows local variables

No

When code isn’t optimized

When code isn’t optimized

Yes

Shows source code

No

When matching symbols and source code

Yes (decompiled)

Yes (decompiled)

Shows execution line

No

When matching symbols and source code + code can’t be optimized

When code isn’t optimized

Yes

Stops latest logs

No

No

No

Yes

Stops HTTP requests and database queries

No

No

No

Yes

And as promised, here's the full webinar

Ozcode Production Debugger

Autonomous exception capture
Don’t chase after exceptions. They automatically appear in your dashboard.

Code-level observability
Get insights into your code down to the deepest levels.

Time-travel fidelity 
View the true state of your code anywhere in the execution flow of an error.

Ozcode Production Debugger - START FOR FREE

Michael Shpilt

Comments

Keep your code in good shape!

Subscribe to our blog and follow the latest news in the .NET debugging industry

Ready to Dive into Your Prod Code?

Easy debugging with full time-travel data

Share on facebook
Share on twitter

Recent Posts

Follow Us

Join Ozcode YouTube Channel

Let’s start debugging, it’s free!

Ozcode Logo

This website uses cookies to ensure you get the best experience on our website.