Documentation

Introduction

What is Ozcode PD?

Ozcode PD (Production Debugger) is a debugging solution that enables you to quickly identify and fix errors across different environments such as Production, Staging, and QA, both on-premises and in the cloud. By recording code execution flow and enabling time travel debugging, Ozcode Production Debugger provides radical observability into exceptions as they occur in runtime systems, removing the need to reproduce errors. This enables a quick and effective root cause analysis to determine why exceptions are thrown so you can rapidly resolve errors before they impact your business.

Who should use Ozcode PD?

Ozcode Production Debugger serves professionals in different roles throughout the software development lifecycle.

  • Developers Developers at every level, from junior engineers to senior software architects, enjoy a new debugging experience in a familiar Visual Studio-like environment. There is no longer any need for hands-on coders to waste time trying to reproduce bugs or recreate the environment in which an error was captured. Ozcode Production Debugger records the full code execution flow with full fidelity time travel debugging, visibility into local variables, method parameter and return values, network requests, database queries, all relevant log entries, and even a timeline of events like HTTP requests. With that information at their fingertips, developers can pinpoint the exact moment of failure and understand the root cause of the problem for a quick resolution.
  • QA Ozcode Production Debugger notifies QA professionals of exceptions that occur in runtime systems and provides all the information needed to create the perfect bug report. Rather than having to collect debug information from separate sources such as log files, screen captures, database queries etc., Ozcode Production Debugger provides all the relevant information through a single shareable link. All QA needs to do is share that link with the developer. And since the developer does not need to reproduce the bug, or recreate the environment in which it occurred, there is no friction between the developer and QA as the developer debugs the exact version of the source code that manifested the error.
  • DevOps/SRE Ozcode Production Debugger provides DevOps practitioners and SREs with tools to maintain application health and reduce MTTR from production errors. Ozcode can be included in the CI/CD pipeline to establish quality gates based on exceptions caught and prevent faulty deployments from being promoted up the pipeline to production.  Once an exception does occur, Ozcode bridges the gap between Dev and Ops using effective communication enabled on the collaboration panel in the Debugging Screen for rapid resolution of errors found in production and pre-production environments.
  • Business Stakeholders Business stakeholders are the ones who are ultimately responsible for the quality of their applications and the service they provide to their customers. Ozcode Production Debugger provides business stakeholders with a clear picture of application health letting them monitor service quality to identify and fix errors before customers notice anything is wrong. When errors do get through to production systems, Ozcode minimizes MTTR by accelerating root cause analysis of the bug for quick and final fix.

Key Features

Ozcode Production Debugger is built to run on production and pre-production environments offering the key features described below.

For Developers

Autonomous Exception Capture The Ozcode Production Debugger agent automatically records the code execution flow leading up to any exceptions that are thrown, so there is no need to reproduce the error or recreate its environment. The flow is provided through a link that gives a direct view to the code that threw the exception. Radical Observability View all the information needed for a root cause analysis including local variables, method parameter and return values, network requests, database queries, relevant log entries and timeline of events leading up to an exception including HTTP requests. You can even trace events across microservices. Time-Travel Pinpoint the exact moment of failure as it threw the exception and see the value of all variables and method calls for every line of code leading up to the exception across the whole call stack.

For QA

Autonomous Exception Capture QA professionals also benefit from Ozcode Production Debugger’s autonomous exception capture since errors automatically show up in the dashboard as soon as they occur. From the dashboard, it is easy to track which application versions exhibit the error to detect regressions and evaluate bug severity by the frequency of exceptions it generates, and the number of customers it has already affected. Perfect Bug Reports With all error information automatically assembled by Ozcode into a single link, QA professionals don’t need to work hard to find the relevant  logs, network and database requests, relevant code version of each component and anything else needed to create the perfect bug report. They can just share the link provided by Ozcode with developers and dispel any question of reproducibility

For DevOps/SRE

System Health at a Glance DevOps practitioners, Release Managers and SREs responsible for moving deployments up the CI/CD pipeline and monitoring and maintaining system health get a high-level view of all exceptions through one convenient dashboard. A simple click traces exceptions back to the first deployment in which they were introduced and shows which specific services were involved in an error. Full Team Collaboration Since all debug information is assembled into a single shareable link, all members of the team, from Dev through Ops work on the same data and code version to resolve errors in any of the production or pre-production systems. With everyone looking at the same data, the collaboration panel keeps communication tight and on point.

Devops Platform

Supported Platforms

Ozcode Production Debugger currently supports debugging programs written in C# on any variant of .NET platforms deployed on Windows or Linux with any cloud provider or on-premises as follows:

  • Microsoft Azure – Windows
    • Azure App Service
    • Service Fabric
    • Azure Functions
    • Virtual Machine (IIS & Windows Services)
  • AWS – Windows
    • Virtual Machine (IIS & Windows Services)
  • Google Cloud Platform and other cloud providers – Windows
    • Virtual Machine (IIS & Windows Services)
    • Desktop (WPF and WinForms)
  • Enterprise data center – Windows
    • Virtual Machine (IIS & Windows Services)
    • Desktop (WPF and WinForms)
  • All platforms – Linux
    • Docker

For detailed installation instructions, please refer to Setting Up the Ozcode Agent.

System Architecture

  system architecture  

Production Debugging Agent

The Ozcode agent is installed alongside your application on the platform on which it is running. When an exception is thrown, the agent performs its instrumentation on the relevant parts of the code and records the code execution flow (For details, please refer to Capturing Exceptions). The Ozcode agent is self-regulating and imparts a negligible CPU overhead which can reach up to 3%, and minimal pressure on garbage collection, with no perceptible impact on performance of code execution, even when faced with a large surge of errors. The recorded error information is captured with a negligible memory impact and transmitted to the Production Debugging Server with a transmission rate that is capped at 10Mb per hour.

Production Debugging Server

You can use Ozcode’s SaaS service as your Production Debugging Server in a multi-tenanted configuration or install a dedicated Production Debugging server in your on-premises infrastructure. The Production Debugging Server exposes a web-based front-end experience that provides dashboards and an interactive IDE-like debugging experience. Customers can log-in to using their own credentials or via Azure Active Directory.

Getting started

Getting started with Ozcode Production Debugger involves 3 basic steps:

  • Creating an account and signing in
  • Creating an application
  • Installing the Ozcode agent

Your account

Before using Ozcode Production Debugger, you need to create an account. In some organizations, an administrator may create your account for you, in which case, you should receive an email with a link inviting you to create your password. If you already have an account for the Ozcode Visual Studio extension, you can simply use your existing account.

TIP: Invite your teammates

You can easily invite your teammates to create an account. Just click the icon in the top right corner of the screen and provide their email address. They will receive an email with instructions on how to activate their account.

Invite a teammate - Ozcode


If you already have an account, enter the credentials provided to you and sign in at
https://app.oz-code.com/sign-in.

Signing in

Creating an account

If you need to create an account, click Sign up.

In the following screen, provide your email address and fill in the details of your account.

After submitting your details, you should receive an email message to the address you provided. After confirming your email address, you will be taken to the Sign In screen.

Creating an application

An application in Ozcode Production Debugger represents the application that you want to debug. After signing in for the first time, you need to create your first application.

TIP
Each application will have its own dashboard in the Production Debugger. To distinguish between multiple deployments of the same app in different environments (such as QA, Staging, and Production), create a separate application for each environment, such as: “My App – QA”, “My App – Production” etc.

Give your application a name and click Create.

Setting up the Ozcode  agent

Ozcode Production Debugger supports deployment on different of cloud platforms, operating system and deployment type as described in Supported Platforms.

TIP – Detailed Installation Instructions
For detailed instructions on installing the Ozcode agent on any of the supported configurations, please refer to the table at the end of this section.

After creating your application, Ozcode Production Debugger will create a token which should be used when setting up the agent on your selected platform for deployment.  You will then be prompted to select your platform, OS and deployment type.  

After selecting your platform, OS and deployment type, Ozcode Production Debugger will provide instructions and any code snippets needed to deploy the agent for your selected configuration. Here are a couple of examples:

production debugger

Instructions for deploying the agent on Microsoft Azure, on a Windows machine as an Azure App Service

Instructions for deploying the agent on Microsoft Azure, on a Windows machine as a Service Fabric microservice

For detailed instructions on how to set up the Ozcode agent for all supported platforms and configurations, click the corresponding link in the installation table below.

Installation Table

OS Platform Windows Linux
Microsoft Azure

Azure App Service

Service Fabric

Azure Functions

Virtual Machine (IIS & Windows Services)

Docker
AWS Virtual Machine (IIS & Windows Services) Docker

Google Cloud Platform and
o
ther cloud providers

Virtual Machine (IIS & Windows Services)

Desktop (WPF and WinForms)

Docker
Enterprise Data Center

Virtual Machine (IIS & Windows Services)

Desktop (WPF and WinForms)

Docker

Verifying agent installation

For Ozcode Production Debugger to work correctly, your agent must be installed in the application you wish to debug, and communication with the Debugger server established. Until those two conditions are met, Ozcode Production Debugger will indicate that it is waiting for agent installation by showing this notification:

In addition, this icon () will be displayed in different locations on the Dashboard for any application that does not have the corresponding agent correctly installed. Once the agent is installed and communicating with the Debugger server, it indicates that the agent is running with the following notification:

Click on start debugging to get to the Ozcode Production Debugger dashboard.

Using Ozcode PD with an APM

Ozcode Production Debugger and Application Performance Monitoring (APM) technologies complement each other. These technologies operate on Production and pre-Production environments, and while an APM is focused on monitoring applications at a high level to help resolve performance issues,  Ozcode let’s you dig deeper to perform code-level analysis and solve bugs and logical errors. Ozcode Production Debugger can work in parallel with the following APM products:

  • New Relic
  • Application Insights
  • Dynatrace
  • DataDog
  • AppDynamics (private beta)
  • Stackify (private beta)

To use Ozcode with any of the above APMs, it is important to remember the following:

  1. Make sure your APM is installed and fully functional BEFORE you install the Ozcode Production Debugger agent.
  2. Once you have your APM and Ozcode Production Debugger working together, if you upgrade your APM, you need to remove and then reinstall the Ozcode Production Debugger agent.

Capturing Exceptions

Once the Ozcode Production Debugger agent is up and running, it autonomously monitors your application for exceptions and enables debugging with radical observability through two phases of exception handling. This two-phase approach is an optimization to ensure that the agent is not intrusive to your runtime code and has negligible impact on performance. Phase 1: Instrumentation The first time a new exception is thrown, the Ozcode agent adds lightweight instrumentation along the code path that led up to the exception. At the end of this phase, you are not able to debug the exception yet, but you do see a notification in the Ozcode Production Debugger Dashboard with visibility into:

  • the exception type
  • the exception message
  • the call stack

Phase 2: Recording and deinstrumentation The next time the same exception is thrown in the same location in the code, the Ozcode agent records and stores:

  • the line of code that threw the exception
  • the code execution flow across the complete call stack
  • local variables, method parameter and return values
  • the latest log entries leading up to the exception
  • network requests
  • database queries
  • in ASP.NET application: http requests (both the incoming request that led to the exception and any outgoing requests to external services)

Once the above information has been recorded and stored, the Ozcode agent removes all the instrumentation that was added in phase 1. Henceforth, for subsequent occurrences of the same exception with the same call stack , the exception count for the corresponding entry in the Dashboard will be incremented, but the code execution flow will not be recorded again. This means that whenever you drill down into the exception to debug it, you will be seeing the code execution flow of the second time the exception occurred.

COMING SOON In future versions, you will be able to request a recapture of an exception to enable Ozcode Production Debugger to add its instrumentation again and re-record the code execution flow for a subsequent occurrence of the exception.

Exception and Error Types

Ozcode Production Debugger supports debugging for the following exception and error types:

  • Failed requests: Failed HTTP requests, WCF requests, and gRPC calls
  • Crashes: any exception that led to a crash
  • Swallowed exceptions: any exception that was caught but not logged anywhere and did not lead to an HTTP failed request or a crash.
    For example:
    
        try         
        {        
            throw new ArgumentException("My exception"); 
        }        
        catch {}
  • Logged errors: All ERROR or FATAL level errors. The following logging tools are supported:
      • Log4net
      • NLog
      • Serilog
      • Microsoft ASP.NET Core logger (Microsoft.Extensions.Logging).

Here’s an example of how you might catch logged errors:


    try         
    {  
        SomethingThatThrowsException();         
    }         
    catch (Exception ex)         
    {  
        Logger.Error(ex, "Operation failed here for some reason");  
    }
    If you’re not using any of the above logging tools, you can use the Ozcode NuGet package to log exceptions.
NOTE
By default, swallowed exceptions are filtered out because they can flood the dashboard with exceptions that are not relevant to debugging your application such as those that occurred in 3rd party code.  To start seeing swallowed exceptions in your dashboard, set the corresponding filter as described under Filtering Exceptions.

Usage

There are two primary usage modules for the Ozcode Production Debugger:

  • The exception dashboard
  • The debugging screen

The Exception Dashboard

The Exception Dashboard is where you land when you log into the Ozcode Production Debugger. It provides a high-level view of your application health by displaying the number and variety of exceptions that the Ozcode agent has detected in your application and sent to the server which displays the dashboard. The Exception Dashboard is divided into the following panels:

  • Errors over time: a timeline showing the number of errors that occurred during the selected time frame.
  • Errors: the list of errors detected
  • Call stack: the call stack leading up to the error
  • Code: the code snippet where the exception was thrown

The Call stack and Code panels are only displayed once a specific error has been selected in the Errors panel. Exception Dashboard - Ozcode

Errors over time panel

The Errors over time panel displays a graph showing the number of errors detected over time. By default, errors are shown one month back. To change the time frame, select one of the preset buttons above the panel. Time frame - Ozcode

You can set a custom time frame for the panel using the Custom button.

Custom time frame - Ozcode

Or just drag your mouse across the time frame you are interested in.

Custom time scale - Ozcode

Click for animation

You can also hover over any part of the graph to see the number of errors reported at a specific point in time. Hover over Errors over Time panel - Ozcode

Errors panel

The Errors panel displays the list of errors detected by the Ozcode Production Debugger agent in the selected application according to your selected time frame. Errors panel - Ozcode Checkbox: Select an error in the list to see when it occurred in the Errors over time panel Exception Type: Indicates which of the supported error types occurred Message: The message associated with the error Last Seen: Indicates the last time that the error occurred Count: The number of times this specific error occurred Debug:  Click Debug to debug the selected error. Ozcode Production Debugger will display the corresponding debugging screen Capturing: If the button displays Capturing button - Ozcode , it means that the selected error has only occurred once and the Ozcode agent is still in the Instrumentation phase described under Capturing Exceptions. At this point, you cannot debug the error yet, but can see the call stack and the code snippet where the exception was thrown. Click the Resolved icon (Resolved icon - Ozcode) to indicate the selected error has been resolved and remove it from the Active errors list. Click the Ignore icon (Ignore icon - Ozcode) to indicate that you want to ignore this error and remove it from the Active errors list. Use the Errors/Operations toggle to switch between viewing only errors, or also viewing the operations that triggered the errors. Operations panel - Ozcode

 

TIP
The search box can be used find a textual match in the Exception Type or Message fields

 

Filtering Exceptions

Use the filter to display specific exception types.

Filtering exceptions - Ozcode

Use the drop list to switch between Active, Resolved and Ignored errors.

Exception status - Ozcode

Call Stack and Code panels

These panels display the corresponding call stack and location in the code that threw the selected exception.

Call stack - OzcodeCode panel - Ozcode

The Debugging Screen

The second time an exception is thrown, the Ozcode agent records the code execution flow together with the relevant associated data enabling you to perform time travel debugging to resolve the error and deploy a fix. To debug an error, click it in the Errors panel or in the Code panel. The Ozcode Production Debugger takes you to the corresponding Debugging screen. Debugging screen - Ozcode The Debugging Screen is divided into the following panels:

  • Exception information: general information about the selected exception
  • Timeline: a timeline of events/network requests leading up to the error
  • Code/Logs: toggle between viewing log entries leading up to the exception, and a set of sub-panels related to the code execution flow
  • Call stack: the call stack leading up to the exception
  • Locals:  local variables relevant to the selected context
  • Code editor: the relevant code snippets corresponding to the call stack
  • Collaboration: a messaging board on which all users who have been invited to participate in debugging the selected application can communicate.

Exception Information panel

The Exception Information panel provides general information about the selected exception:

  • Count: the number of times this exception has been thrown
  • Exception: the fully qualified exception class
  • Last seen: the most recent occurrence of this exception
  • First seen: the first occurrence of this exception
  • Versions: the different code versions in which this exception was detected

Timeline panel

The Timeline panel displays the sequence of network requests and database queries that led up to the error. Events are color-coded to indicate failed and successful requests as well as exceptions according to the legend in the top right corner. Timeline panel - Ozcode Selecting any of the events in this panel creates a tab in the Code/Logs panel that displays detailed information about the event. Detailed event information

  • General: for database queries, this tab displays information about the query including general information such as the name of the database server and database name
  • DB Bind Variables: parameters needed to bind to the database
  • Query: the full SQL query

Code/Logs panel

This panel displays a set of sub-panels related to the code that threw the exception.

Call stack panel

This panel displays the call stack leading up to the exception. Select any element in the call stack to display the corresponding code segment in the Code editor panel Call stack - Ozcode

Locals panel

This panel displays the names and values of local variables and objects in the context of the currently selected stack frame, at the moment the exception occurred – also known as execution point (the line of code marked with a green background), just like in the Visual Studio debugger.

Locals panel - Ozcode

Code editor panel

This panel displays the code execution across the different stack frames that led up to the particular exception. Unlike the Locals window, which shows values relevant to the execution point (the line of code highlighted with a green background), the code editor visualizes the values of each variable, method return value, and property getter, at the time it was evaluated, essentially displaying the same level of information that would be available if you had manually “stepped over” each line of code in a traditional debugger. We refer to this as “time travel fidelity.” The Code Editor panel provides a “heads-up display” for the debugger which provide insights into the code execution flow through different visual elements including:

  1. Every method return value, local variable, and property-getter will have a value adornment presenting its value. Hovering over the expression will open a Data Tip window just like in a regular debugger.
  2. Every expression that was evaluated to true or false will be highlighted in green or red accordingly.
  3. Expressions that were short-circuited and, therefore, not evaluated will appear as strikethrough text.
  4. Every conditional code branch (such as an if/else block, a switch block, etc.) that was not traversed will be grayed out.
  5. The expression that caused the exception to occur will be displayed in red.

Code editor panel - Ozcode

Collaboration panel 

The Collaboration panel facilitates focused and effective communication between team members working together on resolving an issue. Any team member invited by the application’s administrator can participate and add a comment in the panel. To address a specific team member, “mention” their name starting with an ‘@’ sign. The panel will perform name completion according to members in the team. When you add a “mention” in the Collaboration panel, the “mentioned” team member will receive your message by email, with a link to open the Ozcode Production Debugger in the same context you were looking at when sending the message. Collaboration panel - Ozcode

Point-in-time links

Point-in-time links let you direct a collaborator to a specific place in the code. Hover over a variable to display its name and value. Now hover over the name-value popup to display the point-in-time icons.

Point-in-time popup - Ozcode

  • Click this icon ( Point-in-time comment - Ozcoce ) to add the corresponding variable name to the message field in the collaboration panel. The name is added as a link which when clicked, takes you to the this exact location in the code.
  • Click this icon ( Point-in-time link - Ozcode )to copy a link to this exact location in the code to the clipboard so you can send it via any external means of communication you may be using.
  • Click this icon ( Point-in-time value - Ozcode ) to copy the value of the corresponding variable to the clipboard.

Here is an example of a message you could send:

Point-in-time message - Ozcode

In this example, in addition to receiving an email with a direct link to the same place in the code, Michael will also see the message in the Collaboration panel and can click on testClassName to get to that location.

Uninstall

For instructions on how to remove the Ozcode Production Debugger agent, click the link corresponding to your installation type below:

Azure App Service

To remove the Ozcode Production Debugger agent from an Azure App Service, follow the steps below:

  1. Log in to the Azure Portal
  2. Stop your App Service
  3. Go to App Service -> Extensions
  4. Select  Ozcode Production Debugger and press delete.
  5. Start your App Service

Azure Functions App

To remove the Ozcode Production Debugger agent from an Azure Functions App, follow the steps below:

  1. Log in to Azure Portal
  2. Stop your Functions App
  3. Choose Platform features and then Site Extensions
  4. Select Ozcode Production Debugger and press delete.
  5. Start your App Service

IIS and Windows Services

To remove the Ozcode Production Debugger agent from an IIS or Windows Services App, follow the steps below

  1. Run the Ozcode installer again. The installer will display the Welcome screen.
    Installation Wizard - Ozcode
    Click Next.
  2. The installer will display the Modify, Repair or Remove installation screen.
    Modify/Remove - Ozcode
    Select Remove and click Next.
  3. The installer will display a confirmation screen.
    Confirm remove - Ozcode
    Click Remove.
  4. The installer will indicate completion of the removal process.
    Removal complete - Ozcode
    If you are using IIS, make sure to check the Restart IIS checkbox. The Ozcode agent will not be fully removed until IIS is restarted.
    Click Finish to complete removal of the Ozcode agent.

Desktop Application

To remove the Ozcode Production Debugger agent from a desktop application follow the instructions for removing it from IIS and Windows Services.

Docker Image

To remove the Ozcode Production Debugger agent from a Docker image, simply remove the lines added to the dockerfile when you added the Ozcode agent to your Linux container.

Troubleshooting

Please refer to the troubleshooting steps for your installation type using the corresponding links below:

Azure App Service

Check your log files

Using Kudu, you can find Ozcode’s logs under \LogFiles\SiteExtensions\OzCode.ProductionDebugger.AzureSiteExtension. If you’re having trouble, send us your logs to support@oz-code.com. We’d love to take a look and try to help!

Disable Developer Mode in .NET Core

If you are using Developer Mode or calling app.UseDeveloperExceptionPage(), the developer-friendly exception page that ASP.NET Core provides may swallow the exceptions that were thrown by your services, and prevent Ozcode from intercepting them. To prevent this issue, avoid running your website in Developer Mode on Azure.

Using an Application Performance Monitoring tool

If you are using an APM, such as New Relic, Dynatrace, DataDog, Stackify or Raygun alongside Ozcode, please see Using Ozcode with an APM for troubleshooting steps. If you’re having any trouble setting up Ozcode with your APM, email support@oz-code.com and one of our integration engineers will be happy to help set up your integration.

Azure Functions App

Using Consumption Plan

Ozcode Production Debugger currently only supports Function Apps that use the App Service plan. Consumption plan is currently not supported.

Check your log files

Using Kudu, you can find Ozcode’s logs under \LogFiles\SiteExtensions\OzCode.ProductionDebugger.AzureSiteExtension. If you’re having trouble, send us your logs to support@oz-code.com. We’d love to take a look and try to help!

Disable Developer Mode in .NET Core

If you are using Developer Mode or calling app.UseDeveloperExceptionPage(), the developer-friendly exception page that ASP.NET Core provides may swallow the exceptions that were thrown by your services, and prevent Ozcode from intercepting them. To prevent this issue, avoid running your website in Developer Mode on Azure.

Using an Application Performance Monitoring tool

If you are using an APM, such as New Relic, Dynatrace, DataDog, Stackify or Raygun alongside Ozcode, please see Using Ozcode with an APM for troubleshooting steps. If you’re having any trouble setting up Ozcode with your APM, email support@oz-code.com and one of our integration engineers will be happy to help set up your integration.

IIS and Windows Services

If the Ozcode agent setup fails, the corresponding installer screen will include a checkbox offering to display logs which may indicate what caused the failure.

Ozcode Installation failed

You can also send logs found under %programdata%/OzCode/logs to support@oz-code.com. We’d love to take a look and try to help.

Using an Application Performance Monitoring tool

If you are using an APM, such as New Relic, Dynatrace, DataDog, Stackify or Raygun alongside Ozcode, please see Using Ozcode with an APM for troubleshooting steps. If you’re having any trouble setting up Ozcode with your APM, email support@oz-code.com and one of our integration engineers will be happy to help set up your integration.

Docker

Support for Alpine Linux

Docker images that are based on Alpine Linux are not currently supported, but support is coming soon.

Issues obtaining wget

If your docker image doesn’t support wget add the following line of code at the top of your dockerfile:

RUN apt-get update && apt-get -f install && apt-get -y install wget

Getting Support

Feel free to get in touch with us for support in any of the following ways:

 

Table of Contents

Ozcode Logo

This website uses cookies to ensure you get the best experience on our website.