How to Capture and Analyze Memory Dumps from .NET Core Applications in Linux Containers on AKS

Prerequisites

Before we get started, ensure that:

• You have a .NET Core application running in a Linux-based container on AKS.
• You have access to the Azure Kubernetes Service and can execute kubectl commands.
• You are familiar with basic Kubernetes and Docker commands.

Why Capture a Memory Dump?

Memory dumps are useful for diagnosing issues like:

• Memory leaks
• Unmanaged memory usage
• High memory consumption

A memory dump captures a snapshot of the application’s memory at the time of the dump, which can be crucial for analyzing performance problems and debugging.

Step 1: Prepare Your Container

In your Linux-based container, you’ll need to install a few tools to facilitate the process. Let’s begin by ensuring the necessary tools are installed:

apk update
apk add bash
apk add wget

These commands install bash (a shell that makes it easier to execute commands), and wget (for downloading tools like the .NET SDK).

Step 2: Install .NET Core SDK in Your Container

Once the container is prepared, we need to install the .NET SDK, which includes diagnostic tools like dotnet-gcdump and dotnet-dump.

wget https://aka.ms/install-dotnet-sdk-ubuntu-x64
bash install-dotnet-sdk-ubuntu-x64

This will install the latest .NET SDK in your container. You’ll use this SDK to capture memory dumps.

Step 3: Capture the Memory Dump

Now it’s time to actually capture the memory dump. You have two main options:

Option 1: Capture a GC Heap Dump with dotnet-gcdump

dotnet-gcdump is lightweight and designed specifically for capturing memory dumps that focus on the garbage collector’s heap. This method is best for identifying issues related to memory pressure in the .NET heap.

dotnet gcdump collect --process-id <your-process-id>

Replace <your-process-id> with the ID of the process you want to capture the dump from. You can get the process ID using:

ps aux | grep dotnet

This command will generate a .gcdump file, which you can later analyze.

Option 2: Capture a Full Memory Dump with dotnet-dump

If you need a more comprehensive memory dump, use dotnet-dump. This will capture the entire memory space of the process, including both managed and unmanaged memory.

dotnet dump collect --process-id <your-process-id>

his command creates a .dmp file, which can be used to analyze all memory segments of the application.

Step 4: Download the Memory Dump from the Pod

Once the dump has been created, you’ll need to download it from the AKS pod to your local machine for analysis. Use kubectl cp to copy the dump file from the pod:

kubectl cp -n <namespace> <pod_name>:/root/.dotnet/tools/dumpresult1.gcdump ./dumpresult1.gcdump

Replace <namespace> and <pod_name> with your actual Kubernetes namespace and pod name. This will copy the dump file to your local machine.

Step 5: Analyze the Memory Dump

Now that you have your memory dump, it’s time to analyze it. You can use Visual Studio or other diagnostic tools to examine the dump file. For Visual Studio:

1. Open Visual Studio and go to File > Open > Dump File.
2. Navigate to the .gcdump or .dmp file and open it.
3. Visual Studio will show you a detailed analysis of the memory, including objects in memory, allocations, and references.

For dotnet-gcdump files, you can also use the dotnet-gcdump tool to analyze the dump:

dotnet gcdump analyze dumpresult1.gcdump

This command helps you to inspect the contents of the GC heap dump.

Step 6: Clean Up

Once you’ve completed the analysis, remember to clean up the container environment:

1. Remove the .NET SDK and any diagnostic tools you installed.
2. Ensure your container is secure by deleting any unnecessary files.

Troubleshooting Common Issues

• File Corruption During Download: Sometimes, downloading large dump files with cat can result in corruption. Use kubectl cp instead to ensure the file is transferred correctly.
• Permission Issues: Ensure the container has the necessary permissions to create and read the dump file. You may need to run the container as a user with elevated privileges.
• Missing .NET SDK: If wget fails to download the .NET SDK, check for network issues or use an alternate source for the installer.

Conclusion

Capturing and analyzing memory dumps is a powerful technique for diagnosing performance issues in .NET Core applications running on AKS. By following the steps outlined in this guide, you should be able to efficiently capture memory dumps and investigate any memory-related problems. Make sure to clean up your container after finishing the diagnostics to maintain security and performance.

For further reading, check out the official Microsoft .NET Diagnostics documentation, which offers additional insights into memory analysis and troubleshooting.