Debugging DllNotFoundException on Linux and Containers or DLL Hell in 2023

by Ilya Verbitskiy

Today, I want to share my experience finding a bug you rarely see in .NET applications.

As you know, .NET uses managed code to run our applications. .NET managed code is any code written to run under the supervision of the Common Language Runtime (CLR), which is the heart of the .NET Framework. When you build a C# program, it is compiled into an Intermediate Language (IL), not into machine-specific code. The IL code is then compiled into native code by the Just-In-Time (JIT) compiler of the CLR at runtime. This approach allows safety, cross-platform support, and cross-language integration. For example, because all .NET languages compile to the same IL and use the same runtime, it's relatively easy to mix and match languages, calling code written in one language from another.

But living in this sandbox, you may forget that under the hood, .NET still runs machine code and talks to native libraries. Those libraries are platform-specific and may behave differently based on the platform. Surprisingly, it may cause almost forgotten DLL hell issues you rarely expect in a .NET application.

DLL Hell refers to a common issue in older versions of the Windows operating system where applications could interfere with each other by overwriting or updating shared Dynamic Link Libraries (DLLs). It could lead to various problems, including application failures, system instability, and version conflicts, as different programs might require different versions of the same DLL. The term also encompasses difficulties arising from the Windows registry's management of DLL information and the potential for installation programs to inadvertently disrupt the System by installing incorrect DLL versions.

The invention of .NET was the way to avoid the DLL Hell problem on Windows, and it mostly achieved it (at least from my experience). But what would you say if you saw it on Linux in 2023? Let's take a look at the sample project.

The project is a simple console application that reads an image and writes its dimensions to standard output. I used the SkiaSharp library since the project should support Windows, Linux, and MacOS. SkiaSharp is a cross-platform 2D graphics API for .NET platforms based on the Skia Graphics Library, an open-source graphics engine used by Chrome, Android, and other media. It provides a comprehensive set of drawing features ranging from shapes to complex path operations, text rendering, and image manipulation.

using SkiaSharp;
using static System.Console;

var file = new FileInfo("cover.jpg");
WriteLine($"INPUT: {file.Name}");

using var stream = file.OpenRead();
var coverImage = SKBitmap.Decode(stream);
WriteLine($"Got image: {coverImage.Width} x {coverImage.Height} {coverImage.Info.BitsPerPixel} ppi - from file {file.Name}");

Currently, the project references only SkiaSharp 2.88.6 package and works well on Windows.

INPUT: cover.jpg
Got image: 617 x 800 32 ppi - from file cover.jpg

Let's run it on a Docker container using Alpine Linux. I use Alpine Linux for its simplicity, security, and efficiency, particularly in resource-constrained environments or when building minimal Docker containers due to its small footprint.

FROM mcr.microsoft.com/dotnet/aspnet:6.0-alpine AS base
WORKDIR /app

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
COPY ["SkiaSharpTest.csproj", "."]
RUN dotnet restore "SkiaSharpTest.csproj"
COPY . .
WORKDIR "/src"
RUN dotnet build "SkiaSharpTest.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "SkiaSharpTest.csproj" -c Release -o /app/publish /p:UseAppHost=false

FROM base AS final
RUN apk add --no-cache icu-libs
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "SkiaSharpTest.dll"]

docker build -t skia .
docker run --rm skia

INPUT: cover.jpg
Unhandled exception. System.TypeInitializationException: The type initializer for 'SkiaSharp.SKAbstractManagedStream' threw an exception.
 ---> System.DllNotFoundException: Unable to load shared library 'libSkiaSharp' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: Error loading shared library liblibSkiaSharp: No such file or directory
   at SkiaSharp.SkiaApi.sk_managedstream_set_procs(SKManagedStreamDelegates procs)
   at SkiaSharp.SKAbstractManagedStream..cctor()
   --- End of inner exception stack trace ---
   at SkiaSharp.SKAbstractManagedStream..ctor(Boolean owns)
   at SkiaSharp.SKManagedStream..ctor(Stream managedStream, Boolean disposeManagedStream)
   at SkiaSharp.SKCodec.WrapManagedStream(Stream stream)
   at SkiaSharp.SKCodec.Create(Stream stream, SKCodecResult& result)
   at SkiaSharp.SKCodec.Create(Stream stream)
   at SkiaSharp.SKBitmap.Decode(Stream stream)
   at Program.<Main>$(String[] args) in /src/Program.cs:line 9

It is a common problem. SkiaSharp is a native library that must be shipped with your .NET application. By default, the SkiaSharp 2.88.6 package includes only Windows and MacOS binaries. Since I need Linux support, I have two options: install SkiaSharp.NativeAssets.Linux 2.88.6 or SkiaSharp.NativeAssets.Linux.NoDependencies 2.88.6. I recommend using the second option if you do not need fancy font support because it does not require additional Linux packages shipped with your image.

Everything worked after I added the SkiaSharp.NativeAssets.Linux.NoDependencies 2.88.6 package to my project and rebuilt the image.

INPUT: cover.jpg
Got image: 617 x 800 32 ppi - from file cover.jpg

Whatever I have shown till this point is the standard use case of SkiaSharp. But now, imagine you are working on a large project with tens or hundreds of dependencies. You did the above steps, but your app still generates a runtime exception.

Let's dive deeper into "System.DllNotFoundException: Unable to load shared library 'libSkiaSharp' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: Error loading shared library liblibSkiaSharp: No such file or directory" error message.

The System.DllNotFoundException in .NET is thrown when a program tries to load a dynamic link library (DLL) that cannot be found due to reasons like the DLL being absent, located in the wrong directory, the program running on an incompatible architecture, the DLL's dependencies being missing, permission restrictions, or the DLL file being corrupted. Resolving this error requires verifying that the DLL exists, is correctly placed, has the necessary permissions, is not corrupted, and is compatible with the System's architecture.

The easiest way to investigate the issue is to get access to the container's or pod's terminal through Docker or Kubernetes. I am debugging a standalone container, so let's run it and get access to the shell. I have to access the shell on the container startup because the container only runs a standalone app. Keep in mind that Alpine Linux uses sh instead of Bash.

docker run --rm -it --entrypoint /bin/sh skia

If you are debugging a microservice or web application running within a container, you can access the container's shell using the docker exec command. For example, this is the command to get access to the Redis shell when it is running in a container.

$ docker run --name redis -d -p 6379:6379 redis
$ docker ps

CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS          PORTS                    NAMES
73546b9cd887   redis     "docker-entrypoint.s…"   4 minutes ago    Up 4 minutes    0.0.0.0:6379->6379/tcp   redis
613199b8912a   skia      "/bin/sh"                16 minutes ago   Up 16 minutes                            wizardly_saha

$ docker exec -it redis redis-cli
127.0.0.1:6379>

Let's move back to the SkiaSharp issue. First, you must verify that SkiaSharp's native libraries are on the container. Remember that Alpine Linux uses the musl C runtime, so you must verify the linux-musl-x64 runtime identifier.

app # ls
HarfBuzzSharp.dll                 SkiaSharp.dll                     SkiaSharpTest.dll                 SkiaSharpTest.runtimeconfig.json  cover.jpg
SkiaSharp.HarfBuzz.dll            SkiaSharpTest.deps.json           SkiaSharpTest.pdb                 Topten.RichTextKit.dll            runtimes
/app # ls runtimes/
linux-arm       linux-arm64     linux-musl-x64  linux-x64       osx             win-arm64       win-x64         win-x86
/app # ls runtimes/linux-musl-x64/native/
libHarfBuzzSharp.so  libSkiaSharp.so

As you can see, the SkiaSharp native library exists. In this case, you must verify that the Operating System can load it. Use the ldd command to do it on Linux.

/app # cd runtimes/linux-musl-x64/native/
/app/runtimes/linux-musl-x64/native # ldd libSkiaSharp.so
        /lib/ld-musl-x86_64.so.1 (0x7fabcfc23000)
Error loading shared library libfontconfig.so.1: No such file or directory (needed by libSkiaSharp.so)
        libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fabcfc23000)
Error relocating libSkiaSharp.so: FcFontSetDestroy: symbol not found
Error relocating libSkiaSharp.so: FcPatternAddString: symbol not found
Error relocating libSkiaSharp.so: FcInitLoadConfigAndFonts: symbol not found
Error relocating libSkiaSharp.so: FcPatternFilter: symbol not found
Error relocating libSkiaSharp.so: FcPatternGetLangSet: symbol not found
Error relocating libSkiaSharp.so: FcConfigCreate: symbol not found
Error relocating libSkiaSharp.so: FcCharSetDestroy: symbol not found
Error relocating libSkiaSharp.so: FcPatternGetCharSet: symbol not found
Error relocating libSkiaSharp.so: FcPatternGetBool: symbol not found
Error relocating libSkiaSharp.so: FcDefaultSubstitute: symbol not found
Error relocating libSkiaSharp.so: FcPatternAddCharSet: symbol not found
Error relocating libSkiaSharp.so: FcPatternRemove: symbol not found
Error relocating libSkiaSharp.so: FcPatternGetInteger: symbol not found
Error relocating libSkiaSharp.so: FcCharSetHasChar: symbol not found
Error relocating libSkiaSharp.so: FcCharSetAddChar: symbol not found
Error relocating libSkiaSharp.so: FcConfigGetFonts: symbol not found
Error relocating libSkiaSharp.so: FcCharSetCreate: symbol not found
Error relocating libSkiaSharp.so: FcGetVersion: symbol not found
Error relocating libSkiaSharp.so: FcPatternAddWeak: symbol not found
Error relocating libSkiaSharp.so: FcConfigDestroy: symbol not found
Error relocating libSkiaSharp.so: FcPatternGetString: symbol not found
Error relocating libSkiaSharp.so: FcPatternCreate: symbol not found
Error relocating libSkiaSharp.so: FcFontSetAdd: symbol not found
Error relocating libSkiaSharp.so: FcPatternReference: symbol not found
Error relocating libSkiaSharp.so: FcPatternEqual: symbol not found
Error relocating libSkiaSharp.so: FcFontSetCreate: symbol not found
Error relocating libSkiaSharp.so: FcConfigSubstitute: symbol not found
Error relocating libSkiaSharp.so: FcPatternAddLangSet: symbol not found
Error relocating libSkiaSharp.so: FcObjectSetBuild: symbol not found
Error relocating libSkiaSharp.so: FcLangSetHasLang: symbol not found
Error relocating libSkiaSharp.so: FcPatternAddInteger: symbol not found
Error relocating libSkiaSharp.so: FcObjectSetDestroy: symbol not found
Error relocating libSkiaSharp.so: FcStrCmpIgnoreCase: symbol not found
Error relocating libSkiaSharp.so: FcPatternGet: symbol not found
Error relocating libSkiaSharp.so: FcPatternDestroy: symbol not found
Error relocating libSkiaSharp.so: FcFontRenderPrepare: symbol not found
Error relocating libSkiaSharp.so: FcPatternDuplicate: symbol not found
Error relocating libSkiaSharp.so: FcFontMatch: symbol not found
Error relocating libSkiaSharp.so: FcPatternGetMatrix: symbol not found
Error relocating libSkiaSharp.so: FcLangSetDestroy: symbol not found
Error relocating libSkiaSharp.so: FcConfigGetSysRoot: symbol not found
Error relocating libSkiaSharp.so: FcLangSetAdd: symbol not found
Error relocating libSkiaSharp.so: FcLangSetCreate: symbol not found
Error relocating libSkiaSharp.so: FcFontSetMatch: symbol not found

The output tells that multiple functions are not found! I would expect I don't need to install additional Alpine packages since I used SkiaSharp.NativeAssets.Linux.NoDependencies 2.88.6 Nuget package to build my project. It's time to move back to Visual Studio and check project dependencies.

Project Dependencies

As you can see, SkiaSharp's native binaries are references twice, and the SkiaSharp.NativeAssets.Linux ones overwrite the SkiaSharp.NativeAssets.Linux.NoDependencies are the ones that I expected to use. In this case, I have no choice but to set up additional Alpine Linux packages during the image build process.

I fixed the problem as follows:

  • I used SkiaSharp.NativeAssets.Linux Nuget package since it is required by other libraries anyway.
  • I updated my Dockerfile to install fontconfig Alpine Linux package, which includes missing functions discovered above.
RUN apk add --no-cache icu-libs fontconfig

FROM mcr.microsoft.com/dotnet/aspnet:6.0-alpine AS base
WORKDIR /app

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
COPY ["SkiaSharpTest.csproj", "."]
RUN dotnet restore "SkiaSharpTest.csproj"
COPY . .
WORKDIR "/src"
RUN dotnet build "SkiaSharpTest.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "SkiaSharpTest.csproj" -c Release -o /app/publish /p:UseAppHost=false

FROM base AS final
RUN apk add --no-cache icu-libs fontconfig
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "SkiaSharpTest.dll"]

After the fix, everything worked well.

$docker build -t skia .
[+] Building 7.6s (19/19)
...

$ docker run --rm skia
INPUT: cover.jpg
Got image: 617 x 800 32 ppi - from file cover.jpg

P. S. I experienced the above problem with a microservice running on Kubernetes. So, ensure you can access your pod's terminal when needed because you can follow the same step from this article to troubleshoot the runtime error.