> ## Content Index
> Fetch the complete content index at: https://ostreff.info/llms.txt
> Use this file to discover other available public pages before exploring further.

# Solving LMDB Version Incompatibility in PowerDNS: A Post-Mortem
- URL: https://ostreff.info/solving-lmdb-version-incompatibility-in-powerdns-a-post-mortem/
- Published: 2026-07-06T21:02:00.000Z
- Updated: 2026-07-09T21:42:40.000Z
- Description: As system administrators, we often pride ourselves on maintaining "zero-downtime" environments. However, upgrading core system libraries—specifically those handling database engines like LMDB—can sometimes lead to silent incompatibilities that threaten the integrity of our services.
- Author: Jordan Ostreff
- Tags: #PowerDNS, #LMDB, #FreeBSD

Recently, I encountered a critical issue after a routine system upgrade: my PowerDNS instance was running on a newer version of `liblmdb` (1.0.0), while the existing database files were created with an older 0.9.x version. Here is how I navigated the transition, recovered the data, and cleaned up the environment.  
**The Challenge**  
When upgrading the base system, the `liblmdb` library transitioned to version 1.0.0\. Because PowerDNS uses an LMDB backend, the discrepancy between the on-disk file format and the new library version created an immediate risk of data corruption. Simply restarting the service was not a safe option; we needed to ensure the data was compatible with the new ABI.  
**Step 1: Data Extraction**  
The biggest obstacle was the LMDB file structure. PowerDNS uses a sharded storage approach (multiple `.lmdb` files), and standard tools like `mdb_dump` expect a directory rather than a file path, resulting in the dreaded `mdb_env_open failed, error 20: Not a directory`.  
To solve this, I performed a "staging" operation:  

1. Created a temporary directory structure mimicking an LMDB environment.
2. Copied each individual shard (`pdns.lmdb`, `pdns.lmdb-1`, etc.) into this temporary space, renaming them to `data.mdb`.
3. Executed `mdb_dump` on the temporary folder to extract the data into a safe, text-based format.

**Step 2: Verification and Cleanup**  
Once the data was safely backed up, I cleaned the environment. Over the years, the directory had accumulated legacy SQLite artifacts from a previous migration. Since the PowerDNS configuration (`pdns.conf`) was explicitly set to use the `lmdb` backend (`launch=lmdb`), I moved the unused SQLite files to an archive directory, significantly decluttering the system.  
**Step 3: Re-integration**  
With the service stopped, I ensured that all file permissions were correctly set to the `bind` user. Upon restarting the service, PowerDNS detected the existing shard files. Even though my configuration specified `lmdb-shards=1`, the engine intelligently detected the 64 shards on disk and adapted accordingly, successfully loading the backend without data loss.

**Practical Implementation: The Migration Script**  
To automate the data extraction without hardcoding paths or shard counts, we can extract the database location directly from `pdns.conf`. The following `sh` script demonstrates how to safely dump all shards regardless of their number.

**The Final Migration Tool**  
This `sh` script handles the entire lifecycle of the migration process:

1. **Tool Discovery:** Automatically finds and extracts legacy `mdb_dump` versions from your package backups (`/usr/ports/packages/`).
2. **Data Integrity:** Performs a "Stage-Dump-Load" operation, ensuring the new database format is created natively by the current system tools.
3. **Safety First:** Includes multiple sanity checks (root user verification, path safety, and non-destructive cleanup options).
4. **Environment Awareness:** Dynamically identifies PowerDNS ownership and permissions to ensure the service resumes seamlessly after migration.

```bash
#!/bin/sh

set -e

# Ensure script is run as root
if [ "$(id -u)" -ne 0 ]; then
    echo "This script must be run as root."
    exit 1
fi

# Configuration
CONFIG="/usr/local/etc/pdns/pdns.conf"
BACKUP_DIR="/tmp/pdns_lmdb_backup"
STAGING_DIR="/tmp/pdns_migrate/db"
LEGACY_DIR="/tmp/legacy_tools"
PKG_DIR="/usr/ports/packages/portmaster-backup"
SYSTEM_DUMP="/usr/local/bin/mdb_dump"
SYSTEM_LOAD="/usr/local/bin/mdb_load"

# Verify config exists
if [ ! -f "$CONFIG" ]; then
    echo "Error: Configuration file $CONFIG not found!"
    exit 1
fi

# 1. Detect available legacy packages
echo "Searching for lmdb packages in $PKG_DIR..."
PKGS=$(ls "$PKG_DIR"/lmdb-*.pkg 2>/dev/null)

if [ -z "$PKGS" ]; then
    echo "No legacy packages found. Using system default mdb_dump."
    MDB_DUMP_BIN="$SYSTEM_DUMP"
else
    echo "Found the following packages:"
    i=1
    set -- $PKGS
    for p in "$@"; do
        echo "$i) $(basename "$p")"
        i=$((i + 1))
    done

    printf "Select a package to extract mdb_dump from (or press Enter to use system default): "
    read -r choice

    if [ -n "$choice" ] && [ "$choice" -ge 1 ] && [ "$choice" -le "$#" ]; then
        SELECTED_PKG=$(eval echo \${$choice})
        echo "Extracting mdb_dump from $(basename "$SELECTED_PKG")..."
        
        mkdir -p "$LEGACY_DIR"
        tar -xf "$SELECTED_PKG" -C "$LEGACY_DIR" usr/local/bin/mdb_dump
        MDB_DUMP_BIN="$LEGACY_DIR/usr/local/bin/mdb_dump"
    else
        MDB_DUMP_BIN="$SYSTEM_DUMP"
    fi
fi

# Verify binaries exist and are executable
if [ ! -x "$MDB_DUMP_BIN" ]; then
    echo "Error: $MDB_DUMP_BIN is not executable."
    exit 1
fi
if [ ! -x "$SYSTEM_LOAD" ]; then
    echo "Error: $SYSTEM_LOAD not found."
    exit 1
fi

echo "Using: $($MDB_DUMP_BIN -V 2>&1 | head -n 1)"

# Extract DB path from pdns.conf
DB_PATH=$(grep "lmdb-filename" "$CONFIG" | cut -d'=' -f2 | tr -d ' ')
if [ -z "$DB_PATH" ]; then
    echo "Error: Could not determine lmdb-filename from $CONFIG"
    exit 1
fi
DB_DIR=$(dirname "$DB_PATH")

mkdir -p "$BACKUP_DIR" "$STAGING_DIR"

echo "Extracting database from $DB_DIR..."

# Process the main file and all shards
for file in "$DB_DIR"/pdns.lmdb*; do
    case "$file" in
        *-lock) continue ;; 
    esac

    rm -f "$STAGING_DIR/data.mdb"
    cp "$file" "$STAGING_DIR/data.mdb"

    DUMP_FILE="$BACKUP_DIR/$(basename "$file").dump"
    "$MDB_DUMP_BIN" -a "$STAGING_DIR" > "$DUMP_FILE"
    
    echo "Dumped: $(basename "$file") -> $(basename "$DUMP_FILE")"
done

# Recreate
printf "Ready to recreate database in $DB_DIR using new LMDB version? (y/n): "
read -r confirm
if [ "$confirm" = "y" ]; then
    # Security check: ensure DB_DIR is not root
    if [ -n "$DB_DIR" ] && [ "$DB_DIR" != "/" ] && [ "$DB_DIR" != "/etc" ]; then
        echo "Removing old database files from $DB_DIR..."
        rm -f "$DB_DIR"/pdns.lmdb*
    else
        echo "Safety check failed: Refusing to delete files in $DB_DIR"
        exit 1
    fi
    
    echo "Recreating database from dumps..."
    for dump in "$BACKUP_DIR"/*.dump; do
        target=$(basename "$dump" .dump)
        echo "Loading $dump -> $target"
        "$SYSTEM_LOAD" -a "$DB_DIR/$target" < "$dump"
    done
fi

# Cleanup
printf "Migration complete. Keep temporary files? (y/n): "
read -r keep_dumps
if [ "$keep_dumps" = "n" ]; then
    rm -rf "$BACKUP_DIR" "$STAGING_DIR" "$LEGACY_DIR"
    echo "Temporary files cleaned."
fi

# Apply permissions
PDNS_USER=$(grep "^setuid" "$CONFIG" | cut -d'=' -f2 | tr -d ' ')
PDNS_GROUP=$(grep "^setgid" "$CONFIG" | cut -d'=' -f2 | tr -d ' ')

[ -z "$PDNS_USER" ] && PDNS_USER=$(ls -ld "$DB_DIR" | awk '{print $3}')
[ -z "$PDNS_GROUP" ] && PDNS_GROUP=$(ls -ld "$DB_DIR" | awk '{print $4}')

echo "Applying permissions for $PDNS_USER:$PDNS_GROUP on $DB_DIR"
chown -R "$PDNS_USER:$PDNS_GROUP" "$DB_DIR"

# Service restart
echo "Restarting PowerDNS..."
service pdns restart

echo "Migration process finished."
tail -20 /var/log/messages | grep pdns
```

  
**Lessons Learned**  

- **Don't ignore the ldd output:** Checking which shared libraries your binaries are actually linking against is the first step in diagnosing versioning conflicts.
- **Staging is your best friend:** When dealing with strict tool requirements (like `mdb_dump` expecting directories), manual staging is the safest way to perform a low-level migration.
- **Clean house:** If you have migrated from SQLite to LMDB in the past, don't leave the old files sitting in your production directory. They only increase the complexity of backups and debugging.

**Final Thoughts**  
Upgrading database engines doesn't always have to be a "break-and-fix" cycle. By preparing a solid backup, understanding the underlying storage structure, and verifying the service logs post-restart, you can perform complex migrations while keeping your DNS infrastructure rock-solid.

The FreeBSD mentioned this in `/usr/ports/UPDATING` article:

```sh
20260702:
  AFFECTS: users of databases/lmdb
  AUTHOR: delphij@FreeBSD.org

  LMDB 1.0 introduced an incompatible on-disk file format change.
  Versions 0.9.x and 1.0.x databases are mutually incompatible.

  Before upgrading, export all existing databases using the old v0.9
  mdb_dump utility, then import them with the new v1.0 mdb_load after
  upgrading.  There is no support for opening v0.9 database files
  directly with LMDB 1.0.

  Example migration procedure:
  # mdb_dump -a /path/to/db > /tmp/mydb.dump
  # pkg upgrade databases/lmdb
  # mdb_load -f /tmp/mydb.dump /path/to/newdb
```

  
Tags: #PowerDNS #LMDB #SystemAdministration #FreeBSD #DatabaseMigration #DevOps