rocky8.usd21.org | DigitalOcean | cPanel & WHM | ELevate v83 | MySQL Community 8.0.46
Purpose: developer-facing postmortem, reproducible diagnostics, successful recovery steps, and proposed preflight improvements for cPanel ELevate and Leapp.
Download Word Doc – rocky8_elevate_incident_review
Download PDF – rocky8_elevate_incident_review
Executive finding: The distribution conversion itself ultimately succeeded. The incident was a chain of independent problems: (1) a target-critical filesystem package was blocked by a global DNF exclude, (2) cPanel rejected Leapp packages whose provenance was elevate-testing, (3) the generated upgrade initramfs created a mount unit for cPanel's file-backed /usr/tmpDSK secure /tmp and failed before the offline upgrade, and (4) after EL9 booted, EL8-specific DNF versionlocks prevented cPanel from restoring the MySQL 8.0 EL9 RPM set. A console-routing issue initially hid the real initramfs error behind the last visible line "Probing EDD … ok".
1. Environment and scope
| Host | rocky8.usd21.org |
| Provider | DigitalOcean Droplet / KVM / VirtIO |
| Source OS | AlmaLinux 8.10 |
| Target OS | AlmaLinux 9.8 (Olive Jaguar) |
| Final observed kernel | 5.14.0-687.42.1.el9_8.x86_64 |
| cPanel | 136.0 build 38 after distribution conversion |
| ELevate script | v83 |
| Database | Oracle MySQL Community 8.0.46 |
| Root filesystem | XFS |
| Boot | GRUB with BLS |
| Upgrade wrapper | /scripts/elevate-cpanel |
2. High-level incident timeline
- Preflight: cPanel ELevate rejected six Leapp packages installed from the unsupported elevate-testing repository.
- Raw Leapp preupgrade: target_userspace_creator failed because filesystem-3.16-5.el9 was filtered by the source system DNF exclude configuration.
- Preflight repair: Leapp packages were reinstalled from the supported elevate repository and only the filesystem exclusion was removed from DNF configuration.
- Stage 3: target userspace creation and transaction test succeeded; Leapp generated the special upgrade initramfs.
- First upgrade-initramfs boots appeared frozen after "Probing EDD (edd=off to disable)… ok". Testing edd=off did not resolve the boot.
- GRUB console arguments were changed temporarily to expose tty0. The hidden dracut failure was /usr/tmpDSK: Can't lookup blockdev and Failed to mount /sysroot/tmp.
- Inside the dracut shell, systemd showed sysroot-tmp.mount was generated from /etc/fstab with What=/usr/tmpDSK and Where=/sysroot/tmp. Removing that transient mount unit allowed the offline Leapp transaction to proceed.
- The host booted AlmaLinux 9.8 successfully, but /scripts/elevate-cpanel –status reported Failed during stage 4.
- Stage 4 failure was MySQL restoration. cPanel's MySQL job log showed EL9 mysql-community packages filtered out by exclude filtering.
- DNF versionlock inspection revealed several mysql-community packages pinned to 8.0.46-1.el8.*. Removing only the MySQL versionlocks allowed a dry-run EL9 transaction to resolve, and cPanel's background MySQL upgrade then completed successfully.
- ELevate was resumed with –continue; later PECL warnings for imagick/redis remained as post-upgrade cleanup items.
3. Failure 1 – global DNF exclude blocked Leapp target userspace
The first hard Leapp failure occurred while creating the AlmaLinux 9 target userspace. The source system carried a cPanel-oriented global DNF exclude list that also contained the target-critical package filesystem. Because Leapp copies DNF configuration into the target userspace, the exclusion became active during target dependency resolution.
Relevant source DNF configuration
[main]
exclude=bind-chroot dovecot* exim* filesystem p0f php* proftpd* pure-ftpd*The critical error was that filesystem-3.16-5.el9 from almalinux9-baseos was filtered by exclude filtering. That cascaded into otherwise misleading dependency failures involving bash, util-linux, dnf, systemd-libs and python3-systemd.
Diagnostic comparison used on the source system
dnf repoquery filesystem --show-duplicates
# no result under the active excludes
dnf --disableexcludes=all repoquery filesystem --show-duplicates
filesystem-0:3.8-6.el8.x86_64The successful fix was deliberately narrow: remove only filesystem from the global exclude, preserving the other cPanel exclusions. A subsequent Leapp run reached target_userspace_creator, loaded AlmaLinux 9.8 repositories, resolved dependencies, and selected filesystem-3.16-5.el9 from almalinux9-baseos.
Successful evidence from /var/log/elevate-cpanel.log / Leapp output
03:23:16 [INFO] ====> * target_userspace_creator
03:23:16 [INFO] Initializes a directory to be populated as a minimal environment to run binaries from the target system.
03:23:22 [INFO] Extra Packages for Enterprise Linux 9 - x86_64
03:23:35 [INFO] AlmaLinux 9.8 - BaseOS
03:23:44 [INFO] AlmaLinux 9.8 - AppStream
03:24:21 [INFO] Dependencies resolved.
...
filesystem x86_64 3.16-5.el9 almalinux9-baseos 1.1 MConfirmation that the target filesystem package was no longer filtered
03:24:42 [INFO] Running transaction test
03:24:43 [INFO] Transaction test succeeded.
03:24:43 [INFO] Running transaction
03:24:44 [INFO] Running scriptlet: filesystem-3.16-5.el9.x86_64Developer recommendation: Before Stage 3/reboot, resolve target-critical packages using the exact DNF configuration Leapp will copy into the target userspace. A package existing in a repository is not sufficient; the check must account for excludes/versionlocks/plugins that affect the actual transaction.
4. Failure 2 – unsupported elevate-testing package provenance
cPanel ELevate also blocked because six installed Leapp packages had been installed from AlmaLinux's elevate-testing repository. The stable elevate repository offered the same NEVRAs, so package provenance rather than package version was the blocker.
Relevant /var/log/elevate-cpanel.log evidence
2026-09-06 02:25:50 [INFO] Unsupported YUM repo enabled 'csm' without packages installed from /etc/yum.repos.d/csm.repo, these will be disabled before ELevation
2026-09-06 02:25:50 [ERROR] 6 package(s) installed from unsupported YUM repo 'elevate-testing' from /etc/yum.repos.d/elevate-testing.repoSix affected packages
leapp
leapp-data-almalinux
leapp-deps
leapp-upgrade-el8toel9
leapp-upgrade-el8toel9-deps
python3-leappSuccessful provenance repair
dnf reinstall -y \
--disablerepo='*' \
--enablerepo=elevate \
leapp \
leapp-data-almalinux \
leapp-deps \
leapp-upgrade-el8toel9 \
leapp-upgrade-el8toel9-deps \
python3-leappAfter the reinstall, the six packages reported @elevate provenance, elevate-testing was disabled, and /scripts/elevate-cpanel –check reported no blockers.
Developer recommendation: When an unsupported-repository package has an identical NEVRA available from the supported elevate repository, print the offending package list and offer/document a provenance-only reinstall path.
5. Stage 3 success – target transaction and initramfs generation
Once the DNF exclusion and repository-provenance problems were corrected, Leapp successfully built the target environment and completed its transaction validation. This is important because it proves the later boot failure was downstream of package solving.
Key transition from target transaction into the special upgrade initramfs
03:25:54 [INFO] Transaction Summary
03:25:54 [INFO] Install 286 Packages
03:25:54 [INFO] Upgrade 847 Packages
03:25:54 [INFO] Remove 43 Packages
03:25:54 [INFO] Downgrade 7 Packages
03:25:54 [INFO] Total size: 1.3 G
...
03:29:29 [INFO] Complete!
03:29:30 [INFO] ==> Processing phase `InterimPreparation`
03:29:30 [INFO] ====> * mount_unit_generator
03:29:30 [INFO] Sets up storage initialization using systemd's mount units in the upgrade container.
03:29:31 [INFO] ====> * common_leapp_dracut_modules
03:29:31 [INFO] Influences the generation of the initram disk
03:29:31 [INFO] ====> * upgrade_initramfs_generator
03:29:31 [INFO] Creates the upgrade initramfs6. Misleading boot symptom – EDD line was not the failure
The DigitalOcean console repeatedly appeared to hang immediately after the message below. Because no later output was visible, the EDD probe initially looked like the root cause.
Probing EDD (edd=off to disable)... okA one-time GRUB test added edd=off to the ELevate-Upgrade-Initramfs kernel command line. That changed the visible behavior but did not complete the boot. The critical breakthrough came from console routing, not EDD.
GRUB console diagnostic
Original upgrade kernel arguments included:
console=ttyS0,115200n8
Temporary diagnostic edit used:
console=ttyS0,115200n8 console=tty0With tty0 made the last console argument, the DigitalOcean VNC console displayed the real dracut failure and eventually accepted input into the emergency shell.
Observability lesson: "Probing EDD … ok" was only the last visible message, not the failed subsystem. The EDD routine had already printed ok. A cloud console routing mismatch hid the downstream initramfs error.
7. Primary boot blocker – cPanel secure /tmp and sysroot-tmp.mount
Once tty0 output was visible, the upgrade initramfs showed that root, /boot and /boot/efi were mounting successfully. The failure occurred when the real-root /tmp mount was constructed from cPanel's file-backed secure /tmp image.
Actual hidden boot failure
[ OK ] Mounted /sysroot.
[ OK ] Mounted /sysroot/boot.
[ OK ] Mounted /sysroot/boot/efi.
/usr/tmpDSK: Can't lookup blockdev
[FAILED] Failed to mount /sysroot/tmp.
[DEPEND] Dependency failed for Local File Systems.
Generating "/run/initramfs/rdsosreport.txt"
Entering emergency mode.Inspection of the real root showed the cPanel secure /tmp backing file and fstab entry:
Real-root configuration
grep -nE 'tmp|tmpDSK' /sysroot/etc/fstab
15:/usr/tmpDSK /tmp ext4 defaults,noauto 0 0
ls -lh /sysroot/usr/tmpDSK
-rw-------. 1 root root 4.0G ... /sysroot/usr/tmpDSKThe noauto option did not prevent the Leapp upgrade environment from producing a mount unit that became part of local-fs.target. systemd confirmed exactly how the transient unit had been generated:
Generated unit inside the running upgrade initramfs
systemctl cat sysroot-tmp.mount
# /usr/lib/systemd/system/sysroot-tmp.mount
# Automatically generated by systemd-fstab-generator
[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target
[Mount]
Where=/sysroot/tmp
What=/usr/tmpDSK
Type=ext4
Options=defaults,noautoProvenance of the failing mount unit
systemctl show sysroot-tmp.mount -p FragmentPath -p SourcePath
FragmentPath=/usr/lib/systemd/system/sysroot-tmp.mount
SourcePath=/etc/fstabWe also tested changing the real root fstab. /sysroot was initially mounted read-only, so it was remounted read-write and the /usr/tmpDSK line was backed up and commented. That did not repair the already-generated initramfs mount dependency; even a fresh boot of the existing upgrade initramfs still attempted /sysroot/tmp. This established that fixing the source fstab after initramfs generation was too late for this artifact.
Persistent real-root fstab change made during recovery
mount -o remount,rw /sysroot
cp -a /sysroot/etc/fstab /sysroot/etc/fstab.pre-elevate-tmpfix
sed -i '\|^[[:space:]]*/usr/tmpDSK[[:space:]]|s|^|# ELEVATE-TEMP: |' /sysroot/etc/fstab
# Result:
# ELEVATE-TEMP: /usr/tmpDSK /tmp ext4 defaults,noauto 0 08. Successful initramfs surgery
The successful intervention was to neutralize the generated sysroot-tmp.mount unit inside the running ELevate/Leapp upgrade initramfs. This did not modify initramfs-upgrade.x86_64.img on disk; it changed the live emergency environment after the generated unit and its provenance had been confirmed.
Recovery intervention used successfully
systemctl stop sysroot-tmp.mount
rm -f /usr/lib/systemd/system/sysroot-tmp.mount
systemctl daemon-reload
systemctl reset-failed
systemctl status sysroot-tmp.mountExpected verification
Unit sysroot-tmp.mount could not be found.Continue the same upgrade boot
exitImmediately after removing the transient mount unit and exiting the emergency shell, the offline Leapp transaction proceeded. This creates strong A/B evidence that sysroot-tmp.mount was the boot blocker: with the generated unit present, the upgrade initramfs always entered emergency mode; with it removed, the system advanced through the EL8 -> EL9 conversion.
Important classification: This was incident recovery, not a generally supported Leapp procedure. The correct product fix is to prevent the incompatible mount unit from being generated or required in the first place.
9. Distribution conversion succeeded, ELevate Stage 4 did not
After the offline transaction, SSH returned and the host was genuinely running the target operating system, even though the cPanel wrapper reported a Stage 4 failure.
Post-reboot state
Hostname : rocky8.usd21.org
OS : AlmaLinux 9.8 (Olive Jaguar)
Kernel : 5.14.0-687.42.1.el9_8.x86_64
cPanel : 136.0 (build 38)
/scripts/elevate-cpanel --status
Failed during stage 4This distinction is operationally important: the operating-system conversion had succeeded; cPanel's post-distro restoration had not finished. Rolling back Leapp or rerunning leapp upgrade would have been the wrong response.
10. Stage 4 blocker – MySQL EL8 versionlocks prevented EL9 restoration
The authoritative cPanel log showed Stage 4 had already restored large portions of the EasyApache stack and then failed specifically in the MySQL post_distro_upgrade component.
Relevant /var/log/elevate-cpanel.log excerpt
2026-09-06 05:35:03 [DEBUG] Running 'post_distro_upgrade' from MySQL component.
2026-09-06 05:35:03 [INFO] Restoring MySQL 8.0
2026-09-06 05:35:03 [INFO] Beginning upgrade to MySQL 8.0
2026-09-06 05:37:25 [FATAL] FAILED to upgrade to MySQL 8.0
2026-09-06 05:37:25 [ERROR] The elevation process failed during stage 4.
...
/usr/local/cpanel/bin/whmapi1 start_background_mysql_upgrade version=8.0
...
/scripts/elevate-cpanel --continueThe cPanel background MySQL log provided the real package-manager failure:
Relevant /var/cpanel/logs/mysql_upgrade.<ID>/unattended_background_upgrade.log excerpt
Package mysql-community-devel-8.0.46-1.el8.x86_64 is already installed.
Error:
Problem: package mysql-community-server-8.0.46-1.el9.x86_64 from mysql80-community requires mysql-community-icu-data-files = 8.0.46-1.el9, but none of the providers can be installed
- cannot install the best candidate for the job
- package mysql-community-icu-data-files-8.0.46-1.el9.x86_64 from mysql80-community is filtered out by exclude filtering
...
(XID ...) The "mysql" service is disabled.
(XID ...) "/usr/bin/yum" reported error code "1" when it endedGlobal exclude inspection did not show mysql*, and ordinary repoquery could see the required EL9 packages. A real DNF install dry run, however, reproduced the transaction filtering. That pointed to another transaction filter rather than repository absence.
Root cause of the MySQL package filter
dnf versionlock list
mysql-community-client-plugins-0:8.0.46-1.el8.*
mysql-community-client-0:8.0.46-1.el8.*
mysql-community-common-0:8.0.46-1.el8.*
mysql-community-icu-data-files-0:8.0.46-1.el8.*
mysql-community-libs-0:8.0.46-1.el8.*These locks were valid only for the old distribution release. After the host became EL9, they explicitly prevented DNF from replacing the locked EL8 builds with the required EL9 builds.
Successful versionlock repair
dnf versionlock delete 'mysql-community-*'The dry-run transaction immediately changed from filtered failure to a valid EL9 conversion:
Post-versionlock dry-run success
Dependencies resolved.
Installing:
mysql-community-server x86_64 8.0.46-1.el9
Upgrading:
mysql-community-client x86_64 8.0.46-1.el9
mysql-community-client-plugins x86_64 8.0.46-1.el9
mysql-community-common x86_64 8.0.46-1.el9
mysql-community-devel x86_64 8.0.46-1.el9
mysql-community-icu-data-files x86_64 8.0.46-1.el9
mysql-community-libs x86_64 8.0.46-1.el9
Transaction Summary
Install 1 Package
Upgrade 6 Packages
Operation aborted. # --assumeno was intentionalThe real conversion was then left to cPanel's supported database workflow. The successful job log ended with:
Successful cPanel MySQL upgrade evidence
Restarting mysqld service.
Waiting for "mysql" to restart ... finished.
Service Status
mysqld (/usr/sbin/mysqld) is running ...
Startup Log
Started MySQL Server.
Log Messages
/usr/sbin/mysqld: ready for connections. Version: '8.0.46'
mysql restarted successfully.
MySQL upgrade completed successfullyFunctional database verification
mysql -e 'SELECT VERSION();'
+-----------+
| VERSION() |
+-----------+
| 8.0.46 |
+-----------+Developer recommendation: ELevate should inspect dnf versionlock list during –check and flag any source-release locks (for example *.el8.*) on packages it knows must be restored on EL9. MySQL is especially actionable because ELevate already has a dedicated MySQL component and knows its restore target.
11. Later Stage 4 warnings – PECL modules
After MySQL was repaired and ELevate was continued, later Stage 4 components ran. The PECL component emitted warnings rather than fatal errors. These should be treated as post-upgrade validation/remediation rather than as the root cause of the failed distribution conversion.
Observed PECL warnings after Stage 4 resumed
WARNING: Missing pecl package(s) for /opt/cpanel/ea-php82/root/usr/bin/pecl
Please reinstall these packages:
- imagick
- redis
WARNING: Missing pecl package(s) for /opt/cpanel/ea-php83/root/usr/bin/pecl
Please reinstall these packages:
- imagick
WARNING: Missing pecl package(s) for /opt/cpanel/ea-php84/root/usr/bin/pecl
Please reinstall these packages:
- imagick12. Recovery How-To for the same failure family
12.1 Preflight before /scripts/elevate-cpanel –start
Collect source state
/scripts/elevate-cpanel --version
/scripts/elevate-cpanel --check
cat /etc/dnf/dnf.conf
dnf versionlock list
grep -nE 'tmpDSK|[[:space:]]/tmp[[:space:]]' /etc/fstab
rpm -qa | grep -Ei '^(mysql|mysql-community)' | sort- If filesystem is globally excluded, treat it as a Leapp blocker before target userspace creation.
- If mysql-community packages are versionlocked to *.el8.*, remove or migrate those locks before the system crosses to EL9.
- If /usr/tmpDSK /tmp exists, flag it as a high-risk Leapp initramfs compatibility case.
- Verify Leapp package provenance; cPanel ELevate can reject packages installed from elevate-testing.
12.2 If the upgrade boot appears frozen at Probing EDD
Do not assume EDD is the failed subsystem. Edit only the ELevate-Upgrade-Initramfs GRUB entry for a one-time diagnostic boot and ensure tty0 receives output/input. In this incident the useful order was:
console=ttyS0,115200n8 console=tty0If the boot then reports /usr/tmpDSK: Can't lookup blockdev and Failed to mount /sysroot/tmp, enter the dracut shell and prove the generated unit before changing it:
findmnt /sysroot
grep -nE 'tmp|tmpDSK' /sysroot/etc/fstab
ls -lh /sysroot/usr/tmpDSK
systemctl cat sysroot-tmp.mount
systemctl show sysroot-tmp.mount -p FragmentPath -p SourcePathOnly if those commands reproduce the same unit signature should the incident recovery used here be considered:
systemctl stop sysroot-tmp.mount
rm -f /usr/lib/systemd/system/sysroot-tmp.mount
systemctl daemon-reload
systemctl reset-failed
systemctl status sysroot-tmp.mount
exitCaution: The command sequence above is appropriate only after confirming the same transient generated unit and the same cPanel file-backed /tmp source. It is not a generic fix for every dracut emergency shell.
12.3 If EL9 boots but ELevate reports Stage 4 failure
/scripts/elevate-cpanel --status
tail -n 150 /var/log/elevate-cpanel.logFor MySQL failures, inspect the newest directory under /var/cpanel/logs/mysql_upgrade.* and the unattended_background_upgrade.log. Then inspect package state and versionlocks:
rpm -qa | grep -Ei '^(mysql|mysql-community)' | sort
dnf versionlock listIf the exact pattern is EL8-specific mysql-community locks blocking EL9 equivalents, remove only those locks, verify a dry-run transaction resolves, run cPanel's background MySQL upgrade, verify mysqld and SQL connectivity, and then resume ELevate.
dnf versionlock delete 'mysql-community-*'
# dry-run first with --assumeno, then let cPanel perform the real DB work
/usr/local/cpanel/bin/whmapi1 start_background_mysql_upgrade version=8.0
# after successful DB log/service validation
/scripts/elevate-cpanel --continue13. Proposed developer changes
| Area | Observed failure | Detection opportunity | Proposed behavior |
|---|---|---|---|
| Leapp/cPanel preflight | filesystem in global exclude broke target userspace | Before Stage 3 using actual copied dnf.conf | Dependency-resolve mandatory target packages; inhibit with exact offending excludes. |
| Leapp mount generation | /usr/tmpDSK /tmp defaults,noauto became required sysroot-tmp.mount | Before upgrade initramfs generation | Recognize file-backed/loop mounts; honor noauto or suppress unsupported real-root mount units. |
| cPanel securetmp integration | cPanel-specific secure /tmp was incompatible with upgrade initramfs | ELevate knows it is a cPanel host | Temporarily disable/sanitize securetmp fstab entry before Leapp initramfs generation; restore after upgrade. |
| MySQL restore | *.el8.* versionlocks filtered EL9 RPMs in Stage 4 | During –check, before first reboot | Inspect versionlocks and block/migrate source-release locks for packages ELevate will restore. |
| Cloud console | ttyS0-only behavior obscured real dracut failure | When building temporary upgrade boot entry | Preserve provider-compatible VGA output or document tty0 diagnostic procedure. |
| Unsupported repo provenance | elevate-testing packages blocked although supported repo had same NEVRA | During –check | List exact packages and provide provenance-reinstall remediation when same NEVRA exists in stable repo. |
14. Failure chain
AlmaLinux 8.10 + cPanel
|
| global DNF exclude contained filesystem
v
Leapp target_userspace_creator failed
|
| remove only filesystem exclusion
v
Target userspace + transaction test succeeded
|
| Leapp generated ELevate-Upgrade-Initramfs
v
Console appeared frozen at "Probing EDD ... ok"
|
| expose tty0 console
v
REAL ERROR: /usr/tmpDSK: Can't lookup blockdev
Failed to mount /sysroot/tmp
|
| systemd-fstab-generator created sysroot-tmp.mount
| What=/usr/tmpDSK Where=/sysroot/tmp
v
Remove transient generated mount unit
|
v
Offline Leapp transaction succeeded
|
v
AlmaLinux 9.8 booted
|
| ELevate Stage 4 failed restoring MySQL
v
EL8 mysql-community versionlocks filtered EL9 RPMs
|
| remove MySQL versionlocks, verify dry run
v
cPanel MySQL upgrade completed successfully
|
v
/scripts/elevate-cpanel --continue
|
v
Later post-upgrade cleanup (including PECL warnings)Appendix A – Expanded log evidence
The sections below intentionally expand the previously referenced "pasted text" into direct excerpts so this PDF can stand alone when sent to cPanel/Leapp developers.
A.1 Unsupported elevate-testing repository blocker
2026-09-04 07:38:03 [INFO] Self-update of script version 83 requested.
2026-09-04 07:38:03 [INFO] Script is up to date.
2026-09-04 07:38:04 [INFO] Running: /usr/local/cpanel/scripts/check_cpanel_pkgs --list-only
2026-09-04 07:38:37 [INFO] Running: /usr/bin/dnf list extras
2026-09-04 07:38:43 [INFO] Unsupported YUM repo enabled 'csm' without packages installed from /etc/yum.repos.d/csm.repo, these will be disabled before ELevation
2026-09-04 07:38:43 [ERROR] 6 package(s) installed from unsupported YUM repo 'elevate-testing' from /etc/yum.repos.d/elevate-testing.repo
2026-09-04 07:38:43 [INFO] Running: /usr/bin/dnf makecache
2026-09-04 07:38:44 [INFO] ELevate Testing 20 kB/s | 3.0 kB
2026-09-04 07:38:44 [INFO] EA4 ( EasyApache 4 - c8 ) 4.8 kB/s | 2.9 kB
2026-09-04 07:38:45 [INFO] cPanel Plugins project 4.7 kB/s | 2.9 kB
2026-09-04 07:38:45 [INFO] AlmaLinux 8 - BaseOS 17 kB/s | 3.8 kB
2026-09-04 07:38:46 [INFO] AlmaLinux 8 - AppStream 19 kB/s | 4.3 kBA.2 Successful target userspace creation and transaction
2026-09-06 03:23:16 [INFO] ==> Processing phase `TargetTransactionFactsCollection`
2026-09-06 03:23:16 [INFO] ====> * create_iso_repofile
2026-09-06 03:23:16 [INFO] ====> * target_userspace_creator
2026-09-06 03:23:16 [INFO] Initializes a directory to be populated as a minimal environment to run binaries from the target system.
2026-09-06 03:23:22 [INFO] Extra Packages for Enterprise Linux 9 - x86_64
2026-09-06 03:23:35 [INFO] AlmaLinux 9.8 - BaseOS
2026-09-06 03:23:44 [INFO] AlmaLinux 9.8 - AppStream
2026-09-06 03:23:53 [INFO] AlmaLinux 9.8 - CRB
2026-09-06 03:24:21 [INFO] Dependencies resolved.
...
2026-09-06 03:24:21 [INFO] filesystem x86_64 3.16-5.el9 almalinux9-baseos 1.1 M
...
2026-09-06 03:24:42 [INFO] Running transaction test
2026-09-06 03:24:43 [INFO] Transaction test succeeded.
2026-09-06 03:24:43 [INFO] Running transaction
2026-09-06 03:24:44 [INFO] Running scriptlet: filesystem-3.16-5.el9.x86_64A.3 Transition into upgrade initramfs generation
2026-09-06 03:25:54 [INFO] Transaction Summary
2026-09-06 03:25:54 [INFO] Install 286 Packages
2026-09-06 03:25:54 [INFO] Upgrade 847 Packages
2026-09-06 03:25:54 [INFO] Remove 43 Packages
2026-09-06 03:25:54 [INFO] Downgrade 7 Packages
2026-09-06 03:25:54 [INFO] Total size: 1.3 G
...
2026-09-06 03:29:29 [INFO] Complete!
2026-09-06 03:29:30 [INFO] Applying transaction workaround - remove obsolete RPM GPG keys from RPM DB
2026-09-06 03:29:30 [INFO] Applying transaction workaround - IRB directory fix
2026-09-06 03:29:30 [INFO] Applying transaction workaround - import trusted gpg keys to RPM DB
2026-09-06 03:29:30 [INFO] ==> Processing phase `InterimPreparation`
2026-09-06 03:29:30 [INFO] ====> * mount_unit_generator
2026-09-06 03:29:30 [INFO] Sets up storage initialization using systemd's mount units in the upgrade container.
2026-09-06 03:29:31 [INFO] ====> * common_leapp_dracut_modules
2026-09-06 03:29:31 [INFO] Influences the generation of the initram disk
2026-09-06 03:29:31 [INFO] ====> * upgrade_initramfs_generator
2026-09-06 03:29:31 [INFO] Creates the upgrade initramfsA.4 Dracut mount failure exposed after tty0 diagnostic
[ OK ] Mounted /sysroot.
[ OK ] Reached target Initrd Root File System.
Mounting /sysroot/boot...
Mounting /sysroot/tmp...
Starting Mountpoints Configured in the Real Root...
[ OK ] Mounted /sysroot/boot.
Mounting /boot...
Mounting /sysroot/boot/efi...
[ OK ] Mounted /boot.
[ OK ] Mounted /sysroot/boot/efi.
/usr/tmpDSK: Can't lookup blockdev
[FAILED] Failed to mount /sysroot/tmp.
See 'systemctl status sysroot-tmp.mount' for details.
[DEPEND] Dependency failed for Local File Systems.
[ OK ] Started Emergency Shell.
[ OK ] Reached target Emergency Mode.
Generating "/run/initramfs/rdsosreport.txt"
Entering emergency mode. Exit the shell to continue.A.5 Generated mount unit evidence
systemctl cat sysroot-tmp.mount
# Warning: sysroot-tmp.mount changed on disk, the version systemd has loaded is outdated.
# /usr/lib/systemd/system/sysroot-tmp.mount
# Automatically generated by systemd-fstab-generator
[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target
[Mount]
Where=/sysroot/tmp
What=/usr/tmpDSK
Type=ext4
Options=defaults,noauto
systemctl show sysroot-tmp.mount -p FragmentPath -p SourcePath
FragmentPath=/usr/lib/systemd/system/sysroot-tmp.mount
SourcePath=/etc/fstabA.6 Stage 4 MySQL failure from elevate-cpanel.log
2026-09-06 05:31:29 [INFO] Restoring config files for package: 'ea-php84-runtime'
2026-09-06 05:31:29 [INFO] Restoring config files for package: 'ea-profiles-cpanel'
2026-09-06 05:31:29 [INFO] Restoring config files for package: 'ea-wappspector'
2026-09-06 05:35:03 [DEBUG] Running 'post_distro_upgrade' from CryptoPolicies component.
2026-09-06 05:35:03 [INFO] Running: /usr/bin/update-crypto-policies --set DEFAULT:SHA1
2026-09-06 05:35:03 [INFO] Setting system policy to DEFAULT:SHA1
2026-09-06 05:35:03 [DEBUG] Running 'post_distro_upgrade' from MySQL component.
2026-09-06 05:35:03 [INFO] Restoring MySQL 8.0
2026-09-06 05:35:03 [INFO] Beginning upgrade to MySQL 8.0
2026-09-06 05:37:25 [FATAL] FAILED to upgrade to MySQL 8.0
2026-09-06 05:37:25 [FATAL] Unable to install MySQL 8.0.
2026-09-06 05:37:25 [ERROR] The elevation process failed during stage 4.
2026-09-06 05:37:25 [ERROR] You can continue the process after fixing the errors by running:
/usr/local/cpanel/scripts/elevate-cpanel --continue
...
/usr/local/cpanel/bin/whmapi1 start_background_mysql_upgrade version=8.0A.7 MySQL background job package failure
Looking for providers that will cause dependency problems: mysql-server
There are no installed packages that cause known dependency problems
...
Package mysql-community-devel-8.0.46-1.el8.x86_64 is already installed.
Error:
Problem: package mysql-community-server-8.0.46-1.el9.x86_64 from mysql80-community requires mysql-community-icu-data-files = 8.0.46-1.el9, but none of the providers can be installed
- cannot install the best candidate for the job
- package mysql-community-icu-data-files-8.0.46-1.el9.x86_64 from mysql80-community is filtered out by exclude filtering
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
Restarting mysqld service.
The "mysql" service is disabled.
"/usr/bin/yum" reported error code "1" when it ended.A.8 Versionlock discovery and post-fix transaction
dnf versionlock list
mysql-community-client-plugins-0:8.0.46-1.el8.*
mysql-community-client-0:8.0.46-1.el8.*
mysql-community-common-0:8.0.46-1.el8.*
mysql-community-icu-data-files-0:8.0.46-1.el8.*
mysql-community-libs-0:8.0.46-1.el8.*
# after deleting the MySQL locks, dry-run:
Dependencies resolved.
Installing:
mysql-community-server x86_64 8.0.46-1.el9
Upgrading:
mysql-community-client x86_64 8.0.46-1.el9
mysql-community-client-plugins x86_64 8.0.46-1.el9
mysql-community-common x86_64 8.0.46-1.el9
mysql-community-devel x86_64 8.0.46-1.el9
mysql-community-icu-data-files x86_64 8.0.46-1.el9
mysql-community-libs x86_64 8.0.46-1.el9
Transaction Summary
Install 1 Package
Upgrade 6 Packages
Operation aborted. # expected because --assumenoA.9 Successful MySQL job after versionlock removal
Restarting mysqld service.
Waiting for "mysql" to restart ... waiting for "mysql" to initialize ... finished.
Service Status
mysqld (/usr/sbin/mysqld) is running as mysql ...
Startup Log
Sep 06 05:53:35 rocky8.usd21.org systemd[1]: Starting MySQL Server...
Sep 06 05:53:39 rocky8.usd21.org systemd[1]: Started MySQL Server.
Log Messages
2026-09-06T05:53:39Z /usr/sbin/mysqld: ready for connections. Version: '8.0.46' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server - GPL.
mysql restarted successfully.
MySQL upgrade completed successfullyAppendix B – Log locations and evidence map
| Location | What it proved in this incident | Key evidence |
|---|---|---|
| /var/log/elevate-cpanel.log | Wrapper stage progression, preflight repository blockers, Leapp output, Stage 4 component failure | elevate-testing blocker; target transaction success; upgrade_initramfs_generator; MySQL post_distro_upgrade failure |
| /var/log/leapp/leapp-report.txt and .json | Preupgrade findings/inhibitors and Leapp-specific report details | Used to verify zero inhibitors before start and distinguish warnings from blockers |
| /run/initramfs/rdsosreport.txt | Dracut emergency-boot diagnostic report | Generated when /sysroot/tmp mount failed |
| systemctl cat/show sysroot-tmp.mount inside upgrade initramfs | Exact generated mount definition and provenance | What=/usr/tmpDSK; Where=/sysroot/tmp; SourcePath=/etc/fstab |
| /var/cpanel/logs/mysql_upgrade.<ID>/unattended_background_upgrade.log | Actual cPanel MySQL package transaction and service restart results | Filtered EL9 ICU package before fix; MySQL upgrade completed successfully after versionlock removal |
| dnf versionlock list | Hidden transaction filter not visible in dnf.conf excludes | mysql-community packages pinned to 8.0.46-1.el8.* |
| GRUB ELevate-Upgrade-Initramfs entry | Kernel/initramfs and console routing for offline upgrade boot | console=ttyS0 originally; tty0 diagnostic exposed the real error |
15. Bottom line
This incident was not one monolithic ELevate failure. It was a sequence of independently diagnosable integration gaps. Each successful intervention moved the failure boundary forward, which makes the case valuable for upstream engineering: the DNF exclude was validated by target userspace success after removal; the tmpDSK diagnosis was validated by the offline transaction proceeding only after removal of the generated mount unit; and the MySQL diagnosis was validated by the EL9 transaction resolving immediately after deleting EL8-specific versionlocks. Those are strong, reproducible signals for preflight and integration improvements.
