Bug 25069 - remote upgrade 6->7 using ssh and dnf results in an unusable system
Summary: remote upgrade 6->7 using ssh and dnf results in an unusable system
Status: RESOLVED FIXED
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: 7
Hardware: All Linux
Priority: Normal major
Target Milestone: ---
Assignee: Mageia tools maintainers
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-07-08 02:06 CEST by Jeff Robins
Modified: 2019-12-01 19:32 CET (History)
4 users (show)

See Also:
Source RPM:
CVE:
Status comment:


Attachments
dnf_issues1.txt (7.62 KB, text/plain)
2019-07-08 02:07 CEST, Jeff Robins
Details
install.log (52.67 KB, application/x-gzip)
2019-07-14 07:12 CEST, Jeff Robins
Details
ddebug.log (301.87 KB, application/x-gzip)
2019-07-14 07:13 CEST, Jeff Robins
Details
log.txt (811.96 KB, application/x-bzip)
2019-07-14 07:15 CEST, Jeff Robins
Details
latest dnf log (94.05 KB, text/plain)
2019-07-16 16:44 CEST, Jeff Robins
Details
dnf system upgrade log (98.42 KB, application/x-gzip)
2019-07-16 16:45 CEST, Jeff Robins
Details
dnf log that covers the upgrade period (321.39 KB, application/x-gzip)
2019-07-16 16:46 CEST, Jeff Robins
Details

Description Jeff Robins 2019-07-08 02:06:20 CEST
Description of problem:
I have a headless server and was doing an upgrade from Mageia 6 to Mageia 7 using DNF.  Everything was updated and "dnf system-upgrade --releasever 7 download --allowerasing" ran without a probelm.  However, once I ran "dnf system-upgrade reboot" the server kicked me off of ssh and I could not reconnect.

I waited 2 hours after the busy light stopped flashing and I still couldn't connect over ssh (nmap showed the port as closed) to reboot the system.  Once the system was back up I still could not connect over ssh.

After connecting a monitor, keyboard and mouse to the system I was able to see the system had booted, but further efforts to continue the upgrade failed.  "dnf", "urpmi" and "rpm" wouldn't work (I think it was complaining about the wrong magic number).  I attempted to do a network install, but that failed after a while with a long list of dependency issues.

After rebooting again I found that rpm worked, but not urpmi or dnf.  I manually downloaded and installed the following packages, which got urpmi up and running:
1) python2-rpm-4.14.2.1-12.mga7.x86_64.rpm
2) python3-rpm-4.14.2.1-12.mga7.x86_64.rpm
3) rpm-4.14.2.1-12.mga7.x86_64.rpm
4) urpmi-8.118-1.mga7.noarch.rpm
5) perl-5.28.2-1.mga7.x86_64.rpm
6) perl-base-5.28.2-1.mga7.x86_64.rpm
7) perl-Filesys-Df-0.920.0-26.mga7.x86_64.rpm
8) perl-Math-Int64-0.540.0-10.mga7.x86_64.rpm
9) perl-Term-ReadKey-2.380.0-1.mga7.x86_64.rpm
10) perl-URPM-5.21-1.mga7.x86_64.rpm
11) vim-enhanced-8.1.1048-1.mga7.x86_64.rpm

I ran "urpmi.update -a" and "urpmi --auto-select" and worked on the dependencies until I have an almost updated and running system.

1) ssh was updated, but now it says "error: Unable to load host key "/etc/ssh/ssh_host_key": invalid format"
-Prior to this it was complaining that the openssl version was a mismatch
-"OpenSSL version mismatch. Built against 1000212f, you have 10000003"
-"/sbin/sshd version"
OpenSSH_7.5p1, OpenSSL 1.0.0-fips 29 Mar 2010
2) dnf doesn't work, even after uninstalling and reinstalling
- Please see attached console output
3) Conflict: "mageia-release < 7-0.2 conflicts with systemd-units-241-8.mga7.x86_64
"

How reproducible:
I only had one system to try this on.

Steps to Reproduce:
1. Run the instructions from "https://wiki.mageia.org/en/Mageia_7_Release_Notes#Upgrading_online.2C_using_DNF_.28CLI.29"
Comment 1 Jeff Robins 2019-07-08 02:07:09 CEST
Created attachment 11163 [details]
dnf_issues1.txt
Jeff Robins 2019-07-12 07:14:18 CEST

Version: 6 => 7

Comment 2 Marja Van Waes 2019-07-13 10:28:45 CEST
Hi Jeff,

When you wrote:

> I attempted to do a network install, but that failed after a while with a long
> list of dependency issues.

were you then talking about a clean install, or about an upgrade install.

* If the latter, then the logs from the upgrade attempt with DNF and from the reboot after that, should still be available. Please attach log.txt that is the result of running, as root:

   journalctl -a --since="YYYY-MM-DD hh:mm" --until="YYYY-MM-DD hh:mm" > log.txt

and adjust the --since time to right before you started to upgrade with DNF and the --until time to when you saw you couldn't reconnect using ssh.

In the case of a network upgrade install, please do also attach 
  /root/drakx/report.bug 
which has the time stamp of the upgrade. You'll probably need to compress it with xz, but will need to do that in a different directory or need to rename the already existing report.bug.xz (which dates from when you first installed Mageia).

* If you did a clean network install, then please just attach

  /root/drakx/report.bug.xz

Keywords: (none) => NEEDINFO
CC: (none) => mageiatools, marja11, ngompa13

Comment 3 Jeff Robins 2019-07-14 07:11:43 CEST
I did an upgrade and when that failed I tried a clean install.  I have my personal data on separate partitions, so I wasn't worried about losing the root partition's data.

I have a report.bug.xz, but it's from Aug 2013.  I do have recent versions of ddebug.log and install.log.  I gzipped them and I attached them to the bug report.  

I also attached the journal output.  I wasn't able to zero in the time very well from when I started to when I finished. Apparently my ssh is constantly under attack and that puts a lot of crap in my log and that made it hard to find the important stuff.
Comment 4 Jeff Robins 2019-07-14 07:12:31 CEST
Created attachment 11181 [details]
install.log
Comment 5 Jeff Robins 2019-07-14 07:13:04 CEST
Created attachment 11182 [details]
ddebug.log
Comment 6 Jeff Robins 2019-07-14 07:15:37 CEST
Created attachment 11183 [details]
log.txt
Comment 7 Neal Gompa 2019-07-16 11:59:22 CEST
If dnf is working enough for the CLI to run, could you try capturing the output of the system upgrade transaction?

# dnf system-upgrade log --number=-1 > dnf-system-upgrade.log

Also, a copy of /var/log/dnf.log would be helpful, too.
Comment 8 Jeff Robins 2019-07-16 16:44:39 CEST
Created attachment 11189 [details]
latest dnf log
Comment 9 Jeff Robins 2019-07-16 16:45:59 CEST
Created attachment 11190 [details]
dnf system upgrade log

This is probably from the last time I tried to use the system upgrade option to see if it would fix the issues
Comment 10 Jeff Robins 2019-07-16 16:46:42 CEST
Created attachment 11191 [details]
dnf log that covers the upgrade period
Ulrich Beckmann 2019-07-17 15:21:16 CEST

CC: (none) => bequimao.de

Marja Van Waes 2019-10-20 20:35:00 CEST

Keywords: NEEDINFO => (none)
Assignee: bugsquad => mageiatools

Comment 11 Jeff Robins 2019-11-27 07:35:12 CET
So I renamed my /var/lib/dnf directory to /var/lib/dnf.old and now I can run "dnf upgrade" without a problem.  I'll close this as resolved once I actually see it update a package (no updates were needed right now).
Comment 12 Jeff Robins 2019-12-01 19:32:28 CET
I was able to upgrade a bunch of packages, so I'm closing this as Resolved.  The solution was to move the "/var/lib/dnf" directory so a new one was created.

Resolution: (none) => FIXED
Status: NEW => RESOLVED


Note You need to log in before you can comment on or make changes to this bug.