| Summary: | kernel install/remove process taking unreasonably long when there are lots of partitions | ||
|---|---|---|---|
| Product: | Mageia | Reporter: | Felix Miata <mrmazda> |
| Component: | RPM Packages | Assignee: | Martin Whitaker <mageia> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | Normal | CC: | bequimao.de, bittwister2, mageia, mageia, thierry.vignaud, tmb |
| Version: | Cauldron | ||
| Target Milestone: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| See Also: | https://bugs.mageia.org/show_bug.cgi?id=24681 | ||
| Whiteboard: | MGA5TOO | ||
| Source RPM: | drakxtools | CVE: | |
| Status comment: | |||
| Attachments: |
urpme kernel transaction excerpt from journalctl -b-1 log
Script to test partition table read speed journalctl | grep bootloader-config from mga6 host gx62b |
||
|
Description
Felix Miata
2015-04-23 02:21:56 CEST
@Colin: not using the blkid cache does hurt when there's quite a lot of partitions http://gitweb.mageia.org/software/drakx/commit/perl-install?id=ecd8db27c8ffb76ebf799e0de0dcf64c3ce256fe CC:
(none) =>
mageia, thierry.vignaud Maybe it hurts, but the upstream maintainers are advising not to use it. I guess as per the quoted conversation it should be migrated to either use the udev db or lsblk. No improvement as of 3.19.8 kernels.
Samuel Verschelde
2015-05-21 09:05:59 CEST
Summary:
kernel install/remove process taking unreasonably long =>
kernel install/remove process taking unreasonably long when there are lots of partitions
Samuel Verschelde
2015-05-21 09:06:33 CEST
CC:
(none) =>
tmb
Samuel Verschelde
2015-05-21 09:07:41 CEST
Whiteboard:
(none) =>
MGA5TOO FOR_ERRATA Also, do you use grub2? if yes, do you've os-prober installed? (In reply to Thierry Vignaud from comment #4) > Also, do you use grub2? Not as master bootloader, and not on any Mageia installations. Still a problem as of kernel-desktop-4.6.0-1.mga6, ~5 minutes to install on 2.8GHz i686 host gx280 with 31 partitions.
Samuel Verschelde
2016-10-18 13:11:47 CEST
Whiteboard:
MGA5TOO FOR_ERRATA =>
MGA5TOO
Samuel Verschelde
2016-11-01 11:54:14 CET
Assignee:
bugsquad =>
kernel This looks the same terminal - not original - problem as Bug 16010 (now closed); and Bug 18538. If somebody Bugzilla responsible agrees that 18538 & this bug 15752 are effectively duplicates, I suggest closing one of them appropriately: probably this one. Looking back at 16010 re the long time to run os-prober, hence do installations & kernel updates, I have a significant addition (which I shall post also on 18538): Early in that closed bug Barry asked about a certain partition 10; comments 5 & 12. Real EFI/GPT hardware: It happens that I have recently removed Korora [ex Fedora] which was on partition 10, because of its very intrusive nature in NVRAM & the ESP, probably due to it doing 'secure boot'. At the same time I *increased* the number of partitions to 16. I have noticed since that os-prober (at least; yet to try a kernel upgrade or new installation) happens much faster, at least for Mageia 5, yet to try 6. From up-to-date Mageia 5: # time os-prober /dev/sda11:Mageia 6 (6):Mageia:linux /dev/sda13:Mageia 5 (5):Mageia1:linux /dev/sda14:LMDE MATE Edition (1):LinuxMint:linux /dev/sda16:Mageia 5 (5):Mageia2:linux /dev/sda2@/EFI/Microsoft/Boot/bootmgfw.efi:Windows Boot Manager:Windows:efi /dev/sda8:Manjaro Linux (16.10):ManjaroLinux:linux /dev/sda9:openSUSE 12.3 (x86_64):SUSE:linux real 0m25.038s which suggests that the presence of Korora/Fedora *did* matter, rather than the number of partitions. CC:
(none) =>
lewyssmith This bug is probably distinguishable from the others in that in no case is grub2 or os-prober installed on any of my Mageia, Fedora, Debian or openSUSE installations. I do have a few *buntu installations I allow to install and keep grub2, but os-prober is absent from all, and none UEFI/GPT capable have it enabled. I jumped the gun; apologies. Two bugs indeed. Your remark suggests that there is probably another reason (i.e. not os-prober, itself perhaps a victim) for the 10m delays some of us see - which we may have wrongly attributed. Still very much a problem with current 6RC installing or rebuilding initrd for 4.9.25 using 64-bit host fi965 with /dev/sda34, more than 4 minutes. kernel-desktop-4.4.16-1.mga5 took 4 minutes 20 seconds to remove in preparation to upgrade 64-bit host gx62b with 34 sda partitions to mga6rc. Looks like anything that calls 'bootloader-conf' causes these symptoms. I tried removing KDE in order to install TDE prior to upgrading MGA5 to MGA6. The delay waiting for bootloader-conf to complete running 'urpmi trinity-tdebase trinity-tdm trinity-ksnapshot trinity-konsole trinity-mix' was truly noxious for the context. Why bootloader-conf had any reason to be called in order to add a new DE is inexplicable. Not limited to 32-bit. Didn't get any better that I can tell in 6 final. Hardware:
i586 =>
All The background issue is os-prober doing bad things sometimes IMHO Source RPM:
kernel =>
kernel, os-prober None of my installations have grub2 or os-prober installed. On host big41 I just did urpmi upgrade mga6 on /dev/sda32 to mga7b2:
> inxi -GxxSm
System: Host: big41 Kernel: 5.0.2-desktop-1.mga7 x86_64 bits: 64 compiler: gcc v: 8.3.1 Desktop: Trinity wm: Twin
dm: startx Distro: Mageia 7 mga7
Memory: RAM: total: 3.73 GiB used: 252.7 MiB (6.6%)...
CPU: Topology: Dual Core model: Intel Core2 Duo E7600 bits: 64 type: MCP arch: Penryn rev: A L1 cache: 64 KiB
L2 cache: 3072 KiB
flags: lm nx pae sse sse2 sse3 sse4_1 ssse3 vmx bogomips: 12235
Speed: 2762 MHz min/max: 1603/3066 MHz Core speeds (MHz): 1: 1917 2: 1840
Graphics: Device-1: Intel 4 Series Integrated Graphics vendor: Biostar Microtech Intl Corp driver: i915 v: kernel
bus ID: 00:02.0 chip ID: 8086:2e32
Display: tty server: Mageia X.org 1.20.4 driver: modesetting alternate: fbdev,intel,vesa resolution: 1920x1200~60Hz
OpenGL: renderer: Mesa DRI Intel G41 v: 2.1 Mesa 19.0.0 direct render: Yes
mga4 remains on /dev/sda40, last partition. I followed up the upgrade with:
# time urpme kernel-desktop-4.9.56-1.mga6 ... 5m28.118s real.
# time urpme kernel-desktop-4.14.30-3.mga6 ... 5m47.075s real.
# time urpme kernel-desktop-4.14.69-1.mga6 ... 5m28.912s real.
Before the upgrade I forgot to use time, but when I removed three other kernels, each seemed to take closer to 3 minutes than to 5. :(
(In reply to Felix Miata from comment #16) > Before the upgrade I forgot to use time, but when I removed three other > kernels, each seemed to take closer to 3 minutes than to 5. :( Heheh, yup. My guess, the solution will be a grub fix/patch to umount commands. I wrote my own /etc/grub.d/ script to generate menu entries by label and wondered why it was taking so long to run. What I found was caused by systemd taking its time returning from the umount command for each partition I had mounted to checked for /boot. :( Solution was to add --lazy to umount command. CC:
(none) =>
bittwister2 Why would there be any mounting or unmounting going on during kernel installation or removal on an installation with neither os-prober nor grub2* nor syslinux* nor pxelinux installed, as all my mga6+ installations are? From the attached log file, most of the time is being taken between these two messages: Apr 22 18:04:00 gx27b.ij.net bootloader-config[1772]: found a dos partition table on /dev/sda at sector 0 Apr 22 18:12:51 gx27b.ij.net bootloader-config[1772]: guess_geometry_from_partition_table sda: 30401/255/63 Felix, could you confirm that is still the case. If so, the first message comes from partition_table::read_primary(), called at the start of partition_table::read(), and the second message comes from partition_table::dos::guess_geometry_from_partition_table(), called (indirectly) at the end of partition_table::read(). So it would seem to be the libDrakX partition table read subroutine that is slow. CC:
(none) =>
mageia The script bootloader-config is crap. I was not aware that it is actually used in Mageia tools. cf. https://bugs.mageia.org/show_bug.cgi?id=18534 CC:
(none) =>
bequimao.de Created attachment 10945 [details]
Script to test partition table read speed
@Felix, to confirm my analysis, please try running this Perl script. You'll need to run it as root, because it reads the disk partition table. If it shows the same slow behaviour, try installing the perl-Devel-Trace package and run it again by
perl -d:Trace read_pt.pl
This will produce a lot of output, but should pause at the point in the code that is causing the problem. Please capture a few lines of output when it pauses.
Tried with host gx62b Mageia 6 last updated in September:
# cat b15752.sh
#!/bin/perl
use lib qw(/usr/lib/libDrakX);
use partition_table;
print "Detect devices\n";
my @drives = detect_devices::hds();
foreach my $hd (@drives) {
$hd->{file} = devices::make($hd->{device});
}
print "Get geometries\n";
my @drives = partition_table::raw::get_geometries(@drives);
foreach my $hd (@drives) {
my $name = $hd->{file};
print "Read partition table on $name\n";
partition_table::read($hd);
}
# sh b15752.sh
b15752.sh: line 2: syntax error near unexpected token `('
b15752.sh: line 2: `use lib qw(/usr/lib/libDrakX);'
It's a Perl script, not a shell script. Try perl b15752.sh It took only a few seconds to run: # perl b15752.sh Detect devices Get geometries HDIO_GETGEO on /dev/sda succeeded: heads=255 sectors=63 cylinders=60801 start=0 id2hd: 0xdf5ee108=>sda id2edd: 0xdf5ee108=>/sys/firmware/edd/int13_dev80 geometry_from_edd sda 0xdf5ee108: 60801/255/63 Read partition table on /dev/sda running: blkid -o udev -p /dev/sda running: blkid -o udev -p /dev/sda found a dos partition table on /dev/sda at sector 0 guess_geometry_from_partition_table sda: 60801/255/63 More than three minutes later, the disk was still churning, with top showing a lot of systemd-udevd PIDs. Created attachment 10946 [details]
journalctl | grep bootloader-config from mga6 host gx62b
Highlights:
Sep 10 03:01:22 gx62b bootloader-config[9979]: fs::get::device2part: unknown device <</dev/sda19>>
Sep 10 03:01:22 gx62b bootloader-config[9979]: writing grub config to /boot/grub/menu.lst
Sep 10 03:02:57 gx62b bootloader-config[11668]: HDIO_GETGEO on /dev/sda succeeded: heads=255 sectors=63 cylinders=60801 start=0
Sep 10 03:02:57 gx62b bootloader-config[11668]: id2hd: 0xdf5ee108=>sda
Sep 10 03:03:03 gx62b bootloader-config[11668]: guess_geometry_from_partition_table sda: 60801/255/63
Sep 10 03:03:03 gx62b bootloader-config[11668]: running: udevadm settle
Sep 10 03:05:03 gx62b bootloader-config[11668]: running: blkid -o udev -p /dev/sda1
Sep 10 03:05:03 gx62b bootloader-config[11668]: blkid gave: vfat 303F-1AFF P01WINDOS
Sep 10 03:05:11 gx62b bootloader-config[11668]: fs::get::device2part: unknown device <</dev/sda19>>
Sep 10 03:05:11 gx62b bootloader-config[11668]: writing grub config to /boot/grub/menu.lst
Sep 10 03:09:02 gx62b bootloader-config[13600]: HDIO_GETGEO on /dev/sda succeeded: heads=255 sectors=63 cylinders=60801 start=0
Sep 10 03:09:03 gx62b bootloader-config[13600]: id2hd: 0xdf5ee108=>sda
Sep 10 03:09:13 gx62b bootloader-config[13600]: guess_geometry_from_partition_table sda: 60801/255/63
Sep 10 03:09:13 gx62b bootloader-config[13600]: running: udevadm settle
Sep 10 03:11:13 gx62b bootloader-config[13600]: running: blkid -o udev -p /dev/sda1
Sep 10 03:11:13 gx62b bootloader-config[13600]: blkid gave: vfat 303F-1AFF P01WINDOS
Sep 10 03:11:22 gx62b bootloader-config[13600]: adding /boot/vmlinuz-4.14.65-desktop-1.mga6
Sep 10 03:11:22 gx62b bootloader-config[13600]: running: mkinitrd -f /boot/initrd-4.14.65-desktop-1.mga6.img 4.14.65-desktop-1.mga6
Sep 10 03:12:54 gx62b bootloader-config[13600]: running: /usr/share/bootsplash/scripts/make-boot-splash /boot/initrd-4.14.65-desktop-1.mga6.img 1024
Sep 10 03:13:01 gx62b bootloader-config[13600]: current labels: Mageia 6 defkernel Mageia 6 prvkernel linux
4.14.25 & 4.14.65 are the only currently installed kernels.
OK, that latest extract from the journal is quite different from the original mga5 one. And we can see that there is a 2 minute delay after each call to 'udevadm settle', which ties in with what you observe in comment 24.
Lewis Smith
2019-04-17 21:25:56 CEST
CC:
lewyssmith =>
(none)
Ulrich Beckmann
2019-04-19 19:40:43 CEST
See Also:
(none) =>
https://bugs.mageia.org/show_bug.cgi?id=24681 As explained in comment #8, this bug cannot have anything to do with os-prober. This is from a 31 partition installation on 64bit mga6 host g5eas with only one installed kernel: # time urpmi kernel-desktop-latest To satisfy dependencies, the following packages are going to be installed: Package Version Release Arch (medium "Core Backports") kernel-desktop-4.18.20-1.mga6 1 1.mga6 x86_64 kernel-desktop-latest 4.18.20 1.mga6 x86_64 63MB of additional disk space will be used. 57MB of packages will be retrieved. Proceed with the installation of the 2 packages? (Y/n) y http://mirrors.us.kernel.org/mageia/distrib/6/x86_64/media/core/backports/kernel-desktop-4.18.20-1.mga6-1-1.mga6.x86_64.rpm 0% of 0 completed, ETA = --:--:--, speed = 0 0% of 57.4M completed, ETA = 0:06:49, speed = 160k ... 100% of 57.4M completed, ETA = --:--:--, speed = 1472k http://mirrors.us.kernel.org/mageia/distrib/6/x86_64/media/core/backports/kernel-desktop-latest-4.18.20-1.mga6.x86_64.rpm 0% of 0 completed, ETA = --:--:--, speed = 0 100% of 46574 completed, ETA = --:--:--, speed = 494k installing kernel-desktop-latest-4.18.20-1.mga6.x86_64.rpm kernel-desktop-4.18.20-1.mga6-1-1.mga6.x86_64.rpm from /var/cache/urpmi/rpms Preparing... ############################################# 1/2: kernel-desktop-4.18.20-1.mga6 ############################################# 2/2: kernel-desktop-latest ############################################# Creating: target|kernel|dracut args|basicmodules You should restart your computer for kernel-desktop-4.18.20-1.mga6 real 2m56.737s user 0m54.768s sys 0m44.495s # Source RPM:
kernel, os-prober =>
kernel,udev The root cause of this bug is that the libDrakX partition table read subroutine uses the libparted ped_disk_probe() function to query the partition table type. The ped_disk_probe() function opens the raw disk device for read/write access (even though it only wants to read). That causes the kernel to generate a change event for each partition on the device, which triggers udev actions. A MBR/DOS extended partition table is a linked list of entries. Currently the libDrakX partition table read subroutine is calling the ped_disk_probe() function when it reads each entry. This means that the number of kernel events generated is proportional to the square of the number of partitions on the device. We can fix this by caching the value returned by ped_disk_probe(). Status:
NEW =>
ASSIGNED Please test drakxtools 18.13-1 on your mga7 system when it reaches the mirrors. Looks like you did well!
# rpm -qa | egrep 'kernel|drakxtools'
drakxtools-backend-18.13-1.mga7
drakxtools-curses-18.13-1.mga7
kernel-firmware-nonfree-20190314-1.mga7.nonfree
kernel-desktop-latest-5.0.2-1.mga7
kernel-firmware-20190114-1.mga7
kernel-desktop-4.14.104-2.mga6-1-1.mga6
kernel-desktop-5.0.2-1.mga7-1-1.mga7
kernel-userspace-headers-5.0.2-1.mga7
# blkid | tail -n1
/dev/sda41: LABEL="mga4-h8s" UUID="52d99d42...
# time urpme kernel-desktop-4.14.104-2.mga6
removing kernel-desktop-4.14.104-2.mga6-1-1.mga6.x86_64
removing package kernel-desktop-4.14.104-2.mga6-1-1.mga6.x86_64
1/1: removing kernel-desktop-4.14.104-2.mga6-1-1.mga6.x86_64
#################...
real 0m30.208s
user 0m2.416s
sys 0m0.770s
# inxi -S
System: Host: big41 Kernel: 5.0.2-desktop-1.mga7 x86_64 bits: 64 Desktop: Trinity Distro: Mageia 7 mga7
# time urpmi kernel-desktop-latest
To satisfy dependencies, the following packages are going to be installed:
Package Version Release Arch
(medium "Core Release")
kernel-desktop-5.0.9-1.mga7 1 1.mga7 x86_64
kernel-desktop-latest 5.0.9 1.mga7 x86_64
65MB of additional disk space will be used.
59MB of packages will be retrieved.
Proceed with the installation of the 2 packages? (Y/n) y
http://mirror.math.princeton.edu/pub/mageia/distrib/7/x86_64/media/core/release/kernel-desktop-5.0.9-1.mga7-1-1.mga7.x86_64.rpm
http://mirror.math.princeton.edu/pub/mageia/distrib/7/x86_64/media/core/release/kernel-desktop-latest-5.0.9-1.mga7.x86_64.rpm
installing kernel-desktop-latest-5.0.9-1.mga7.x86_64.rpm kernel-desktop-5.0.9-1.mga7-1-1.mga7.x86_64.rpm from /var/cache/urpmi/rpms
Preparing... ########################...
1/2: kernel-desktop-5.0.9-1.mga7
########################...
2/2: kernel-desktop-latest ########################...
1/1: removing kernel-desktop-latest-5.0.2-1.mga7.x86_64
########################...
Creating: target|kernel|dracut args|basicmodules
You should restart your computer for glibc, kernel-desktop-5.0.9-1.mga7, systemd
real 1m50.080s
user 0m14.137s
sys 0m4.296s
# uname -a
Linux gx280 4.14.106-desktop-1.mga6 #1 SMP Thu Mar 14 19:13:32 UTC 2019 i686 i686 i686 GNU/Linux
## fdisk -l tail
/dev/sda34 976752063 976768064 16002 7.8M 11 Hidden FAT12
# grep RETT /etc/os-release
PRETTY_NAME="Mageia 7"
# rpm -qa | grep nel-desk
kernel-desktop-4.14.106-1.mga6-1-1.mga6
kernel-desktop-4.14.30-3.mga6-1-1.mga6
kernel-desktop-4.14.70-2.mga6-1-1.mga6
kernel-desktop-latest-4.14.106-1.mga6
kernel-desktop-4.14.10-1.mga6-1-1.mga6
kernel-desktop-4.14.44-2.mga6-1-1.mga6
# rpm -qa | grep kxtools-back
drakxtools-backend-18.13-1.mga7
[root@gx280 ~]# time urpme kernel-desktop-4.14.30-3.mga6
removing kernel-desktop-4.14.30-3.mga6-1-1.mga6.i586
removing package kernel-desktop-4.14.30-3.mga6-1-1.mga6.i586
1/1: removing kernel-desktop-4.14.30-3.mga6-1-1.mga6.i586
###########################...
writing /var/lib/rpm/installed-through-deps.list
real 0m41.981s
user 0m11.606s
sys 0m1.934s
# time urpme kernel-desktop-4.14.10-1.mga6
removing kernel-desktop-4.14.10-1.mga6-1-1.mga6.i586
removing package kernel-desktop-4.14.10-1.mga6-1-1.mga6.i586
1/1: removing kernel-desktop-4.14.10-1.mga6-1-1.mga6.i586
###########################...
real 0m39.472s
user 0m9.102s
sys 0m1.600s
# time urpme kernel-desktop-4.14.44-2.mga6
removing kernel-desktop-4.14.44-2.mga6-1-1.mga6.i586
removing package kernel-desktop-4.14.44-2.mga6-1-1.mga6.i586
1/1: removing kernel-desktop-4.14.44-2.mga6-1-1.mga6.i586
###########################...
writing /var/lib/rpm/installed-through-deps.list
real 0m37.221s
user 0m8.736s
sys 0m1.758s
# # time urpme kernel-desktop-4.14.70-2.mga6
removing kernel-desktop-4.14.70-2.mga6-1-1.mga6.i586
removing package kernel-desktop-4.14.70-2.mga6-1-1.mga6.i586
1/1: removing kernel-desktop-4.14.70-2.mga6-1-1.mga6.i586
###########################...
writing /var/lib/rpm/installed-through-deps.list
real 0m35.255s
user 0m7.424s
sys 0m1.674s
# time urpmi kernel-desktop-latest
# time urpmi kernel-desktop-latest
To satisfy dependencies, the following packages are going to be installed:
Package Version Release Arch
(medium "Core Release")
kernel-desktop-5.0.9-1.mga7 1 1.mga7 i586
kernel-desktop-latest 5.0.9 1.mga7 i586
59MB of additional disk space will be used.
53MB of packages will be retrieved.
Proceed with the installation of the 2 packages? (Y/n) y
http://mirror.math.princeton.edu/pub/mageia/distrib/7/i586/media/core/release/kernel-desktop-5.0.9-1.mga7-1-1.mga7.i586.rpm
http://mirror.math.princeton.edu/pub/mageia/distrib/7/i586/media/core/release/kernel-desktop-latest-5.0.9-1.mga7.i586.rpm
installing kernel-desktop-latest-5.0.9-1.mga7.i586.rpm kernel-desktop-5.0.9-1.mga7-1-1.mga7.i586.rpm from /var/cache/urpmi/rpms
Preparing... #################...
1/2: kernel-desktop-5.0.9-1.mga7
#################...
2/2: kernel-desktop-latest #################...
1/1: removing kernel-desktop-latest-4.14.106-1.mga6.i586
#################...
Creating: target|kernel|dracut args|basicmodules
remove-boot-splash: Format of /boot/initrd-5.0.9-desktop-1.mga7.img not recognized
You should restart your computer for dbus, glibc, kernel-desktop-5.0.9-1.mga7, systemd
real 3m3.160s
user 0m56.462s
sys 0m17.606s
Good, setting this to fixed then. Resolution:
(none) =>
FIXED |