Created attachment 4980 [details]
This the rdsosreport with rd.debug in the kernel command line.

This the rdsosreport with rd.debug in the kernel command line.

CC: (none) => brouard.nicolas

(In reply to Nicolas Brouard from comment #0)
> Steps to Reproduce:
> 1. Copy /usr on a logical volume named /usr
> 2.  Add this new volume in /etc/fstab with other like /home, /var etc.
> 3. Boot

This is not sufficient. The task of mounting /usr belongs to the initrd. As we use host-only initird's you have to regenerate your initrd if you make fundamental changes to your storage configuration. /usr is considered one of those fundamental changes (unlike, say, /var).

If you regenerate your initrd (normally just a matter of running "dracut -f") you should be fine.

dracut should only initialse the VGs that it knows are essential for early boot. In your case it should not initialise home or var. It stores the fact it needs to initialise these things via the /etc/cmdline.d/ files inside the initrd. There should be one file which specifies which VGs are needed.

Keep a backup of your current initrd for safety!

HTHs

I should also note that normally, the lvm2 tools (vgchange etc) wouldn't even be included in the initrd, but because you have a swap on LVM and because we need to check swap for hibernation images, dracut has pulled this in. You're kinda lucky that this was available otherwise it would make booting a bit trickier!!

The alternative is to use a non-host-only, generic initrd, but these are much larger and thus slow down general booting at the expense of future reconfiguration flexibility.

As far as I'm concerned, this is not a Mageia bug.
It would have working if you'd done an install on LVM.
If you manually switch to LVM, you've to manually fix your system.

Status: NEW => RESOLVED
Resolution: (none) => INVALID

(In reply to Colin Guthrie from comment #2)
> (In reply to Nicolas Brouard from comment #0)
> > Steps to Reproduce:
> > 1. Copy /usr on a logical volume named /usr
> > 2.  Add this new volume in /etc/fstab with other like /home, /var etc.
> > 3. Boot
> 
> This is not sufficient. The task of mounting /usr belongs to the initrd. As
> we use host-only initird's you have to regenerate your initrd if you make
> fundamental changes to your storage configuration. /usr is considered one of
> those fundamental changes (unlike, say, /var).
> 
> If you regenerate your initrd (normally just a matter of running "dracut
> -f") you should be fine.
> 
> dracut should only initialse the VGs that it knows are essential for early
> boot. In your case it should not initialise home or var. It stores the fact
> it needs to initialise these things via the /etc/cmdline.d/ files inside the
> initrd. There should be one file which specifies which VGs are needed.
> 
> Keep a backup of your current initrd for safety!
> 
> HTHs

Thank you for your kind reply. Yes, in order to speed the boot process, dracut doesn't take an "external" /usr into account anymore. Many people are considering this regression to be a bug and patches were circulating (gentoo, fedora) to have it back. It concerns only /usr, neither /var or /home.

Yes, it was a raw installation with a brand new UEFI server, and using the LVM at installation. 

Unfortunately, the Mageia 4 standard installation DVD crashes at the very beginning because uefi does not support vesa and you are stuck with a very low resolution (huge characters on the screen). Sometimes you can't see the OK or "Next" button and you have to use the TAB and guess that you are on the correct icon. 

For the LVM step, everything worked well: once a free partition has been created by Windows 8 server (it is mandatory to let Windows 8 shrinks itself and creates an empty huge partition), you can define a small / partition and an huge LVM partition, named here vg-mga. Within this vg-mga LVM partition, you can create the swap, /usr, /var, /home.

Even if these partitions are created and formatted (ext4), and if installation of softwares worked well, I have been suprised by the fact that none of the softwares were installed on the LVM partitions! For an unknown reason, everything went on the / partition, creating /usr /var /home again on the / partition and letting the LVM partitions empty. Did I make an error? I am not sure because the LVM swap partition was always mounted correctly. 

When I said that the installation went well, I am exagerating. 

Thanks to https://wiki.mageia.org/en/UEFI_how-to because it is mandatory (when you are new to UEFI) to follow these instructions. But even grub2 did not work.

I had to: chroot /mnt grub2-install -target-x86_64-efi -efi-directory=/boot/EFI/ -booloader-id-Mageia -recheck -debug 

This step went fine and I was able to boot in UEFI mode on a Mageia 4 new install!

It should be noticed that on new PC, and if you want to keep Windows 8 in a dual boot, you must install Linux in UEFI mode. Fedora is able to manage UEFI correctly but I am used to Mandrake since a while...

Then came the LVM problem noticed here. No problem to boot Mageia 4 on / but on a heavy server you need to install LVM.

After discovering that my /usr and /var /home LVM partitions were empty and unmounted, I feed them and tried to boot. Unfortunately dracut crashed and exit to a dracut shell. As explained, doing lvm vgchange -ay and exit, booted on the /usr of the native / partition, did not mount the lvm /usr, but mounted the /var and /home from the /etc/fstab!

According to your advices, and to summary my workaround for the /usr LVM "bug", you just have:
- to boot, 
- hit F3 and add:
 rd.lvm.lv=vg-mga/usr on the kernel command line.

It supposes that your /etc/fstab has the line:
/dev/vg-mga/usr /usr ext4 defaults 1 2
among others:
/dev/vg-mga/tmp /tmp ext4 rw,nosuid,nodev,relatime,data=ordered 1 2
/dev/vg-mga/var /var ext4 rw,nosuid,nodev,relatime,data=ordered 1 2
/dev/vg-mga/swap swap swap defaults 0 0


And you lvm /usr partition will be mounted correctly (hiding the old usr from the / partition).

But in order to have it permanently fixed, you can create a directory /etc/cmdline.d and a file /etc/cmdline.d/90lvm.conf
with a single line:
rd.lvm.lv=vg-mga/usr

As you explained, it is not necessary to add the swap because it is added by dracut because of the resume in the kernel command line.

And thus, you have to go to /boot, copy your initrd-3.12.9-server-1.mga4.img to initrd-3.12.9-server-1.mga4-ori.img for safety
and use a "dracut -f" command to create a newer initrd-3.12.9-server-1.mga4.img initrd image.

Many thanks to all,

Status: RESOLVED => REOPENED
Resolution: INVALID => (none)

(In reply to Nicolas Brouard from comment #5)
> Thank you for your kind reply. Yes, in order to speed the boot process,
> dracut doesn't take an "external" /usr into account anymore.

This is not true. I'm not 100% sure what you mean by "external" /usr here, but it certainly does work. I have server systems where /usr is on RAID and they certainly is mounted by the generated initrd.

> Many people are
> considering this regression to be a bug and patches were circulating
> (gentoo, fedora) to have it back. It concerns only /usr, neither /var or
> /home.

If you're talking about having a system install being able to mount /usr by itself (i.e. outside of initrd) this is definitely NOT a regression or something which we will be adding back.

It's a massive pain to maintain this and there are numerous race conditions around early boot when /usr is not mounted. The correct solution is to (as we do) ensure that the initrd is responsible for mounting /usr.

I've followed the Gentoo stuff and even want to the talk they had on their approach to this at last year's FOSDEM. It's was laughably clear that they had absolutely no clue about all the intricacies involved here and such a system is doomed to failure (and indeed they have pretty much abandoned this approach now).

I know the fedora people very well and I know they do NOT consider the old way as sensible and will not be going back.

Again, the statement you made here about this being a regression is simply wrong - it's just that the responsibilities of whose job it is has moved. It's no longer userspace's job, it's the initrd's.

I'd be happy to explain all the reasons why this is all so wrong, but it's probably off topic for this bug - mail me if you want to discuss.

 

> Then came the LVM problem noticed here. No problem to boot Mageia 4 on / but
> on a heavy server you need to install LVM.

LVM is almost certainly the wrong solution these days too, but that's another story. It is still supported here :)
 
> After discovering that my /usr and /var /home LVM partitions were empty and
> unmounted, I feed them and tried to boot. Unfortunately dracut crashed and
> exit to a dracut shell. As explained, doing lvm vgchange -ay and exit,
> booted on the /usr of the native / partition, did not mount the lvm /usr,
> but mounted the /var and /home from the /etc/fstab!
> 
> According to your advices, and to summary my workaround for the /usr LVM
> "bug", you just have:
> - to boot, 
> - hit F3 and add:
>  rd.lvm.lv=vg-mga/usr on the kernel command line.
> 
> It supposes that your /etc/fstab has the line:
> /dev/vg-mga/usr /usr ext4 defaults 1 2
> among others:
> /dev/vg-mga/tmp /tmp ext4 rw,nosuid,nodev,relatime,data=ordered 1 2
> /dev/vg-mga/var /var ext4 rw,nosuid,nodev,relatime,data=ordered 1 2
> /dev/vg-mga/swap swap swap defaults 0 0
> 
> 
> And you lvm /usr partition will be mounted correctly (hiding the old usr
> from the / partition).

Yup, that's all good so far.


> But in order to have it permanently fixed, you can create a directory
> /etc/cmdline.d and a file /etc/cmdline.d/90lvm.conf
> with a single line:
> rd.lvm.lv=vg-mga/usr

No, sorry this isn't needed. I was just explaining the details of how things work *inside* the initrd - this file/folder is NOT needed on your system itself
 - it's created automatically by dracut when it generates the initrd and you do not need to create this manually. The only thing you need to do is recreate your initrd after a successful boot by running "dracut -f". It will automatically generate this file for you (in the initrd) and the next boot should not require any manual intervention.

Dracut includes the "usrmount" module to take care of this for you.

 
> As you explained, it is not necessary to add the swap because it is added by
> dracut because of the resume in the kernel command line.

Yes, and likewise for the /usr mount point. dracut does this automatically too.

> And thus, you have to go to /boot, copy your initrd-3.12.9-server-1.mga4.img
> to initrd-3.12.9-server-1.mga4-ori.img for safety
> and use a "dracut -f" command to create a newer
> initrd-3.12.9-server-1.mga4.img initrd image.

Yes, but as mentioned all you need to do is get the thing to boot and then run "dracut -f". That's all, no other changes needed and this is expected behaviour when you change / or /usr partitions.

Closing again as it's not a bug (there may be some bugs in installation process as you outlined but that's unrelated to this).

Status: REOPENED => RESOLVED
Resolution: (none) => INVALID

(In reply to Colin Guthrie from comment #6)
> (In reply to Nicolas Brouard from comment #5)
> > Thank you for your kind reply. Yes, in order to speed the boot process,
> > dracut doesn't take an "external" /usr into account anymore.
> 
> This is not true. I'm not 100% sure what you mean by "external" /usr here,
> but it certainly does work. I have server systems where /usr is on RAID and
> they certainly is mounted by the generated initrd.

It wasn't in my case, using LVM.

> It's a massive pain to maintain this and there are numerous race conditions
> around early boot when /usr is not mounted. The correct solution is to (as
> we do) ensure that the initrd is responsible for mounting /usr.

I agree, that is the reason for my report.

> I've followed the Gentoo stuff and even want to the talk they had on their
> approach to this at last year's FOSDEM. It's was laughably clear that they
> had absolutely no clue about all the intricacies involved here and such a
> system is doomed to failure (and indeed they have pretty much abandoned this
> approach now).
> 
> I know the fedora people very well and I know they do NOT consider the old
> way as sensible and will not be going back.
> 
> Again, the statement you made here about this being a regression is simply
> wrong - it's just that the responsibilities of whose job it is has moved.
> It's no longer userspace's job, it's the initrd's.
> 
> I'd be happy to explain all the reasons why this is all so wrong, but it's
> probably off topic for this bug - mail me if you want to discuss.
> 
>  
> 
> > Then came the LVM problem noticed here. No problem to boot Mageia 4 on / but
> > on a heavy server you need to install LVM.
> 
> LVM is almost certainly the wrong solution these days too, but that's
> another story. It is still supported here :)
Any hint for a better solution. In fact /var/lib/mysql are growing and should be on a separate lv, as well as /var/log . Since AIX, logical volume manager was a very good solution. I am interested in discovering alternate solutions.

>  
> > After discovering that my /usr and /var /home LVM partitions were empty and
> > unmounted, I feed them and tried to boot. Unfortunately dracut crashed and
> > exit to a dracut shell. As explained, doing lvm vgchange -ay and exit,
> > booted on the /usr of the native / partition, did not mount the lvm /usr,
> > but mounted the /var and /home from the /etc/fstab!
> > 
> > According to your advices, and to summary my workaround for the /usr LVM
> > "bug", you just have:
> > - to boot, 
> > - hit F3 and add:
> >  rd.lvm.lv=vg-mga/usr on the kernel command line.
> > 
> > It supposes that your /etc/fstab has the line:
> > /dev/vg-mga/usr /usr ext4 defaults 1 2
> > among others:
> > /dev/vg-mga/tmp /tmp ext4 rw,nosuid,nodev,relatime,data=ordered 1 2
> > /dev/vg-mga/var /var ext4 rw,nosuid,nodev,relatime,data=ordered 1 2
> > /dev/vg-mga/swap swap swap defaults 0 0
> > 
> > 
> > And you lvm /usr partition will be mounted correctly (hiding the old usr
> > from the / partition).
> 
> Yup, that's all good so far.
> 
> 
> > But in order to have it permanently fixed, you can create a directory
> > /etc/cmdline.d and a file /etc/cmdline.d/90lvm.conf
> > with a single line:
> > rd.lvm.lv=vg-mga/usr
> 
> No, sorry this isn't needed. I was just explaining the details of how things
> work *inside* the initrd - this file/folder is NOT needed on your system
> itself
>  - it's created automatically by dracut when it generates the initrd and you
> do not need to create this manually. The only thing you need to do is
> recreate your initrd after a successful boot by running "dracut -f". It will
> automatically generate this file for you (in the initrd) and the next boot
> should not require any manual intervention.
> 
> Dracut includes the "usrmount" module to take care of this for you.

That was not the case for me, sorry. Did you try with a /usr on an lvm logical volume?
I agree that usrmount module is loaded by standard initrd and I was surprised that it did not the job (the usrmount was mounting /home /var /tmp from /etc/fstab but not /usr). Sorry, it is a fact.

> 
>  
> > As you explained, it is not necessary to add the swap because it is added by
> > dracut because of the resume in the kernel command line.
> 
> Yes, and likewise for the /usr mount point. dracut does this automatically
> too.

Not true, unfortunately.

> > And thus, you have to go to /boot, copy your initrd-3.12.9-server-1.mga4.img
> > to initrd-3.12.9-server-1.mga4-ori.img for safety
> > and use a "dracut -f" command to create a newer
> > initrd-3.12.9-server-1.mga4.img initrd image.


> Yes, but as mentioned all you need to do is get the thing to boot and then
> run "dracut -f". That's all, no other changes needed and this is expected
> behaviour when you change / or /usr partitions.
This new initrd will not include /usr from the lvm. Did you try?
You need to had rd.lvm.lv=vg-mga/usr on the kernel command line or in /etc/cmdline.d/90lvm.conf in order to create a correct initrd with dracut -f .

> 
> Closing again as it's not a bug (there may be some bugs in installation
> process as you outlined but that's unrelated to this).

I am re-opening the bug unless someone did the test that he/she can have a lvm logical volume with /usr and that he can boot without exiting via the dracut shell OR that a simple dracut -f will generate a new initrd which will mount a /usr logical volume.


I discovered another problem after Mageia 4 installation. The gpt EFI partition got a surprising boot flag "pmbr_boot" which makes some workstations ignore the scanning of the EFI partition (see bug https://bugzilla.redhat.com/show_bug.cgi?id=844551). And thus I could no more get a UEFI boot of the hard drive. The hard drive disappeared from the Boot setup. The station could only boot on legacy mode (or on Live CD with UEFI support).

The solution consists in clearing this boot flag, for example with parted:
 disk_set pmbr_boot off
 quit

Then Windows 8.1 appeared again. I spent a few days on this problem. And will probably not be alone. It could be an important information when Mageia 5 will support UEFI.

In any case, I thank you for the time spent to answer me.

Status: RESOLVED => REOPENED
Resolution: INVALID => (none)

So there is quite a lot to quote, so I won't bother but will try to address everything.

I think you should open a new bug for the UEFI/GPT related stuff. I suspect grub2 is at fault here for setting the flag. It's usually to blame which is why I avoid it like the plague :)

As for the /usr thing, yes I have tried this several times.

I've actually just fired up a VM to recreate the conditions of your install.

1. I created a / partition of 4GB
2. I created a LVM VG for the remaining 4GB of space and create a swap and /usr mount within it.
3. I verified the installer did mount /usr into the "install tree" (i.e. /mnt: so my / was mounted as /mnt and the LVM was correctly mounted as /mnt/usr).
4. In order to simulate your installer problems, I umounted /mnt/usr and continued the install.
5. After install I edited my /mnt/fstab to ensure that /usr was not mounted (you have to do this just before rebooting the installer).

6. First boot went OK and /usr was (obviously) not mounted and all the /usr content was in the /usr folder of the / partition (i.e. not a mountpoint).

7. I mkdir'ed /usr2, mounted the LVM /usr there and did rsync -aHAX /usr/ /usr2/
8. I umounted  /usr2, editied fstab to reinstate it and did mount /usr.
9. I regenerated initrd with: dracut -f
10. I rebooted and /usr is correctly mounted in the unitrd

In order to verify 10, I booted with rd.break=pre-pivot. This stage should see /sysroot/usr NOT yet mounted, but /sysroot should be mounted OK.

In order to verify things, I did a cat /etc/cmdline.d/90lvm.conf (still in the initrd shell) and it included the lvs for both swap and usr as expected, so all is well there.

After typing exit, we go to the second breakpoint (before switch root) and at this point in time /sysroot/usr IS now mounted correctly.

Typing exit one last time and the system boots (just as it does without any rd.break) and /usr is obviously mounted OK.

In order to be doubly sure, and restore the "lost" space in / for the content overmounted in /usr, I booted with rd.break=pre-pivot, and in the shell I did: rm -rf /sysroot/usr; mkdir /sysroot/usr


So all in all, I cannot reproduce your problem, and everything appears to be working correctly. I'm at somewhat of a loss to explain why this solution is not working for you.

I suspect that for some reason or another, the 90lvm.conf file generated in your initrd does not include the /usr lvm.

So I would ask the following:

1. On a booted system, what does /etc/fstab say regarding /usr?
2. What is the output from dracut -vvv -f (pipe it to a file and attach it)
3. Assuming your initrd is /boot/initrd.img, can you do:
 a) mkdir foo
 b) cd foo
 c) cat /boot/initrd.img | cpio -id
 d) cat etc/cmdline.d/90lvm.conf

(note the LACK of a leading / in point d)) 

This might provide some clues as to why things are not working for you.


Oh and regarding LVM and RAID alternatives, it's off topic so we probably shouldn't discuss here, but btrfs is the future. I'm not sure I'd personally trust it for MySQL data just yet tho', so LVM is a reasonable solution in the mean time and my previous tongue-in-cheek comment was a little factitious! (I actually use LVM myself for snapshot-based MySQL backups which is way faster for backup and recovery for MySQL than dump based ones - of course btrfs will provide the same advantages in this cases which I'm looking forward to :))

(In reply to Colin Guthrie from comment #8)
> So there is quite a lot to quote, so I won't bother but will try to address
> everything.
> 
> I think you should open a new bug for the UEFI/GPT related stuff. I suspect
> grub2 is at fault here for setting the flag. It's usually to blame which is
> why I avoid it like the plague :)

I will do it.
> 
> As for the /usr thing, yes I have tried this several times.
>
Thanks, I think that I understood most of the errors. The error was probably in mounting /usr with the installer. The /usr lvm was mounted as any other filesystem with nosuid: 
/dev/vg-mga/usr /usr ext4 rw,nosuid,nodev,relatime,data=ordered 1 2
And I remarked it because I could not use su.
And thus dracut -f probably did not include it in the initrd (etc/cmdline/90lvm.conf).
Currently my /etc/fstab concerning /usr is:
  /dev/vg-mga/usr /usr ext4 defaults 1 2
I don't know if it is ok. But it works.

Concerning the swap, I never changed it and was defined by the installer as:
  /dev/vg-mga/swap swap swap defaults 0 0


Thank you, for telling me how to look into an image with cpio -id because I have been able to see what was generated with the initrd in file etc/cmdline/90lvm.conf 

I discovered that the img is gzipped, thus it has to be unzipped before unarchiving it (it wasn't mention in your post):
 mkdir foo; cd foo;
 dracut -vvv -f foo.img >foo-img.log 2>&1
 gzip -dc foo.img | cpio -id 
 cat etc/cmdline.d/90lvm.conf


I still have a 3.12.8 kernel img created from the first install:
(dated late févr. 10 02:40) initrd-3.12.8-server-2.mga4.img
and there is only "rd.lvm.lv=cg-mga/swap" in etc/cmdline.d/90lvm.conf

But in my current img I have both of them:
  rd.lvm.lv=vg-mga/usr rd.lvm.lv=vg-mga/swap

Thus, you are RIGHT, it is not necessary to add in /etc/cmdline.d/90lvm.conf in the / partition. I verified that adding rd.lvm.lv=/var in this file, doesn't add /var in the corresponding initrd command line.

Thus, yes, there is no bug if you do a correct installation. But it is not possible to do a correct installation of Mageia 4 on a brand new PC with Windows 8.1 because only UEFI mode is possible (unless destroying the all partitions and going back to MBR and 2TB limitation).

We could stop here, thank you a lot, and close the bug. But if I want to remember where was my error during the installation, here are some advices for people willing to install Mageia 4 in dual boot (probably most users with a brand new PC) with Windows 8.1 . The PC runs on an Intel E3 1225v3 processor . The advantage is that it is very powerful, silent and uses 240W only.  No additional video card because the Intel video is in the processor: enough for a non gamer. I found this processor in the HP Z230SFF, which is quite compact for a workstation. I will use it on Linux permanently. But I need to test some programs under native Windows 8.1 (ie without Virtual Box).

------------------

As I previously said, I used the diskdrake installer to create the lvm partitions. As far as I remember, I asked automatic partitioning (I never know which are the current size for /usr /var swap etc.). But the /usr has been created on the root partition (as you know the screen was huge and I couldn't see all the window) and not on the lvm even if (from what I remember, they have been formatted with diskdrake). May be I move back to the / partition instead of vg-mga and asked wrongly for automatic partitioning from there.

And as I said in a previous post too, finishing installation was complex because UEFI and grub2 did not work (you need to install grub2-efi from a ALt-F2 terminal and moved back to the installer). Setting X11 was also a nightmare and I think that I booted on the live DVD in uefi mode and copied the generated xorg.conf in order to get X11 working. I also created grubx64.efi from the live CD.


> This might provide some clues as to why things are not working for you.


Thanks a lot for your time.

> 
> Oh and regarding LVM and RAID alternatives, it's off topic so we probably
> shouldn't discuss here, but btrfs is the future. I'm not sure I'd personally
> trust it for MySQL data just yet tho', so LVM is a reasonable solution in
> the mean time and my previous tongue-in-cheek comment was a little
> factitious! (I actually use LVM myself for snapshot-based MySQL backups
> which is way faster for backup and recovery for MySQL than dump based ones -
> of course btrfs will provide the same advantages in this cases which I'm
> looking forward to :))
Thank you, but will wait for brrfs...

Status: REOPENED => RESOLVED
Resolution: (none) => FIXED

For information the pmbr_boot flag bug has been reported on https://bugs.mageia.org/show_bug.cgi?id=12822 .