Addendum : If I want to use Mageia 2 (Beta3), I need to delete and format my crypted /home partition, then I will loose all my personnal data.
Can you find a fix for this big problem ?
Thanks.

ALT+F3 console output :

*lang:fr charset:iso-8859-15.....

installer window :

7. I click on the button Use to select the filesystem /home to map it on this filesystem but the window ask me to type the "Key to crypt" (in french : Clé de chiffrement) 

ALT+F3 console output :

* running: /sbin/insmod /tmp/dm_crypt.ko
* running: /sbin/insmod /tmp/gf128mul.ko
* running: /sbin/insmod /tmp/xts.ko
* running: /sbin/insmod /tmp/cbc.ko
* running: /sbin/insmod /tmp/sha256_generic.ko
* running: /sbin/insmod /tmp/aes_generic.ko
* running: /sbin/insmod /tmp/aes_x86_64.ko
* running: cryptsetup luksOpen /dev/vg0/1 crypt_vg0_1 --key-file /tmp/.dmcrypt_key-706
* running: udevadm settle
* running: dmsetup table crypt_vg0_1
* could not find the device 252:3 for mapper/crypt_vg0_1
* running: blkid -o udev -p /dev/mapper/crypt_vg0_1  
* blkid gave: ext4 e2e53..xxxxxxxxxxxxxxxxxxxx (UUID or serial number ????)
* dmcrypt: found mapper/crypt_vg0_1 type ext4 with rootDevice

installer window :

Now in partition window, I have "connected to crypt_vg0_1" (in french: à connecter sur crypt_vg0_1)
but no "mount point/map" button (in french: "Point de montage") with the others Format, Use and Delete buttons.

installer window :
    
I reclick on the Use button for the secondth time and I have this 
console log :
* running: cryptsetup luksOpen /dev/vg0/1 crypt_vg0_1 --key-file /tmp/.dmcrypt_key-706
* error: cryptsetup failed

window message :
i have an error window with the error text : "cryptsetup failed"

I found that my bug 5661 is the same bug that : 
cannot use luks encrypted partitions during the install 
https://bugs.mageia.org/show_bug.cgi?id=3749

There have been work on this lately.
I suggest you try the network installation method (use a freshly downloaded boot.iso) and it will pull latest versions of everything including installer from the repos.


Now, a workaround tip for comment 1 : 
I once backupped data from filesystem on partition in an encrypted lvm using the live cd from sysresccd.org.  Of course i had to enter all commandas to decrypt vgscan mount etc in IIRC i found it all from sysresccd great on site manual, and possible some google hit.

First, if possible, i suggest making any kind of complete copy of all your valuable files.

CC: (none) => fri

Hello Morgan,

>I suggest you try the network installation method (use a freshly downloaded
>boot.iso) and it will pull latest versions of everything including installer
>from the repos.

I have tested the network installation method with this wiki article : https://wiki.mageia.org/en/Installation_Media#Network_Installs and next https://wiki.mageia.org/en/Boot.iso_install.

With the network installation method, I have the Mageia 2 -RC installer but with this new installer I have the same problem. The bug is the same with this new boot.iso and installer files with 20120428 release date that does not work.

Any idea to give me some command line to use for sending you more information to fix this problem ?

Hello Morgan,

>Now, a workaround tip for comment 1 :
Yes but is not a good solution if an user want to keep his home directory/folder with all his custom settings for his window manager or other programs. Sometimes, if you want to redo all these custom parameters, it's a big work and cost time.
If mageia developers can find a solution, it will be the best solution and more easy for our Mageia users.
Don't forget that Mageia is a linux distribution for newbies.

After the password has been entered, does the tab for vg0 appear?

Are you clicking on the tab for vg0 before trying to select the /home
filesystem?

CC: (none) => davidwhodgins

@ comment 6:  of course best is if it works OK. But always good to have copies.  I sometimes do images of disk or partitions using gparted running from alternate system booted on USB (sysrresccd.org is splendid) sometimes (more flexible) i copy the filesystem, there fsarchiver is very nice.

I don't know if the problem I encountered when I installed Mageia 2-RC from Mageia-2-rc-LiveCD-KDE4-Europe1-Americas-x86_64-CD is related or not, but it seems to me it's similar enough to this bug so I'll comment here.

When I installed, I selected custom partitioning and created a 128 MB /boot partition -- /dev/sda1. I then created an LVM volume group using all remaining disk space, dividing it up into the following volumes:

  vg-mga-root
  vg-mga-usr
  vg-mga-swap
  vg-mga-home

I entered labels for all volumes since I hate dealing with UUIDs if (when) a disk crashes. I then selected the encrypted filesystem option for vg-mga-home, which the installer created using the password I supplied. When the install finished and the system rebooted, it prompted for my encrypted /home passphrase, but it hung when it was starting the storage subsystems.

From dmesg, I saw that dracut (by way of cryptroot-ask) had called:

  cryptsetup luksOpen /dev/mapper/vg--mga-home luks-<UUID>

The mount line from /etc/fstab:

  /dev/mapper/crypt_vg_mga_home /home ext4 noatime 0 0

I then did a little digging and determined that I needed to create /etc/crypttable with the following line:

  crypt_vg_mga_home /dev/vg-mga/home

After a reboot, the system prompted for the encrypted /home password and started properly.

Should the disk partitioning done during install have included creation of /etc/crypttab??

CC: (none) => scott

Hi,

This bug was filed against cauldron, but we do not have cauldron at the moment.

Please report whether this bug is still valid for Mageia 2.

Thanks :)

Cheers,
marja

Keywords: (none) => NEEDINFO

To answer the the question whether this still applies: It does.
I just did a new install (mga2, x86_64) and created a LVM which should contain everything but /boot (i.e., /, /home, ...) in encrypted format. Unfortunately, it was not possible to access updated rpms at install time--but with the "stock mga2 x86_64 ISO", it was (still) not possible to mount /[root].

CC: (none) => ueberall

I second Markus' comment.  Just installed Mageia 2 x86_64 with all encrypted except /boot and swap.  System appears to boot and then cannot find root - never prompted for my passphrases.

Filesystems mount perfectly onto another machine as external mounts - passphrase accepted and works just fine.

CC: (none) => heymanj

Please look at the bottom of this mail to see whether you're the assignee of this  bug, if you don't already know whether you are.


If you're the assignee:

We'd like to know for sure whether this bug was assigned correctly. Please change status to ASSIGNED if it is, or put OK on the whiteboard instead.

If you don't have a clue and don't see a way to find out, then please put NEEDHELP on the whiteboard.

Please assign back to Bug Squad or to the correct person to solve this bug if we were wrong to assign it to you, and explain why.

Thanks :)

**************************** 

@ the reporter and persons in the cc of this bug:

If you have any new information that wasn't given before (like this bug being valid for another version of Mageia, too, or it being solved) please tell us.

@ the reporter of this bug

If you didn't reply yet to a request for more information, please do so within two weeks from now.

Thanks all :-D

can reproduce with a netinstall

Priority: Normal => release_blocker
Summary: Mageia2 Beta3 Installer DVD 64bits : Can not unlock and set /home on my crypted home partition on logical volume (LVM) to use it => Can not unlock and set /home on my crypted home partition on logical volume (LVM) to use it
Source RPM: (none) => drakxtools
Whiteboard: MGA2TOO => MGA2TOO 3alpha3

Is it still valid for up to date cauldron ?

CC: (none) => ennael1

(In reply to Anne Nicolas from comment #15)
> Is it still valid for up to date cauldron ?

Sorry, but I have actually no computer or free partition for testing this.

I'll test with the beta3 x86-64 iso shortly.

Created attachment 3597 [details]
gzipped report.bug

Bug is still present.

After clicking on the encrypted partition, selecting use,
and entering the passphrase, the cryptsetup luksOpen command
does work, but, from the log, it appears the dmsetup command
is failing, and then diskdrake doesn't show any filesystems.

Clicking on another tab, then clicking on the volume group tab,
the filesystems are shown, but selecting the encrypted filesystem
just shows the use button again.  I think the relevant part of
the log is ...
* running: cryptsetup luksOpen /dev/13/root crypt_13_root --key-file /tmp/.dmcrypt_key-90
* running: udevadm settle
* running: dmsetup table crypt_13_root
* could not find the device 252:0 for mapper/crypt_13_root
* running: blkid -o udev -p /dev/mapper/crypt_13_root
* blkid gave: ext4 81a9f656-1d14-4366-84c4-910d097213ca

As discussed on irc, part of the problem is the kernel modules not being
loaded properly. Once they are loaded, creating/opening the encrypted
device works properly, but the partitions shown still go blank, so
it isn't possible to specify a mount point, or mount the partition yet.

Still trying to figure out exactly where the problem is.

I've made a little bit of progress.  Comparing Mageia 1, where this works,
to Mageia 2, where it doesn't, it looks like changes that were put in to
fix having a lvm physical volume on an encrypted partition, broke having
an encrypted filesystem within an lvm volume group.

Still trying to figure out how to actually fix it.

Going to need some help from someone who knows perl better than I do.

In Mageia 2, Line 963 and 964 of /usr/lib/libDrakX/diskdrake/interactive.pm has
    # Detect LVMs on top of dmcrypt
    $all_hds->{lvms} = [ fsedit::lvms($all_hds) ];
which is not present in the Mageia 1 version.

Commenting out that line stops the partitions shown from going blank,
but still doesn't show the change from being an unopened encrypted
device, to an open one.

I think the line needs to be changed such that it's only run if the
dmcrypt device is not in an lvm volume group.

Then it needs to do whatever is required to detect what's in the
open encrypted device.

Created attachment 3626 [details]
screenshot

I've been doing a little testing with: Mageia-3-beta3-x86_64-DVD.iso

My tests are for creating new encrypted partitions, testing inside a virtualbox vm.   I can get a number of error messages and little success.

Test1: 
Custom disk partitioning:
1. create a small /boot partition, ext4
2. create a large partition, as LVM
3. in the LVM partition, create a ext4, / partition
4. in the LVM partition, create a xfs /home partition, tick the encrypted checkbox.  enter: testtest as the encryption key.
5. screenshot1.png is displayed.    This is not an error message, but is a problem.
In the console (F1), we see: "Gtk-CRITICAL **: gtk_cell_view_set_cell_data: assertion `cell_view->priv->displayed_rows != NULL` failed at /usr/lib/libDrakX/interactive/gtk.pm line 524.
6. click on the sda tab. 
7. click on the LVM volume tab.  Now the contents of the LVM volume are displayed agan.
8. selecting the crypted home partition, shows it as Encrypted (inactive)., However, this is not true, looking at ls /dev/mapper/crypt_vg_mga_home exists. IE:  cryptsetup has the device open.

*** Bug 8194 has been marked as a duplicate of this bug. ***

CC: (none) => callimera.42

Created attachment 3644 [details]
Instructions to 'fix' the Mageia 2 encryption installation issue

Just wondering if the is any progress on this bug yet?

I think that when it's resolved, it might help resolve a number of related bugs, like: #5983, #7807

I can just confirm that the bug is still there in Mageia 3 beta 4

CC: (none) => makowski.mageia

Colin, sg has changed in initscripts and it now need /etc/crypttab to be filled.
Is there a way to reverse that?

CC: (none) => mageia
Summary: Can not unlock /home on my crypted LVM to use it (need to write /etc/crypttable) => Can not unlock /home on my crypted LVM to use it (need to write /etc/crypttab)

Humm, actually we've code in order to do this...
See fs::dmcrypt::save_crypttab()

@Theirry, not quite sure what you mean here. AFAIUI the installer doesn't rely on initscripts :s

However, I'm not sure all the necessary rules are in place. 

As per step 8 in Glen's comment above, the device /dev/mapper/crypt_vg_mga_home should be created. In my mga2 based test a similar layout did not create this device on reboot, only in the installer. After I manually fixed my VM to use a different /dev/mapper/luks-$UUID value (which was equiv) the system booted fine. Even after upgrading to mga3, the /dev/mapper/crypt_vg_mga_home did not appear...

So I think there may be some issues with udev rules still, but not 100% sure it is related (although it easily could be).

(of course my experience could just be fallout from an mga2 installer bug so don't take too much credence of it just yet!)

OK, I've just tested this layout and can confirm it's a problem still in mga3. The structure I used for testing is deliberately complex and not really that practical.

Procedure:
 1. MGA3 (cauldron) boot.iso from today
 2. HTTP install (mirrors.kernel.org)
 3. All final partitions are ext4
 4. Create /boot
 5. Create *unencrypted* LVM VG, vg-mga, the add one *encyrpted* LV on it for / (using all space on VG), password "pp"
 6. Create *encrypted* LVM VG, vg-secret, password "oo", then add one *unencrypted* LV on it for /usr
 7. Create *unencrypted* LVM VG, vg-variable, then add one *unencrypted* LV on it for /var (unneeded for this example, but was used for testing other issue!)
 8. Install works with the caveate that diskdrak in the installer does NOT show the encrypted /  in the tabs. If I ignore this issue and continue the install works fine (same was true in mga2 when the same procedure was performed FWIW)
 9. In the installer the /dev/mapper/crypt_vg_mga_root device exists and is mounted correctly. The resulting menu.lst file for grub has this as it's root= argument.
 10. On first boot, dracut prompts from the two passwords (repeatedly it seems - it appears to try and remount the encrypted partition over and over when the real device doesn't show up). Drops to a debug shell.
 11. In order to fix boot, do:
 11a. cd /dev/mapper/
 11b. ln -s luks-* crypt_vg_mga_root
 11c. cd /lib/dracut/hooks/initqueue/finished
 11d. rm -f *
 11e. exit
 12. Boot works.
 13. Edit /boot/grub/menu.lst
 14. Fix root= to refer to the full /dev/mapper/luks-* path.
 15. Reboot
 16. Still fails waiting for crypt_vg_mga_root :( (different to mga2)
 17. Fix by doing rm /lib/dracut/hooks/initqueue/finished/*crypt_vg_mga_root.sh
 18. exit
 19. Boots.
 20. Regenerate initrd (dracut -f)
 21. Reboot.
 22. Enter passwords once ("oo" first, then "pp" although both say sda6 at the prompt :s)
 23. Boot works fine :)
 24. No crypt_vg_mga_root is created in /dev/mapper.

This could be a very different problem, but it's still a problem to solve :D

The big questions for me here is:
 1. What in the installer creates /dev/mapper/crypt_vg_mga_root and why does it not create it in the real system?
 2. Why does diskdrake not show the encrypted / parition on top of an unencrypted LVM?

Just for the record :
doing the test as in comment 22, after rebooting at the end of the install process, booting in safe rescue mode ,writing the correct value in /etc/crypttab, rebooting solve the problem, the system run correctly

@Colin: the name crypt_vg_mga_root came from 'dmsetup table' output after having called cryptsetup with the LV name

OK, so I'm getting somewhere:

The dmsetup table output clearly shows the difference in names between the installer and the booted machine.

On the rebooted machine, I have a /etc/crypttab but it only shows *one* of the *two* encrypted partitions. Correcting the crypttab to add in the definition for crypt_vg_mga_root and the appropriate uuid, regenerating initrd and refixing the menu.lst and fstab and rebooting... and it fails, but that's because dracut was generated when it still knew about the luks-* name. A quick "rm /lib/dracut/hooks/initqueue/finished/*luks*" and the boot continued and regenerating dracut a second time from that boot allows a smooth boot for next time. yay.

So the problem here is that the installer misbehaved in some way. It either didn't write a crypttab entry for the "LVM + Encryption" partition at all, or overwrote the crypttab with the "Encryption + LVM" entry (i.e. overwrite, not append).

Still not sure it's the same issue of course :)

Actually, I think this is beginning to make sense and explains a number of things.

Reporter: Can you confirm that your LVM is NOT encrypted but the home LV within it IS encrypted?

I think the read_crypttab() and save_crypttab() functions in dmcrypt.pm are perhaps only looping over physical disks and therefore not seeing the encryption *on top* of the LVM.

If this is the case, this likely explains why the crypttab is not written and perhaps also why it doesn't ultimately show up in the GUI during the install. It might also explain this bug - tho' still not sure on that front.

Summary: Can not unlock /home on my crypted LVM to use it (need to write /etc/crypttab) => Can not unlock encrypted /home on my LVM to use it (need to write /etc/crypttab)

I am not the reporter, but in my case, yes, the LV within the the VG is encrypted.

Confirmed that after attempting a reinstall it fails to show up (in the same way as my initial install). Unlike my initial install where I specified the mountpoint, there is no way to do that with a reinstall until it shows up in the GUI and thus I cannot continue (as my part is / :D). Even Reloading the Partition table doesn't help here.

I've confirmed that "dmsetup table" and blkid can see everything, so it is presumably a problem in our code "seeing" it rather than it not actually being available.

The fact that the UI suggests it's " (inactive)" means that "$part->{dm_name}" must evaluate to false (i.e. be empty).

Tracing futher, in my ddebug log I see the line:

"* could not find the device 252:1 for mapper/crypt_vg_mga_root"

This is coming from the "sub _get_existing_one" in dmcrypt.pm:

    if (my $raw_part = find { fs::get::is_same_hd($active_dmcrypt, $_) } @$fstab) {
	$part->{rootDevice} = $raw_part->{device};
	$raw_part->{dm_name} = $active_dmcrypt->{name};
	$raw_part->{dm_active} = 1;
    } else {
	log::l("could not find the device $active_dmcrypt->{major}:$active_dmcrypt->{minor} for $part->{device}");
    }


So the fs:get::is_same_hd bit is busted when loading a crypt on top of LVM.

From what I can tell, the dmsetup table output has the correct major+minor of the containing drive. It *should* be parsed corrected and the is_same_hd() function *should* match.

So either, a) The original $part's major/minor are not read properly, OR the $active_part's major/minor are not parsed correct from the dmsetup table. Both look as if they *should* be fine.

Interestingly the GUI sometimes seems to the think the partition is Btrfs even tho' it's a crypto_LUKS container... Not sure if that's just fallout from it not doing the whole DM stuff properly tho' (I suspect because dm_active is not set on the part due to the above bug).

That's as much debugging as I can do just now. If I get time, I'll add a load of extra debugging into the process to see where it is tripping up. I'm crap a perl tho', so if someone else can spot the problem, please shout. I know I'm close!!

Created attachment 3757 [details]
does that patch helps?

(In reply to Thierry Vignaud from comment #40)
> Created attachment 3757 [details]
> does that patch helps?

With that patch applied in Mageia 2, diskdrake fails to start with
Not a HASH reference at /usr/lib/libDrakX/fs/type.pm line 320.

(In reply to Thierry Vignaud from comment #40)
> Created attachment 3757 [details]
> does that patch helps?

No I don't think so as on this system there is no /etc/crypttab (it's my / that's encrypted) and there is none in /etc/ or in /tmp/stage2/etc of the installer.

The code flow comes from the "Use" button binding which doesn't ever call read_crypttab() directly but you can follow the code flow to where the error in my logs is printed. I'll do some digging later tonight and find why that bit is failing. My test procedure is pretty simple, just need to hack up the stage2 :) Will report back later.

I meant to alter write_crypttab(), not read_crypttab() :-(

Created attachment 3758 [details]
real suggestion

Attachment 3757 is obsolete: 0 => 1

I'm still not sure in this case. It might be needed but all my problems are before we even get to the install stage and therefore there is no writing of crypttabs yet. I'll start my digging now :)

Problem narrowed down to the fact that in interactive.pm, it calls dmcrypt_open() which in turn calls dmcrypt::open_part() all where the $part variable has no major or minor values set. This has the knock on effect of causing is_same_hd to fail. Still digging.

(In reply to Colin Guthrie from comment #46)
> Problem narrowed down to the fact that in interactive.pm, it calls
> dmcrypt_open() which in turn calls dmcrypt::open_part() all where the $part
> variable has no major or minor values set. This has the knock on effect of
> causing is_same_hd to fail. Still digging.

Thank you for all this digging Colin.  It is really nice to know that we are making some progress.

(In reply to Thierry Vignaud from comment #44)
> Created attachment 3758 [details]
> real suggestion

Makes no difference.

After the encrypted filesystem in a vg is created and
formatted, the area where partitions within the vg goes
blank, so the partition cannot be selected, to assign a
mount point.

As per Comment 21, this was working in Mageia 1, but was
likely broken by the changes made to get lvm on an
encrypted partition working properly.

OK, so I got to the bottom of the lack of major/minors in the raw partition. A very simple bug in the end... seems to be a mixup with $::prefix and paths in /dev/... AFAICT we never want to use $::prefix/dev/... paths?

Sadly this didn't fully fix the issue (although is needed).

I eventually tracked it down but haven't got a fix yet. The problem is that interactive::dmcrypt_open() will ultimately write several keys into the passed in $part dict (namely dm_active and dm_name, but also a rootDevice into the encrypted part added in this process). This is all done correctly (now the above above mentioned bug is fixed), but the code at the end of the function to reread everything to detect LVM on top of encrypted devices triggers some kind of full re-read and the dict keys carefully written previously are lost. Commenting out this code at the end the function allows things to work perfectly (although obviously at the expense of no longer detecting LVMs on top of encrypted partitions.

So, we need to work out how to read the current state from nothing but what we see....

Ahh thanks for pointing out comment 21 Dave - you're bang on on that front (wish I'd seen it earlier as I spend a while poking before I realised that's where the problem lay and why), but I think the problem is wider than just conditionalising that call. Even when commenting that out (with my fix so it's automatically activated properly), doing a reread of the partition table nukes things again even although all the devices are up and running. This is because the code that reads things from scratch never sets the dm_active=1 key nor the dm_name key. If it did, then calling the lvm detection code would be quite safe (even if unnecessary)

Overall, the code here is showing it's age. It uses a myriad of tools to probe things (dmsetup tables, blkid, stat (to extract majors/minors) and others) and various other things that don't lend themselves to the dynamic nature of things (there are lots of calls to "udevadm settle"). Really the code should be ripped apart and replaced with full udev support where we get informed when devices appear and can get all the needed metadata directly via udev without any manual probing. That, however, is a *lot* of work and certainly nothing to undertake just now :(

Created attachment 3759 [details]
Use correct path to /dev (don't look in the $::prefix before it's even mounted!)

OK, so as per comment 21, I think rather than the line

$all_hds->{lvms} = [ fsedit::lvms($all_hds) ];

could be replace by something like (not real perl code as I don't know perl enough):

$newlvms = [ fsedit::lvms($all_hds) ];
foreach ($all_hds->{lvms} as $lvm) {
  if (my $newlvmdev = find($lvm->{device} in $newlvms) {
    if ($lvm->{dm_active}) $newlvmdev->{dm_active} = $lvm->{dm_active};
    if ($lvm->{dm_name}) $newlvmdev->{dm_name} = $lvm->{dm_name};
  }
}
$all_hds->{lvms} = $newlvms;


i.e. ensure that the dm_active and dm_name keys are preserved when rescanning for lvm.

If I've understood things properly that should fix the immediate problem (still doesn't fix the issue of the "reread partition table nuking that data, but it's maybe enough to fix this problem for now)

(In reply to Colin Guthrie from comment #50)
> via udev without any manual probing. That, however, is a *lot* of work and
> certainly nothing to undertake just now :(

When it is done, the luks encrypted "space" should be treated as a block
device, appearing on a tab like lvm volume groups, but the options to
create a partition table, or a file system using the entire device.  As
an example, I just encrypted sda8, opened the encrypted block device, used
 cfdisk /dev/mapper/crypt_sda8 to create a partition table, and two
partitions, one linux, one lvm pv.  Then ran mkfs.ext4 on the first, and pvcreate on the second

# blkid |grep sda8
/dev/sda8: UUID="4fa398fb-08d6-412c-8dbc-faf5fe37f591" TYPE="crypto_LUKS"
/dev/mapper/crypt_sda8p1: UUID="95a4a906-cdfd-44d5-96bb-1272970c3700" TYPE="ext4" LABEL="cryptedpart1"
/dev/mapper/crypt_sda8p2: UUID="bVQ8PF-ILZx-WOfS-nAiF-H8ms-XTfD-X7kwsh" TYPE="LVM2_member"

I could then create encrypted block devices within the lvm pv.  I don't
know what the limit is on nesting block devices, but it can be done, and
should be handled by diskdrake.

It made sense to treat encryption as a file system attribute when loop
encryption was used, but not with luks, since it is a block device that
it creates when opened.  That block device may only contain one filesystem,
or it can be partitioned.

(In reply to Dave Hodgins from comment #48)
> (In reply to Thierry Vignaud from comment #44)
> > Created attachment 3758 [details]
> > real suggestion
> 
> Makes no difference.
> 
> After the encrypted filesystem in a vg is created and
> formatted, the area where partitions within the vg goes
> blank, so the partition cannot be selected, to assign a
> mount point.

Thinking a bit about this after some sleep (I was up waaaaay to late hacking on this last night - dead today!) I think this fix is needed too, but it's effect won't be seen until the first reboot.

Basically I suspect that without this fix, the crypttab will not contain the necessary line to name the mapper correctly and it'll get the default luks-* name instead. This will not correspond to the root= kernel command line and dracut will barf. So all in all there are now three different fixes needed here I think - one of them still only implemented in my :)

As for comment 53 I think we should probably leave that for later! (I probably shouldn't have brought it up really, but I was cranky at poking round the circular code paths in a language I don't really grok!!) After mga3 is out we can discuss what we want the installer to support ongoing (just because something is theoretically possible, doesn't mean we should encourage or support it!). I'll create some "feature" pages to cover this (as this seems like the best way to keep track.

(In reply to Colin Guthrie from comment #52)
You means sg like this?

my @newlvms = fsedit::lvms($all_hds);
my %newlvms = map { $_->{device} => $_ } @newlvms;
foreach my $lvm (@{$all_hds->{lvms}}) {
  if (my $newlvm = $newlvms{$lvm->{device}}) { # faster
    $newlvm->{dm_active} = $lvm->{dm_active};
    $newlvm->{dm_name} = $lvm->{dm_name};
  }
}
$all_hds->{lvms} = \@newlvms;

(In reply to Colin Guthrie from comment #49)
As for droping $::prefix for /dev entries, just go on.
What's more we now bind /dev on /mnt/dev anyway...

(In reply to Thierry Vignaud from comment #55)
> (In reply to Colin Guthrie from comment #52)
> You means sg like this?
> 
> my @newlvms = fsedit::lvms($all_hds);
> my %newlvms = map { $_->{device} => $_ } @newlvms;
> foreach my $lvm (@{$all_hds->{lvms}}) {
>   if (my $newlvm = $newlvms{$lvm->{device}}) { # faster
>     $newlvm->{dm_active} = $lvm->{dm_active};
>     $newlvm->{dm_name} = $lvm->{dm_name};
>   }
> }
> $all_hds->{lvms} = \@newlvms;

Damn, it works as intended, but sadly not what I had in mind.. it lvms array is just the VGs, not the LVs and it's the LVs we need to copy this data over on.

I'll try and narrow down to where the data is read and work on a fix... can only do another hour tonight tho' - too tired.

> (In reply to Colin Guthrie from comment #49)
> As for droping $::prefix for /dev entries, just go on.
> What's more we now bind /dev on /mnt/dev anyway...

Done!

OK, so I've now got some data which I hope is sufficient to let some perl expert (i.e. Thierry) craft the right code.

I'll attach three files, one a Data::Dumper output of $allhds->{lvms} before the call and one after. The differences are clear so it should be semi-obvious for a proficient perl coder to work out how to preserve the data...

Of course it *might* be sensible to try and work out how to *read* the data in the first place rather than preserve it, but as should be obvious from the output some fields cannot be read (e.g. dmcrypt_key (which is user input)).

Hopefully this is sufficient info to be able to fix things Thierry. I can test any thing you do. Note the at there are a few places where this is done, so probably worth putting it into a lvm::reread() sub or something.

Created attachment 3761 [details]
$allhds->{lvms} before a reread.

Created attachment 3762 [details]
$allhds->{lvms} after a reread.

Created attachment 3763 [details]
$allhds->{lvms} diff from before+after reread.

Created attachment 3765 [details]
add read_crypttab_() & save_crypttab_() that enables to use an alternative crypttab location

I think it's simpler to just add read_crypttab_() & save_crypttab_() that enables to use an alternative crypttab location

Just add calls to them where appropriate & test

Created attachment 3766 [details]
eg for reloading partition tables

Created attachment 3767 [details]
eg: for detecting lvms on top of dmcrypt

Obviously, even it's OK to use such temp file in drakx, it's not in diskdrake for sensitive info, so if that work, I'll replace in both patches:

    my $tmp_file = '/tmp/crypttab';

by:

    my (undef, $tmp_file) = mkstemp('/tmp/crypttabXXXXX');

BTW if those 3 patches works, please retest with mkstemp :-)

(In reply to Thierry Vignaud from comment #61)
> Created attachment 3765 [details]
> add read_crypttab_() & save_crypttab_() that enables to use an alternative
> crypttab location
> 
> I think it's simpler to just add read_crypttab_() & save_crypttab_() that
> enables to use an alternative crypttab location
> 
> Just add calls to them where appropriate & test

Looks good, but there is a recursive call in read_crypttab() due to a missing _ on the called function name. Other than that ACK. Please commit (my tests are not yet showing it working but I think the approach is valid!)

(In reply to Thierry Vignaud from comment #63)
> Created attachment 3767 [details]
> eg: for detecting lvms on top of dmcrypt

Small typo "my ($all_hds) = @_," should be "my ($all_hds) = @_;"


I've tested these calls now. I tested the mkstemp approach first, but it failed as it was not defined. Probably missing a require or a more explicit call name (file::temp::mkstemp()?), but I didn't investigate and just reverted to the static name.

Sadly these changes are not quite enough.

It did improve the situation however. Compared to previously, the dm_name is now filled in correctly, but dm_active and faked_device are still missing. I suspect that your changes are valid but we need to do a bit more to ensure we check for active devices when parsing/interpreting the crypttab generally as there is only one place where dm_active is set just now.

So with the above caveats, I say, commit these changes and we'll move on and work out how to set dm_active (and faked_device too if it's really needed).

Untested (I'm at work now so can't test), but I suspect this patch to the end of read_crypttab_() will do the trick:


        $raw_part->{dm_name} = $dm_name;
+	my $active_dmcrypt = _parse_dmsetup_table($dm_name, run_program::get_stdout('dmsetup', 'table', $dm_name));
+       _get_existing_one([$raw_part], $active_dmcrypt);

Thanks Thierry, those commits seem to have allowed me to set everything up. Sadly going to the next stage failed saying one of my partitions was unknown... will investigate that issue separately.

FWIW, there is a UI issue still when unlocking the partition on top of an LV (not unlocking a partition that contains an LVM). Switching tabs and back again sorts the UI out, so it's not a critical issue, but would still be nice to fix. Due to the new values inside $allhds->lvms I suspect the UI loses it's current reference and doesn't repaint itself, so we probably need a way to tell it to refind the same $part->{device} in the new data if it loses its reference.

OK, good news! Turns out the problem I encountered is that when you "unlock" an LVM by using an encrypted partition, it does not get activated. A simple call to "lvm2 vgchange -ay" in the term allowed the install/upgrade to complete. I deleted the current /etc/crypttab and at the end of the upgrade it was correctly written :)

So I think the crux of this bug is now confirmed fixed. Just a small extra issue to fix to actually activate the LVs before trying to use them!

FWIW, I've now fixed both the LVM activation and the UI display issue after unlock. I think this bug can be closed, but I'd rather the original reporters do this. My final two fixes will not be available until 15.42

Thank you for all the work you have put into this! :)

Some old bugs i reported may have had same cause you fixed here.
Unfortunately i do not have any encrypted part in an LVM, 
but encrypt the whole LVM nowadays. I might try next install :)

I am happy to test this as well.  Is it in Mageia-3-RC-x86_64-DVD yet?

Glen

I just tested diskdrake from drakxtools-curses-15.42-1.mga3 and
I'm not seeing any changes.

(In reply to Glen Ogilvie from comment #72)
> I am happy to test this as well.  Is it in Mageia-3-RC-x86_64-DVD yet?
> 
> Glen

Just wondering if this is in the RC DVD iso yet for testing, or if I should test from cauldron boot.iso?

Ping! drakxtools-curses-15.42-1.mga3 either doesn't include the fixes, or
they are not working.

With diskdrake from drakxtools-curses-15.44-1.mga3,
the display of logical volumes within the volume
group is still going blank, after entering the
passphrase for an encrypted logical volume.

I only worked on the GUI, I've not tested ncurses display.

Oops, hit send before reply.

I'll try and take a look, but I'm not 100% sure I'll have time.

The fact that the scenario you describe doesn't work is likely also a bug in mga2? Certainly this scenario worked fine in mga2's GUI - i.e. a new tab shows up automatically.

Just to be clear, in comment 76 I was testing with Mageia 3 KDE x86-64,
fully updated.  I only mentioned curses as that's the result of
# rpm -q -f /usr/sbin/diskdrake
drakxtools-curses-15.44-1.mga3

I am using the gui version of diskdrake.

Created attachment 3811 [details]
Snapshot of diskdrake after selecting encrypted file system within a lvm vg

As shown in the attachment, I have a volume group, that I've
called vga.  Within that volume group, I have a logical volume
called vgaroot, which contains a luks encrypted ext4 filesystem.
(Incorrectly shown by diskdrake as Btrfs).

This is just before I click on the Use button, and enter the
passphrase.

Created attachment 3812 [details]
Snapshot of diskdrake after entering passphrase to open an encrypted file system within a lvm vg

As shown in this screenshot, once I've selected the use button,
and entered the passphrase, the luksOpen does run, but the display
no longer shows anything, where it should be showing the logical
volume.  No way to select the now open encrypted filesystem, to
assign a mountpoint.

This is exactly the same behaviour as currently happens in Mageia 2.

This is with a fully updated Mageia 3 kde x86_64 install.

Hmmm, this is specifically a case I fixed... :s I presume it draws OK if you flip the tabs over and back again?

I'll try again with the current stage2 on the mirrors.

Yes it does.  Clicking on the tab for sda, then on the tab for vga,
does now show the encrypted ext4 filesystem is open, and I can select
it for assigning a mount point etc.  So it is an improvement, in that
it can now be mounted, but shouldn't require switching tabs to reload
the list of logical volumes.

(In reply to Dave Hodgins from comment #83)
> Yes it does.  Clicking on the tab for sda, then on the tab for vga,
> does now show the encrypted ext4 filesystem is open, and I can select
> it for assigning a mount point etc.  So it is an improvement, in that
> it can now be mounted, but shouldn't require switching tabs to reload
> the list of logical volumes.

I'm not sure that's an improvement really. It was like that in mga2 in my tests.

It's this that I specifically fixed by effectively redrawing the tabs.

http://svnweb.mageia.org/soft/drakx/trunk/perl-install/diskdrake/hd_gtk.pm?r1=7964&r2=7963&pathrev=7964

It was done on April 21, which should have been in 15.42 although other bugs may have been to blame so 15.44 is the version to test... So I guess there are still cases where this fix does not work :(

(In reply to Colin Guthrie from comment #84)
> (In reply to Dave Hodgins from comment #83)
> > Yes it does.  Clicking on the tab for sda, then on the tab for vga,
> > does now show the encrypted ext4 filesystem is open, and I can select
> > it for assigning a mount point etc.  So it is an improvement, in that
> > it can now be mounted, but shouldn't require switching tabs to reload
> > the list of logical volumes.
> 
> I'm not sure that's an improvement really. It was like that in mga2 in my
> tests.
> 
> It's this that I specifically fixed by effectively redrawing the tabs.
> 
> http://svnweb.mageia.org/soft/drakx/trunk/perl-install/diskdrake/hd_gtk.
> pm?r1=7964&r2=7963&pathrev=7964
> 
> It was done on April 21, which should have been in 15.42 although other bugs
> may have been to blame so 15.44 is the version to test... So I guess there
> are still cases where this fix does not work :(

Just tested this against cauldron, (boot.iso).  Version: 15.45

For a system with an existing encrypted LVM partition, clicking on it, then clicking use, entering the passphrase, tab's do not redraw.  Clicking between tabs then shows it opened OK, to mapper/crypt_vg_mga_root

I just tested it again a few days ago too, and it worked fine for me :(

Something strange is going on. Perhaps you can dig into the code? easiest way is to set up urpmi-proxy then hack on the stage2 squashfs file (mount it, extract all the files to a tree, hack on it, then resquash it again (mksquashfs my-hacked-tree/ mdkinst2.sqfs -all-root -noappend)  then go through the install proceedure again. I don't think the boot.iso matters here, as it should be all related to stage2.

Perhaps your hardware is reacting more slowly than on my VM? Perhaps just a simple sleep injected into the redraw code would make it work? (that's not a fix, just a test to see if it helps!)

(In reply to Colin Guthrie from comment #86)
> I just tested it again a few days ago too, and it worked fine for me :(
> 
> Something strange is going on. Perhaps you can dig into the code? easiest
> way is to set up urpmi-proxy then hack on the stage2 squashfs file (mount
> it, extract all the files to a tree, hack on it, then resquash it again
> (mksquashfs my-hacked-tree/ mdkinst2.sqfs -all-root -noappend)  then go
> through the install proceedure again. I don't think the boot.iso matters
> here, as it should be all related to stage2.
> 
> Perhaps your hardware is reacting more slowly than on my VM? Perhaps just a
> simple sleep injected into the redraw code would make it work? (that's not a
> fix, just a test to see if it helps!)

I'll be happy to have a dig, although may take me a few days. My hardware is actually pretty quick, (i7, 32GB ram, SSD). 

I can also provide the VM image to you if you want (it's just a test vm with test passwords). It has two virtual disks, and LVM, so this is the 3rd tab which does not refresh. In this specific case, / is on LVM, and it uses grub2 to read it, as it does not have a separate /boot.

The VM I have is a KVM based VM, about 4GB.

Hmm, your rig is definitely faster than mine :D

Can you just describe your layout fully then and I'll try and recreate. / on LVM + grub2 but which is the encrypted partition? (or is grub2 really that insane that it can boot from an encrypted / on top of LVM? I really hope not!!)

(In reply to Colin Guthrie from comment #86)
> set up urpmi-proxy then hack on the stage2 squashfs file

Beware of bug 5797 (urpmi-proxy not updating stage2 using default config)
Workaround in https://bugs.mageia.org/show_bug.cgi?id=5797#c16

( which reminds me i should verify that fix myself! )

In this case we're taking advantage of that "bug"... :) We *want* the local (hacked) version, not the remote one :)

OK, i see. Just make sure you get the right version to hack ;)

Yes, being able to serve a local copy of stage2 is really handy.. If we "fix" that bug, maybe some sort of local mode for urpmi proxy would still be helpful for testing stage2.



filesystem layout on one of the test systems is a little unusual, but that's the point, since I am testing the unusual:

One test I have been testing with, is:
2 x 4GB disks, each with 1 partition on them, (/dev/vda1 and /dev/vda2), both of type: 8e (lvm).

Both partitions are added to a volume group.
in the volume group, I have the following LVs.

boot (368MB), xfs filesystem
ntfs (688MB), ntfs filesystem
opt (160MB), reiserFS
root (5.65GB), encrypted XFS.
swap  (492m)


In the installer, when I first open up the VG and click on the encrypted partition, it shows an encrypted, and the file system type as brtfs (which it is not)..  When I click the use button, it asks me for the passphrase.  After entering it, the tab is not refreshed and goes blank, however, if I switch to a disk tab, then switch back to the volume group tab, it shows up correctly, and I can click on the encrypted xfs partition and click view, to view the files.

It is configured with grub2.

The above test system won't actually boot up, because it didn't cope with the encrypted root file system, so it ends up in drakcut, complaining about not being able to find /dev/mapper/crypt_vg_mga_root, however, that's not what this bug is about.

I spent some time looking into this but couldn't make much progress due to my lack of perl coding skills and getting lost in all the callbacks etc.

I'm pretty sure I was doing the redraws at the right time, but it didn't seem to work. I dunno.

Either way, what remains now is a UI issue rather than a fatal one. I suggest we open a new bug for this issue and close this one. If we have to release with the installer like this I think it's generally OK (not ideal but unless someone can come up with some quick fix here I think we'll have to accept that :))

I've opened bug #9993. Please feel free to adjust it as you see fit. Closing this one now.

Status: NEW => RESOLVED
Resolution: (none) => FIXED