Bug 27765 - recovery tools to be included in the ISO
Summary: recovery tools to be included in the ISO
Status: RESOLVED FIXED
Alias: None
Product: Mageia
Classification: Unclassified
Component: Release (media or process) (show other bugs)
Version: Cauldron
Hardware: All Linux
Priority: release_blocker enhancement
Target Milestone: Mageia 8
Assignee: ISO building group
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-12-06 16:56 CET by Giuseppe Ghibò
Modified: 2021-01-19 15:41 CET (History)
6 users (show)

See Also:
Source RPM:
CVE:
Status comment:


Attachments

Description Giuseppe Ghibò 2020-12-06 16:56:03 CET
It would be useful to include these backup/recovery admin tools to the Live ISO. They are pretty small in size (smaller than 1MB or a few MBs):

- clonezilla. helpful for cloning and restoring. There are pretty standalone clonezilla images, but it could be handy to have in Live, also for having the same hardware detection as mga.

- ext4magic, useful for recovering files which were discarded by mistake.

- extundelete, same as above

- photorec
Comment 1 Giuseppe Ghibò 2020-12-06 23:38:26 CET
also:

+ partclone (+1.9MB)

+ fdupes (+0.04MB)
Aurelien Oudelet 2020-12-08 16:26:59 CET

Assignee: bugsquad => isobuild
Target Milestone: --- => Mageia 8
CC: (none) => mageia

Comment 2 Morgan Leijström 2020-12-08 18:35:14 CET
+ testdisk - recover deleted partitions, FAT tables, NTFS boot, undeletes FAT, NTFS files... "companion to photorec" which is requested in comment 0, which finds files by scanning. https://www.cgsecurity.org/wiki/TestDisk

+ fsarchiver - archives/restores contents of filesystems, compress and encrypt. https://www.fsarchiver.org/

( I see  gparted is already aboard :)

I have not checked what is aboard, but maybe some libs/tools for more actions on more file systems?

CC: (none) => fri

Comment 3 Giuseppe Ghibò 2020-12-08 19:31:23 CET
The list of packages and the corresponding occupied size is in file "Mageia-8-beta2-Live-Plasma-x86_64.lst.full". Actually from a quick count there are 0.9GB (installed) of multilanguage gimp-help files, and 1.16GB (installed) of multilanguage help files, and 224MB (installed) multilanguage fonts.

testdisk (0.9MB + 20MB) is not there

fsarchiver (0.27MB) is not there

I'd say also:

+filezilla (16MB)
+partimage (0.8MB + 0.1MB + 0.02MB)
+partclone (2.3MB)
+fio (2MB)
+smartmontools (1.7MB), but not enabled in systemd
+inxi (2MB)
+baobab (1.4MB)

For filesystems there is already abroad:

91197 exfatprogs
175430 lib64reiserfsprogs0
370872 reiserfsprogs
4598459 btrfs-progs
5054664 e2fsprogs
5071336 xfsprogs
Comment 4 Thomas Backlund 2020-12-08 22:32:30 CET
(In reply to Giuseppe Ghibò from comment #3)
> The list of packages and the corresponding occupied size is in file
> "Mageia-8-beta2-Live-Plasma-x86_64.lst.full". Actually from a quick count
> there are 0.9GB (installed) of multilanguage gimp-help files, and 1.16GB
> (installed) of multilanguage help files, and 224MB (installed) multilanguage
> fonts.
> 

You do realize that the isos are multilang, so obviously you will find a lot of multilang stuff...
Comment 5 Morgan Leijström 2020-12-08 22:46:08 CET
One idea may be to make a "service iso" with english only, and no Gimp or other DTP aimed big programs.  Just boot to a GUI and have system tools, plus internet browser, pdf reader etc to facilitate searching for knowledge.  For occasional soho/home admins.

Well yes a case for https://wiki.mageia.org/en/Draklive2

Maybe an unofficial community effort?

One may think it is just like https://www.system-rescue.org/ - which is good - but they have stopped making 32 bit (IIRC) and with Live persistence Mageia is much easier to quickly add tools to, and save files, notes... to.
Comment 6 Giuseppe Ghibò 2020-12-08 23:03:03 CET
Yes. Well, another thing that could be interesting to have in live is the calibre, but that's extra 620MB of space for all extra libraries.

Indeed what you suggested can be made in another way, i.e. provide an extra superISO with a size bigger than 4.3GB, other than the current standard ISOs. Today the Live ISO is used mainly in a pendrive (though it might be used from the DVD-RW reader), for which 8 or 16GB is a pretty common size. Furthermore, blue-ray disks with 25GB size are also common too, so that a 7-8GB iso can be burnt there pretty easy. For smaller sizes/devices there are the current standard ISOs.
Comment 7 Martin Whitaker 2020-12-09 00:14:36 CET
These requests for different flavours of Live ISO are why I spent a lot of time making draklive2 easier to use than draklive and documenting it. Using the GUI mode, anybody who has used the classical installer should have no trouble building their own Live ISO that includes exactly what they want.

The QA team is stretched to the limit as it is. We can't add more official ISOs.
Comment 8 Martin Whitaker 2020-12-14 23:03:13 CET
Why are you asking for multiple tools to do the same job:

 - clonezilla/partclone/partimage
 - ext4magic/extundelete

Please pick the best one.

I don't see a good argument for filezilla - you can easily download files from FTP and HTTP servers using Firefox.

I don't see fio as being useful for system recovery. You can run that on your installed system.

baobab is a GNOME tool and is already on the GNOME Live ISO.
Comment 9 Giuseppe Ghibò 2020-12-15 00:04:28 CET
There is no "best" tool or preferred one. According to circumstances sometimes one performs better than the other. E.g. in case of ext4magic/extundelete, sometimes you get better recovery with ext4magic, sometimes with extundelete. That's why I included both.

As for filezilla, it supports several protocol that firefox doesn't, such as scp, sftp.

For clonezilla/partclone/partimage, it's not said that you backup and restore with the same tool originated from the live. Sometime you need to just restore or just backup.

For baobab the plasma equivalent it's k4dirstat (btw, I just found k4dirstat segfaults as soon as it starts, so needs its own bug report).
Comment 10 Giuseppe Ghibò 2020-12-15 00:07:59 CET
For fio sometimes you need to test performance (e.g. degreded or not as supposed to be) before recovering.
Comment 11 Morgan Leijström 2020-12-15 00:12:05 CET
Similar tools with slightly different features as well as bugs: together they cover more cases, and user files may be very valuable to save.

It is also about what operator is used to and best can handle to not miss some recovery opportunity, or make things worse.
Comment 12 Thomas Backlund 2020-12-15 00:30:43 CET
(In reply to Morgan Leijström from comment #11)
> Similar tools with slightly different features as well as bugs: together
> they cover more cases, and user files may be very valuable to save.
> 
> It is also about what operator is used to and best can handle to not miss
> some recovery opportunity, or make things worse.

And that's exactly why this idea dont scale...

Everyone wants different tools, ahd everyone thinks their favorite tool is the best one... so basically we should put the whole repo on the image and be done with it .... NOT ...

Point is they can boot up with live image, add online medias and install their preferred tool and go from there...

or precreate the the stuff they need with the persistenced stuff...

yes, some of the recovery tools do even belong on recovery image, but the same goes there... we need to think twice before blindly adding stuff... not to mention duplicates...

We are a linux distro, not a recovery tool image builder
Comment 13 Thomas Backlund 2020-12-15 00:32:19 CET
(In reply to Giuseppe Ghibò from comment #9)


> As for filezilla, it supports several protocol that firefox doesn't, such as
> scp, sftp.
> 


so does ssh. check.
Comment 14 Thomas Backlund 2020-12-15 00:34:58 CET
oh, and rescue images have dropbear ssh server...
Comment 15 Giuseppe Ghibò 2020-12-15 08:31:53 CET
(In reply to Thomas Backlund from comment #13)
> (In reply to Giuseppe Ghibò from comment #9)
> 
> 
> > As for filezilla, it supports several protocol that firefox doesn't, such as
> > scp, sftp.
> > 
> 
> 
> so does ssh. check.

It has also ftp over tls (which is not sftp).
Comment 16 Giuseppe Ghibò 2020-12-16 14:24:02 CET
(In reply to Giuseppe Ghibò from comment #9)

> For baobab the plasma equivalent it's k4dirstat (btw, I just found k4dirstat
> segfaults as soon as it starts, so needs its own bug report).

k4dirstat has just been fixed in release 3.2.1. So it could be included in the plasma live.
Comment 17 Giuseppe Ghibò 2021-01-06 13:47:06 CET
Little update: still valid in mageia8-rc isos. According to the *.lst.* included files, none of the packages listed above were added, except 'inxi'. E.g. checking with:

grep -E \
        'ext4magic|extundelete|inxi|photorec|partclone|testdisk|fsarchiver|filezilla|partimage|partclone|fio|smartmontools|k4d
irstat' \
        *.lst.*
David Walser 2021-01-06 23:43:18 CET

Priority: Normal => release_blocker
CC: (none) => luigiwalser

Comment 18 David Walser 2021-01-06 23:46:58 CET
(In reply to Martin Whitaker from comment #8)
> Why are you asking for multiple tools to do the same job:
> 
>  - clonezilla/partclone/partimage
>  - ext4magic/extundelete

They do completely different jobs.  The clonezilla stuff is for disk/partition cloning, which is typically done from Live DVDs (usually an Ubuntu-based one, would be nice to have a Mageia one easily available).  The ext stuff is rescue stuff for digging into a filesystem while it's not in use (hence wanting it on a Live).

Let's not let this bug get out of hand with extra requests.  Those tools are enough (and are small).  We don't need any additional GUI tools.
Comment 19 Martin Whitaker 2021-01-07 13:17:18 CET
(In reply to David Walser from comment #18)
> Let's not let this bug get out of hand with extra requests.  Those tools are
> enough (and are small).  We don't need any additional GUI tools.

It has got out of hand, which is why I haven't done anything about it. See Thomas's remarks in comment 12.

It's an enhancement request, so how can it be a release blocker?
Comment 20 Giuseppe Ghibò 2021-01-07 13:30:16 CET
Here is the summary (without inxi, which is already been added):

-ext4magic
-extundelete
-photorec
-partclone
-testdisk
-fsarchiver
-filezilla
-partimage
-partclone
-fio
-smartmontools
-k4dirstat (only for plasma, equivalent of gnome's baobab)

there is still plenty of room in the ISO, that I don't see particular space problem for these tiny apps.
Comment 21 Nicolas Lécureuil 2021-01-07 13:47:59 CET
(In reply to Martin Whitaker from comment #19)
> (In reply to David Walser from comment #18)
> > Let's not let this bug get out of hand with extra requests.  Those tools are
> > enough (and are small).  We don't need any additional GUI tools.
> 
> It has got out of hand, which is why I haven't done anything about it. See
> Thomas's remarks in comment 12.
> 
> It's an enhancement request, so how can it be a release blocker?

for me it can't, you are right.

CC: (none) => mageia

Comment 22 David Walser 2021-01-07 16:14:05 CET
Obviously it has to be a release blocker.  Once we've released, the ISOs are done.

filezilla and k4dirstat are not needed.  Some of the other tools really aren't that useful even having packaged in the distribution unless they're included on a Live ISO.
Comment 23 Giuseppe Ghibò 2021-01-07 17:52:01 CET
I was speaking about the Live ISOs. Regarding k4dirstat, I don't see why it shouldn't be included. We have already baobab in the GNOME Live ISO, but not in Plasma Live ISO, so I don't see why why shouldn't we have k4dirstat in the Plasma live, which is it's dual equivalent in the Plasma world of baobab. As for filezilla, I've already motivated, it would be fine to have a GUI tool supporting several protocols (including FTP over TLS which is not SCP), useful for downloading for instance restore images locally. sshfs-fuse would be also a good complement (peanuts in term of package disk usage).
Comment 24 Morgan Leijström 2021-01-07 18:07:28 CET
Do we have some size number in MB for the whole list?

---

For mga9 we could (in advance time...) discuss more what to include and not.

 - Especially now when it is so easy to create persistent Live to install more, I think some things can be excluded.  I.e not support *all* languages in full.  Maybe not Gimp - most users do not understand it yet it is default for viewing photos on Xfce (which i have even pondered to issue a bug for - it should be an efficient leighter weight app more for casual use to be default... regardless of this ISO size discussion). Etc...
Comment 25 Giuseppe Ghibò 2021-01-07 18:16:43 CET
Well, in the future we might also evaluate exceeding the size of 4.3GB for some ISOs or Live ISOs.  Considering that most of installing media would be a USB pendrive (whose size is usually 8 or 16GB or more). Even on Windows and Mac the installing image size have exceeded 4.3GB since a lot of time.
Comment 26 David Walser 2021-01-07 18:28:28 CET
lftp should be able to handle ftps.
Comment 27 Morgan Leijström 2021-01-07 18:33:41 CET
ISO size is also about download time, and the size it eats from persistence space.

Maybe as a wiki page based community effort we could create a configuration for Draklive2 to create a minimal Live base, with option to add lists of tools.
Comment 28 David Walser 2021-01-07 20:39:33 CET
Yes, that's a fine idea.  The tools we're proposing in this bug should be included on the standard Live though, otherwise they'll be practically inaccessible to a lot of people.  We'd like to be able to advertise our Live as a good recovery and cloning tool to people who aren't currently Mageia users and would not have the ability, interest, or inclination to generate their own ISO.
Comment 29 Martin Whitaker 2021-01-14 22:21:18 CET
Latest 8-rc ISOs include

  clonezilla (which requires partclone and partimage)
  fsarchiver
  extundelete
  photorec
  testdisk

which covers the request for simple backup and recovery tools.

It's really not hard to add other tools if you want them. Just start MCC, add the distribution media, and install whatever you need. If you enable persistence, you'll only need to do it once.

Status: NEW => RESOLVED
Resolution: (none) => FIXED

Comment 30 Giuseppe Ghibò 2021-01-15 11:11:47 CET
Thanks. Please, can you add ext4magic too? It can complement extundelete (i.e. when extundelete fails it can still recover something and viceversa). It's just 250 kbytes long.
Comment 31 Martin Whitaker 2021-01-18 22:23:28 CET
So, I've added partimage, and now I find a "partimag" user appears in the DM greeter and in drakuser. This I don't like.
Comment 32 Giuseppe Ghibò 2021-01-18 22:31:28 CET
Probably this is a bug of partimage package as it adds a user  with UID greater than system users which do not appears in a DM.
Comment 33 David Walser 2021-01-19 05:23:56 CET
(In reply to Martin Whitaker from comment #31)
> So, I've added partimage, and now I find a "partimag" user appears in the DM
> greeter and in drakuser. This I don't like.

Yes that is quite odd.  It not only doesn't make it a system user, it gives it /bin/bash as a shell.  Is there a valid reason for that?
Comment 34 Nicolas Lécureuil 2021-01-19 09:41:37 CET
yes in sddm for ex we only show users with ID > 1000 
If partimage add a user we need to be sure the ID is less than 1000
Comment 35 Martin Whitaker 2021-01-19 10:13:58 CET
It's added as a system user and gets an ID < 1000. But both userdrake and lightdm show it - I guess because it has a home directory in /home.

Remember that user IDs used to start at 500, and people who have upgraded from older versions of Mageia may still have such user IDs, so we shouldn't filter out IDs < 1000.
Comment 36 Thomas Backlund 2021-01-19 10:22:29 CET
(In reply to Martin Whitaker from comment #35)


> Remember that user IDs used to start at 500, and people who have upgraded
> from older versions of Mageia may still have such user IDs, so we shouldn't
> filter out IDs < 1000.

iirc the filter < 1000 only apply on installs, on upgraddes the old < 500 is used
Comment 37 David Walser 2021-01-19 14:51:22 CET
OK it does have the -r, but is there a reason it has /bin/bash as a shell???

Or could we just use our standard Mageia useradd macro instead of this:
%pre
/usr/sbin/groupadd -r -f partimag > /dev/null 2>&1 ||:
/usr/sbin/useradd -g partimag -d /home/partimag -r -s /bin/bash partimag > /dev/
null 2>&1 ||:
Comment 38 Aurelien Oudelet 2021-01-19 15:17:29 CET
(In reply to David Walser from comment #37)
> OK it does have the -r, but is there a reason it has /bin/bash as a shell???
> 
> Or could we just use our standard Mageia useradd macro instead of this:
> %pre
> /usr/sbin/groupadd -r -f partimag > /dev/null 2>&1 ||:
> /usr/sbin/useradd -g partimag -d /home/partimag -r -s /bin/bash partimag >
> /dev/
> null 2>&1 ||:

partimage should never have a shell.

CC: (none) => ouaurelien

Comment 39 Giuseppe Ghibò 2021-01-19 15:19:58 CET
It seems it got the useradd|groupadd of the original imported package. Deb uses /bin/false as shell for instance.

We might try to use the mageia's useradd standard macro, which in the end uses the /usr/share/rpm-helper/add-user that furthermore checks if the entry has been already created. Also we might try using the /bin/false as shell. Maybe we might use a home outside /home too, e.g. /var/lib/partimage?
Comment 40 David Walser 2021-01-19 15:24:41 CET
Fix just submitted for partimage.
Comment 41 Giuseppe Ghibò 2021-01-19 15:33:41 CET
Shouldn't have to use also %pre_groupadd for the group? BTW no groupdel|userdel on uninstalling?
Comment 42 David Walser 2021-01-19 15:38:54 CET
No, the macro creates a group too.
Comment 43 Giuseppe Ghibò 2021-01-19 15:41:42 CET
ok

Note You need to log in before you can comment on or make changes to this bug.