Bug 34826 - dorsync fixing, improving, and packing to rpm.
Summary: dorsync fixing, improving, and packing to rpm.
Status: ASSIGNED
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: 9
Hardware: All Linux
Priority: Normal normal
Target Milestone: ---
Assignee: QA Team
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2025-12-11 06:01 CET by Tony Blackwell
Modified: 2026-01-17 02:12 CET (History)
8 users (show)

See Also:
Source RPM: mga-dorsync
CVE:
Status comment: Package in comment 84
marja11: affects_mga9+


Attachments
shellcheck log (18.56 KB, text/plain)
2025-12-13 02:27 CET, Dan Fandrich
Details
patch, v1 (3.36 KB, patch)
2025-12-13 03:30 CET, Frédéric "LpSolit" Buclin
Details | Diff
replace sha1sum by sha3-512sum (3.08 KB, patch)
2025-12-13 18:56 CET, Frédéric "LpSolit" Buclin
Details | Diff
patch, v3 (4.53 KB, patch)
2025-12-14 21:41 CET, Frédéric "LpSolit" Buclin
Details | Diff

Description Tony Blackwell 2025-12-11 06:01:36 CET
Hmmm, I distantly recall this dorsync script issue before.
Last line is fix.

Tony

$ ./dorsync
You need to install udisks to run this program

# urpmi udisks
No package named udisks
# urpmi udisks2
Package udisks2-2.10.1-1.2.mga9.x86_64 is already installed


Fix: in line 46 of dorsync change udisks to udisks2
Comment 1 Marja Van Waes 2025-12-11 13:37:10 CET
Assuming someone from QA team will fix this, thus assigning to QA.

Please assign back if I'm wrong.

Assignee: bugsquad => qa-bugs
CC: (none) => marja11

Marja Van Waes 2025-12-11 13:37:32 CET

Summary: won't run until single char fix => dorsync won't run until single char fix

Comment 2 Thomas Andrews 2025-12-11 17:36:08 CET
Before the edit, dorsync refuses to run, as described in comment 0.

After the edit, it does run, though with one fault yet. 

Dorsync checks for the sha1 checksum, which we no longer use, and reports it as "missing." That should be fixed as well, but it doesn't affect the rest of the script. Checks for md5 and sha512 are still performed, so I believe that covers this bug.

Validating, but I don't know where it is to go from here.

Keywords: (none) => validated_update
CC: (none) => andrewsfarm, sysadmin-bugs

Comment 3 Marja Van Waes 2025-12-11 18:16:28 CET
Well, the "Function to list USB sticks" at end of the file seems to need a correction too.

Shouldn't "udisksctl info" be used, instead of "udisks --show-info"?

And maybe more changes are needed, if the udisksctl output differs from the old udisks one?
Comment 4 Tony Blackwell 2025-12-12 03:14:21 CET
Yes to Comment 3.  I'd not tested that.  See below:

$ ./dorsync

Do you want to Rsync (R), skip Rsync and just dump to usb (D) or quit (Q): d

egrep: warning: egrep is obsolescent; using grep -E
egrep: warning: egrep is obsolescent; using grep -E
egrep: warning: egrep is obsolescent; using grep -E
egrep: warning: egrep is obsolescent; using grep -E
egrep: warning: egrep is obsolescent; using grep -E
egrep: warning: egrep is obsolescent; using grep -E
./dorsync: line 289: udisks: command not found
./dorsync: line 293: [: =: unary operator expected
No USB drives found. Please insert one now.
Press Q to quit or any other key to continue:
Comment 5 Tony Blackwell 2025-12-12 03:16:49 CET
... but I really like dorsync - worth fixing!
Comment 6 Dan Fandrich 2025-12-12 18:45:54 CET
This bug is for Cauldron only and no RPMs are listed. That doesn't seem right.

CC: (none) => dan

Comment 7 Thomas Andrews 2025-12-12 19:21:55 CET
There isn't an rpm in the repos. The dorsync script is available for download at https://gitweb.mageia.org/qa/dorsync/plain/dorsync

I don't have the know-how or know the procedure, or who would do it, but the script that is there needs to be repaired/updated and replaced. 

Should this be a "Website" bug? That doesn't really sound quite right, either.
Thomas Andrews 2025-12-12 19:23:51 CET

Source RPM: https://gitweb.mageia.org/qa/dorsync/plain/dorsync => (none)

Comment 8 Dan Fandrich 2025-12-13 00:06:15 CET
I have no idea what this script is for. Is it for QA people to run on their workstations? Then IMHO it really ought to be put into an RPM so those people can easily install the correct version to run it, the same way as mgaadv (for example). Is it just used in some kind of automation on the servers? Then the Infrastructure group would make more sense, although there's a good case for making an RPM out of it in that case as well, such as mgapeople or youri (for example). Asking QA people to install a script from a gitweb site and keep it up to date manually seems a lot to ask.
Comment 9 Morgan Leijström 2025-12-13 00:14:51 CET
This script is used to download ISOs for testing, and later update them.
Yes having it as an rpm would be nice.

CC: (none) => fri

Comment 10 Thomas Andrews 2025-12-13 01:46:40 CET
It is a command line script to do essentially the same thing as Mageiasync. More properly, since dorsync was written first by several years, Mageiasync is a GUI version of the dorsync script.

Usage is described in https://wiki.mageia.org/en/ISO_testing_rsync_tools#Dorsync

I don't see any reason why it shouldn't be made into an rpm. Maybe tweaked a little here and there to make it easier for a novice, but then making it easier for a novice is why Mageiasync was created.

Why do we need/want both? Because some users prefer a GUI, while others prefer the command line.
Thomas Andrews 2025-12-13 01:49:28 CET

Keywords: validated_update => (none)

Comment 11 Frédéric "LpSolit" Buclin 2025-12-13 01:56:23 CET
I think I can fix the script. Let me see how far I can go.

Assignee: qa-bugs => LpSolit
Status: NEW => ASSIGNED

Comment 12 Dan Fandrich 2025-12-13 02:27:48 CET
Created attachment 15210 [details]
shellcheck log

Since someone is updating the script anyway, shellcheck has a lot of suggestions for improving it, most of which look good to me. I'm happy to implement the changes but I don't know how to test them. I'll upload the shellcheck log for posterity.
Comment 13 Frédéric "LpSolit" Buclin 2025-12-13 03:30:58 CET
Created attachment 15211 [details]
patch, v1

This fixes issues reported above.
Comment 14 Frédéric "LpSolit" Buclin 2025-12-13 03:40:19 CET
Updating the summary as the code was broken. The output of udisks and udisksctl are different, and they do not display the same fields. That's why I also use udevadm to get the missing data.

Summary: dorsync won't run until single char fix => dorsync doesn't run anymore because udisks is now obsolete

Comment 15 Mageia Robot 2025-12-13 05:52:56 CET
commit 197650b84d9ff5c696fb1dafa9237c3354542137
Author: Frédéric "LpSolit" Buclin <lpsolit@...>
Date:   Sat Dec 13 03:30:58 2025 +0100

    Fix dorsync with udisks2 (mga#34826)
    
    The output of udisks and udisksctl are different, and they do not
    display the same fields. Now use udevadm to get the missing data.
---
 Commit Link:
   https://gitweb.mageia.org/qa/dorsync/commit/?id=197650b84d9ff5c696fb1dafa9237c3354542137
Comment 16 Dan Fandrich 2025-12-13 05:54:19 CET
I've pushed your change in comment 13 as well as fixes for most of the shellcheck issues.

Does someone want to make an RPM out of this? I could tag a release of this first if so.
Comment 17 Frédéric "LpSolit" Buclin 2025-12-13 18:56:36 CET
Created attachment 15213 [details]
replace sha1sum by sha3-512sum

Here is another patch which replaces sha1sum by sha3-512sum. This patch also fixes arguments passed to rsync.

I cannot commit the patch myself to the dorsync repo: lpsolit DENIED by fallthru :(
Comment 18 Dan Fandrich 2025-12-14 08:28:57 CET
I've submitted that patch as well. I'll try to figure out next week why you don't have git access. What git URL are you using?
Comment 19 Bruno Cornec 2025-12-14 12:51:46 CET
I can work on a RPM.

CC: (none) => bruno

Comment 20 Frédéric "LpSolit" Buclin 2025-12-14 12:58:08 CET
(In reply to Bruno Cornec from comment #19)
> I can work on a RPM.

Wait before creating one. There is still some work to do first. Parameters such as the login and password must be moved out of the script, else they will be erased after each new update. I will work on it later next week.
Comment 21 Bruno Cornec 2025-12-14 13:20:28 CET
Package submitted to cauldron. Please test.

I use version 2.91. We should to tag the next version with that name and adapt the spec file to refer to it instead of master as of now.
Comment 22 Bruno Cornec 2025-12-14 13:22:21 CET
(In reply to Frédéric "LpSolit" Buclin from comment #20)
> (In reply to Bruno Cornec from comment #19)
> > I can work on a RPM.
> 
> Wait before creating one. There is still some work to do first. Parameters
> such as the login and password must be moved out of the script, else they
> will be erased after each new update. I will work on it later next week.

It's easy to source a conf file which would contain these info and not be part of the package.

if [ -e /etc/dorsync.conf ]; then
    source /etc/dorsync.conf
fi

declare in it the variables you need 

Bruno.
Anyway the package is also in test so I can update easily once you've pushed your modifications.
Comment 23 Thomas Andrews 2025-12-14 14:08:00 CET
This bug was filed against Cauldron but once this is ready I think it should be made available in Mageia 9, too. Consider, what better way to thoroughly test than with the many rounds of Mageia 10 isos to come?

I suppose policy would dictate that it be a backport for Mageia 9, but exceptions to that policy have been made in the past if there was sufficient reason. 

What do you guys think?
Comment 24 Bruno Cornec 2025-12-14 15:42:57 CET
I agree with you. I can make it easily a backported RPM or just an update. This only make sense for QA anyway I think, so won't disturb others. Let me know what's you preferred solution.
Comment 25 Frédéric "LpSolit" Buclin 2025-12-14 16:58:20 CET
(In reply to Dan Fandrich from comment #18)
> What git URL are you using?

ssh://git@git.mageia.org/qa/dorsync
Comment 26 Frédéric "LpSolit" Buclin 2025-12-14 18:28:17 CET
(In reply to Bruno Cornec from comment #22)
> It's easy to source a conf file which would contain these info and not be
> part of the package.
> 
> if [ -e /etc/dorsync.conf ]; then
>     source /etc/dorsync.conf
> fi

Shouldn't dorsync.conf be located at ~/.config/Mageia/dorsync.conf ? I don't see the point to have it in /etc/.
Comment 27 Frédéric "LpSolit" Buclin 2025-12-14 18:29:51 CET
(In reply to Frédéric "LpSolit" Buclin from comment #26)
> Shouldn't dorsync.conf be located at ~/.config/Mageia/dorsync.conf ? I don't
> see the point to have it in /etc/.

Moreover, this would be consistent with mageiaSync.conf which is also in ~/.config/Mageia/.
Comment 28 Frédéric "LpSolit" Buclin 2025-12-14 21:41:47 CET
Created attachment 15216 [details]
patch, v3

In this patch, I bump version to 3.0 with the following improvements:
- Store parameters in ~/.config/Mageia/dorsync.conf. When dorsync is executed for the first time, it creates this config file if not found.
- If dorsync.skip is missing, dorsync asks the user if he really wants to download all ISOs or if he wants to select which ones to download, and stores the user choice in dorsync.skip for future reuse.
- Update README.md
Comment 29 Bruno Cornec 2025-12-16 02:54:27 CET
New Package v3.00 submitted to cauldron (pending git update)
Comment 30 Dan Fandrich 2025-12-16 07:22:56 CET
I've submitted the patch in comment 28. Should I tag it as v3.0?

FWIW, it's better to use $XDG_CONFIG_HOME instead of $HOME/.config/ if the former is available, since that allows the user to specify an alternate location. This patch (minimally tested) should be enough:

diff --git a/dorsync b/dorsync
index c98fb37..624b636 100644
--- a/dorsync
+++ b/dorsync
@@ -15,7 +15,8 @@
 # progress indicator when dumping to USB.  #
 ############################################

-conf_file="$HOME/.config/Mageia/dorsync.conf"
+XDG_CONFIG_HOME="${XDG_CONFIG_HOME:-$HOME/.config}"
+conf_file="$XDG_CONFIG_HOME/Mageia/dorsync.conf"

 generate_config_file () {
   cat << EOF > "$conf_file"

I can commit that if you would like.
Comment 31 Dan Fandrich 2025-12-16 07:47:54 CET
w.r.t. comment 25, I figured out the permission issue: commits to /qa/… repos are restricted to members of mga-qa and sysadmins. Are you on the QA team? Does it make sense for you to be added?
Comment 32 Thomas Andrews 2025-12-16 13:33:48 CET
He's on the list of ISO testers. A late entry, added to the bottom. Was supposed to have received the email with password, etc.
Comment 33 Marja Van Waes 2025-12-16 14:26:40 CET
(In reply to Thomas Andrews from comment #32)
> He's on the list of ISO testers. A late entry, added to the bottom. Was
> supposed to have received the email with password, etc.

That doesn't automatically make him a member of the mga-qa ldap group ;-)
https://people.mageia.org/g/mga-qa.html
Comment 34 Marja Van Waes 2025-12-16 14:28:44 CET
(In reply to Marja Van Waes from comment #33)

> https://people.mageia.org/g/mga-qa.html

Shouldn't the top two entries be removed? Stormi and wilcal?
Comment 35 Thomas Andrews 2025-12-16 15:24:47 CET
You're right, being inactive they probably should be. I'd leave the rest.
Comment 36 Frédéric "LpSolit" Buclin 2025-12-16 19:22:37 CET
(In reply to Bruno Cornec from comment #29)
> New Package v3.00 submitted to cauldron (pending git update)

Thank you Bruno for the new RPM. :)


(In reply to Bruno Cornec from comment #24)
> I agree with you. I can make it easily a backported RPM or just an update.
> This only make sense for QA anyway I think, so won't disturb others. Let me
> know what's you preferred solution.

I guess making it available as an update for Mageia 9 would make it easier to discover. No need to enable backports.


(In reply to Dan Fandrich from comment #31)
> w.r.t. comment 25, I figured out the permission issue: commits to /qa/…
> repos are restricted to members of mga-qa and sysadmins. Are you on the QA
> team? Does it make sense for you to be added?

I'm not a member of the QA team. So I guess you will have to commit my patches in the future, if you agree to do it. :)


(In reply to Thomas Andrews from comment #32)
> He's on the list of ISO testers. A late entry, added to the bottom. Was
> supposed to have received the email with password, etc.

I got your email, yes. Thank you.


Do we want to close this bug as FIXED, or do you first want to see this new RPM in Mageia 9 ?
Comment 37 Thomas Andrews 2025-12-16 19:49:44 CET
I think we should wait for the Mageia 9 version. I believe that's where most of us would be using it until Cauldron stabilizes a bit more.
Comment 38 Dan Fandrich 2025-12-16 20:07:16 CET
w.r.t. comment 35, I've removed stormi and wilcal from mga-qa and mga-qa-committers, but they're still members of several other groups.

I've tagged the release as 3.0 and updated the .spec file to pull the sources based on the tag.
Comment 39 Thomas Andrews 2025-12-16 22:42:39 CET
I was going to try it in one of my two shiny new Cauldron installs, but it won't install because of a bad signature. Then again, there are something like 525 updates available, and I'm told several of them have bad/no signatures, as well. 

I'm suspecting a mirror syncing issue. I'll try again tomorrow when things might settle down a bit.
Comment 40 Bruno Cornec 2025-12-17 00:12:23 CET
(In reply to Dan Fandrich from comment #38)
> w.r.t. comment 35, I've removed stormi and wilcal from mga-qa and
> mga-qa-committers, but they're still members of several other groups.
> 
> I've tagged the release as 3.0 and updated the .spec file to pull the
> sources based on the tag.

Have you submitted a new RPM or do you want me to do it ?
Same question for mga9 ?
Comment 41 Frédéric "LpSolit" Buclin 2025-12-17 00:34:27 CET
Dan updated the RPM for Cauldron, but not for Mageia 9: https://svnweb.mageia.org/packages?view=revision&revision=2302848
Comment 42 Bruno Cornec 2025-12-17 01:30:39 CET
I have submitted a mga9 RPM for updates_testing. If you find it correct, you can move it to updates then.

Assignee: LpSolit => qa-bugs

Morgan Leijström 2025-12-17 10:01:58 CET

Summary: dorsync doesn't run anymore because udisks is now obsolete => dorsync fixing, improving, and packing to rpm.
Whiteboard: (none) => MGA9TOO

Comment 43 Morgan Leijström 2025-12-17 13:35:04 CET
Great to have this packaged.
I am testing it and is working on revising https://wiki.mageia.org/en/ISO_testing_rsync_tools#Dorsync to this packaged version.

One minor bug so far:
After downloading it asked if i want to dump to USB: it seems not to recognise big USB drives?
Not: Corsair  Voyager GTX 128GB  nor JetFlash Transcend 256GB
OK: Samsung  Flash Drive FIT 32GB
Comment 44 Thomas Andrews 2025-12-17 14:35:37 CET
If using qarepo in Mageia 9, the rpm to search for is 
mga-dorsync-3.0-3.1.mga9.noarch.rpm
Comment 45 Frédéric "LpSolit" Buclin 2025-12-17 15:50:08 CET
(In reply to Morgan Leijström from comment #43)
> After downloading it asked if i want to dump to USB: it seems not to
> recognise big USB drives?
> Not: Corsair  Voyager GTX 128GB  nor JetFlash Transcend 256GB
> OK: Samsung  Flash Drive FIT 32GB

What is the output of this command ?
 lsblk -ndpro NAME,RM,TRAN,TYPE | grep "usb disk$"

In my case, I get this:
 /dev/sda 0 usb disk (for my external hard drive)
 /dev/sdb 1 usb disk (for my USB key)

Only my USB key is listed as removable (0 = not removable, 1 = removable), and only removable USB drives are available to copy an ISO on it. I guess this is to avoid to erase your external hard drive.

When I updated dorsync to work with udisks2, I didn't remove this test. This test was already there to only select USB drives seen as removable.

CC: (none) => LpSolit

Comment 46 Thomas Andrews 2025-12-17 15:57:57 CET
No problem here dumping to a Teamgroup 128GB drive. That drive had been formatted to NTSC before detecting. No idea if that makes a difference.

But I have another issue. I set the conf to download to a different location than the one I had used with Mageiasync. The first time I ran it, I told it to sync/download just the Live Plasma and x86_64 CI isos. That worked OK.

Then I went back later to try getting one of the isos I had skipped before. But I wasn't given the chance. Entering "R" sent me straight to re-syncing the isos already downloaded and not looking at the others.
Comment 47 Frédéric "LpSolit" Buclin 2025-12-17 16:00:43 CET
(In reply to Thomas Andrews from comment #46)
> Then I went back later to try getting one of the isos I had skipped before.
> But I wasn't given the chance. Entering "R" sent me straight to re-syncing
> the isos already downloaded and not looking at the others.

That's because dorsync.skip has been created and now dorsync uses it to know what to update. You can manually edit dorsync.skip.
Comment 48 Frédéric "LpSolit" Buclin 2025-12-17 16:14:41 CET
(In reply to Frédéric "LpSolit" Buclin from comment #47)
> That's because dorsync.skip has been created and now dorsync uses it to know
> what to update. You can manually edit dorsync.skip.

I could easily update dorsync to ask if you want to edit the list, if you think that would be useful.
Comment 49 Thomas Andrews 2025-12-17 16:15:46 CET
Oh, yes - I almost forgot. A minor niggle, probably something to address in the future:

When I went to dump the iso, I had neglected to insert the usb drive. USB "drives" were detected, but they were from my generic card reader. 4 of them, sda through sdd, for different card types, all of them empty.

There was no way to insert a stick at that point and use it, except to quit, and run the script again. When I did that, the sticks I had inserted (sde and sdf) were detected, as were the card readers.

I think the script should ignore usb devices with a size of 0, and if none that are usable are found it should ask the user to insert one and try again, or quit.

For what it's worth, Isodumper ignores my card readers if they are empty.
Comment 50 Morgan Leijström 2025-12-17 16:45:15 CET
(In reply to Frédéric "LpSolit" Buclin from comment #45)
> What is the output of this command ?

[morgan@svarten ~]$ lsblk -ndpro NAME,RM,TRAN,TYPE | grep "usb disk$"
/dev/sdc 0 usb disk  (JetFlash Transcend 256GB)
/dev/sdd 1 usb disk  (Flash Drive FIT 32GB)

[morgan@svarten ~]$ dorsync

Do you want to Rsync (R), skip Rsync and just dump to usb (D) or quit (Q): d


USB drives found:

    Device      Size    When Found                      Model
1) /dev/sdd     59.7Gb  ons 17 dec 2025 16:05:26 CET    Samsung Flash_Drive_FIT



(In reply to Thomas Andrews from comment #46)
> No problem here dumping to a Teamgroup 128GB drive. That drive had been
> formatted to NTSC before detecting. No idea if that makes a difference.

My 256MB is exfat (directly from package never used before)
The 32 and 128 MB both had Mageia 9 Live systems on them.

(In reply to Frédéric "LpSolit" Buclin from comment #48)
> I could easily update dorsync to ask if you want to edit the list, if you
> think that would be useful.

Yes!  Maybe add a new option to the first question:
 S=Select ISOs (rewrite dosrync.skip)

(In reply to Thomas Andrews from comment #49)
> When I went to dump the iso, I had neglected to insert the usb drive. USB
> "drives" were detected, but they were from my generic card reader. 4 of
> them, sda through sdd, for different card types, all of them empty.
...
> For what it's worth, Isodumper ignores my card readers if they are empty.

Likewise, Isodumper lists both my big USB flash that dorsync do not list, so maybe look at how Isodumper do, or ask its developer.
Comment 51 Thomas Andrews 2025-12-17 16:46:44 CET
(In reply to Frédéric "LpSolit" Buclin from comment #48)
> (In reply to Frédéric "LpSolit" Buclin from comment #47)
> > That's because dorsync.skip has been created and now dorsync uses it to know
> > what to update. You can manually edit dorsync.skip.
> 
> I could easily update dorsync to ask if you want to edit the list, if you
> think that would be useful.

I dunno... I could go either way on that one. I haven't used dorsync before this bug came up, so I had to look up the wiki to learn where dorsync.skip is. But once I looked at it, I saw how easy it would be to edit, so it probably wouldn't be an issue except for new users. What do others think?

Either way, I can see we'll have to remember to remind that dorsync.skip should be edited for each new release the same way we remind to make changes in the file/folder names.
Comment 52 Thomas Andrews 2025-12-17 17:00:01 CET
I ran it again, this time inserting a card reader stick with a 128GB SD card, a 128GBusb stick, a 16GB usb stick, and just for giggles a 2TB external drive. I get this:

$ dorsync

Do you want to Rsync (R), skip Rsync and just dump to usb (D) or quit (Q): d


USB drives found:

    Device      Size    When Found                      Model
1) /dev/sda     0Gb     Wed Dec 17 10:52:24 AM EST 2025 Generic- SD_MMC
2) /dev/sdb     0Gb     Wed Dec 17 10:52:24 AM EST 2025 Generic- Compact_Flash
3) /dev/sdc     0Gb     Wed Dec 17 10:52:24 AM EST 2025 Generic- SM_xD-Picture
4) /dev/sdd     0Gb     Wed Dec 17 10:52:24 AM EST 2025 Generic- MS_MS-Pro
5) /dev/sde     14.6Gb  Wed Dec 17 10:52:24 AM EST 2025 General UDisk
6) /dev/sdf     116.0Gb Wed Dec 17 10:52:24 AM EST 2025 SMI USB_DISK
7) /dev/sdg     115.0Gb Wed Dec 17 10:52:24 AM EST 2025 Mass Storage_Device

Please choose which USB to use or Q to quit and press <enter>: 

The 2TB drive was not detected by dorsync, though it was by the Plasma device notifier. Isodumper detects the 2TB drive, too.
Comment 53 Frédéric "LpSolit" Buclin 2025-12-17 17:18:59 CET
OK, so I will update dorsync to:
- ignore the "removable" bit and list *all* USB drives.
- ignore drives for which the size is zero.
- let the user change his selection even if dorsync.skip already exists
- store in dorsync.skip the release for which the list has been created, and if it doesn't match the "release" variable in dorsync.conf, dorsync will ask the user to select which ISOs to download/sync again and invalidate the previous selection.


@Dan: do you want to give me permissions to update the dorsync git repo, or do you prefer to commit my patches again as you did these last few days ?
Comment 54 Morgan Leijström 2025-12-17 18:09:47 CET
Sounds good to me, except:

> - store in dorsync.skip the release for which the list has been created, and if it doesn't match the "release" variable in dorsync.conf, dorsync will ask the user to select which ISOs to download/sync again and invalidate the previous selection.

Users are supposed to edit dorsync.skip, dorsync.conf, and names of all the files, at a new release. (documented already)  Unless we want to change that?

IF we are going to change it maybe automate it all the way like mageiasync?
Comment 55 Morgan Leijström 2025-12-17 18:25:29 CET
Minor bug to fix:

When the path set by location= in dorsync.conf does not exist, and user selects to Dump without downloading, the script hangs. (exit by ctrl-C)

It should exit with a message that that path is invalid.


(It works OK if user selects to download, then that folder is created)
Comment 56 Morgan Leijström 2025-12-17 19:14:36 CET
Usability: May I suggest a change in this interaction:

"No dorsync.skip file found. Do you really want to download/update all ISOs?
Press C to choose which ISOs to select, or any other key to continue with all ISOs  "

On the question "Do you really want to download/update all ISOs?" Ii is easy to answer N for no but that means yes...

I suggest to change key to Y and text to: 

"No dorsync.skip file was found. Do you want to download/update all ISOs?
Press Y to download all, or any other key to start selection."
Comment 57 Dan Fandrich 2025-12-17 20:29:51 CET
> do you want to give me permissions to update the dorsync git repo, or do you prefer to commit my patches again as you did these last few days ?

It makes sense that you get direct write access, but the way that's currently done is to add you to mga-qa. If that's not possible, I don't mind being an intermediary if it's not going to be a high volume of changes, but I ask that you upload or e-mail patches using 'git format-patch' to make the job easier.
Comment 58 Thomas Andrews 2025-12-17 21:02:58 CET
(In reply to Morgan Leijström from comment #54)
> 
> IF we are going to change it maybe automate it all the way like mageiasync?

Mageiasync isn't automated "all the way" either. When releases are changed, like from alpha1 to alpha2, or alpha to beta, we still have to rename the files/folders we downloaded for the previous release unless we want to download the whole thing over again.
Comment 59 Frédéric "LpSolit" Buclin 2025-12-18 01:22:30 CET
(In reply to Dan Fandrich from comment #57)
> > do you want to give me permissions to update the dorsync git repo, or do you prefer to commit my patches again as you did these last few days ?
> 
> It makes sense that you get direct write access, but the way that's
> currently done is to add you to mga-qa.

OK, let's do that.
Comment 60 Dan Fandrich 2025-12-19 18:18:34 CET
Thomas, as QA team leader, do you approve of adding lpsolit to the mga-qa group?
Comment 61 Thomas Andrews 2025-12-19 19:13:24 CET
OK with me.
Comment 62 Dan Fandrich 2025-12-19 21:04:06 CET
lpsolit has now been added. You should now be able to commit directly to the dorsync repo.
Comment 63 Herman Viaene 2025-12-20 15:54:35 CET
MGA9-64 server Plasma Wayland on Compaq H000SB.
No installation issues.
Copied settings from mageia-sync on my desktop, with one exception:
the edition is in mageia-sync mageia10-alpha1qaiso, while the release line in dorsync.conf has to be mageia10-alpha1.
$ dorsync 

Do you want to Rsync (R), skip Rsync and just dump to usb (D) or quit (Q): R

No dorsync.skip file found. Do you really want to download/update all ISOs?
Press C to choose which ISOs to select, or any other key to continue with all ISOs: C

For each of the following ISOs, press Y to download/update it, or any other key to skip it:
  Mageia-10-alpha1-Live-GNOME-x86_64 : n
  Mageia-10-alpha1-Live-Plasma-x86_64 : n
  Mageia-10-alpha1-Live-Xfce-i686 : y
  Mageia-10-alpha1-Live-Xfce-x86_64 : Y
  Mageia-10-alpha1-i686 : n
  Mageia-10-alpha1-x86_64 : n

Your selection has been saved in dorsync.skip.

Starting rsync..
receiving incremental file list
./
Mageia-10-alpha1-Live-Xfce-i686/
Mageia-10-alpha1-Live-Xfce-i686/DATE.txt
             32 100%   31.25kB/s    0:00:00 (xfr#1, to-chk=19/23)
Mageia-10-alpha1-Live-Xfce-i686/Mageia-10-alpha1-Live-Xfce-i686.iso
  3,722,596,352 100%    4.61MB/s    0:12:50 (xfr#2, to-chk=18/23)
Mageia-10-alpha1-Live-Xfce-i686/Mageia-10-alpha1-Live-Xfce-i686.iso.md5
             70 100%    0.10kB/s    0:00:00 (xfr#3, to-chk=17/23)
Mageia-10-alpha1-Live-Xfce-i686/Mageia-10-alpha1-Live-Xfce-i686.iso.sha3
            166 100%    0.23kB/s    0:00:00 (xfr#4, to-chk=16/23)
Mageia-10-alpha1-Live-Xfce-i686/Mageia-10-alpha1-Live-Xfce-i686.iso.sha512
            166 100%    0.23kB/s    0:00:00 (xfr#5, to-chk=15/23)
Mageia-10-alpha1-Live-Xfce-i686/Mageia-10-alpha1-Live-Xfce-i686.langs
             27 100%    0.04kB/s    0:00:00 (xfr#6, to-chk=14/23)
Mageia-10-alpha1-Live-Xfce-i686/Mageia-10-alpha1-Live-Xfce-i686.lst
         76,770 100%  106.64kB/s    0:00:00 (xfr#7, to-chk=13/23)
Mageia-10-alpha1-Live-Xfce-i686/Mageia-10-alpha1-Live-Xfce-i686.lst.full
         54,956 100%   75.59kB/s    0:00:00 (xfr#8, to-chk=12/23)
Mageia-10-alpha1-Live-Xfce-i686/Mageia-10-alpha1-Live-Xfce-i686.lst.leaves
          3,728 100%    5.12kB/s    0:00:00 (xfr#9, to-chk=11/23)
Mageia-10-alpha1-Live-Xfce-i686/Mageia-10-alpha1-Live-Xfce-i686.lst.names
         36,964 100%   50.42kB/s    0:00:00 (xfr#10, to-chk=10/23)
Mageia-10-alpha1-Live-Xfce-x86_64/
Mageia-10-alpha1-Live-Xfce-x86_64/DATE.txt
             32 100%    0.04kB/s    0:00:00 (xfr#11, to-chk=9/23)
Mageia-10-alpha1-Live-Xfce-x86_64/Mageia-10-alpha1-Live-Xfce-x86_64.iso
  4,194,183,168 100%    4.85MB/s    0:13:44 (xfr#12, to-chk=8/23)
Mageia-10-alpha1-Live-Xfce-x86_64/Mageia-10-alpha1-Live-Xfce-x86_64.iso.md5
             72 100%    0.08kB/s    0:00:00 (xfr#13, to-chk=7/23)
Mageia-10-alpha1-Live-Xfce-x86_64/Mageia-10-alpha1-Live-Xfce-x86_64.iso.sha3
            168 100%    0.18kB/s    0:00:00 (xfr#14, to-chk=6/23)
Mageia-10-alpha1-Live-Xfce-x86_64/Mageia-10-alpha1-Live-Xfce-x86_64.iso.sha512
            168 100%    0.18kB/s    0:00:00 (xfr#15, to-chk=5/23)
Mageia-10-alpha1-Live-Xfce-x86_64/Mageia-10-alpha1-Live-Xfce-x86_64.langs
             27 100%    0.03kB/s    0:00:00 (xfr#16, to-chk=4/23)
Mageia-10-alpha1-Live-Xfce-x86_64/Mageia-10-alpha1-Live-Xfce-x86_64.lst
         78,995 100%   82.33kB/s    0:00:00 (xfr#17, to-chk=3/23)
Mageia-10-alpha1-Live-Xfce-x86_64/Mageia-10-alpha1-Live-Xfce-x86_64.lst.full
         57,064 100%   58.66kB/s    0:00:00 (xfr#18, to-chk=2/23)
Mageia-10-alpha1-Live-Xfce-x86_64/Mageia-10-alpha1-Live-Xfce-x86_64.lst.leaves
          3,706 100%    3.81kB/s    0:00:00 (xfr#19, to-chk=1/23)
Mageia-10-alpha1-Live-Xfce-x86_64/Mageia-10-alpha1-Live-Xfce-x86_64.lst.names
         39,035 100%   39.87kB/s    0:00:00 (xfr#20, to-chk=0/23)

sent 638 bytes  received 7,919,066,034 bytes  4,966,488.98 bytes/sec
total size is 7,917,131,666  speedup is 1.00

Completed rsync

7.4G    Total

3.5G    Mageia-10-alpha1-Live-Xfce-i686
4.0G    Mageia-10-alpha1-Live-Xfce-x86_64

Starting checks, please ensure the dates are correct

PATH: Mageia-10-alpha1-Live-Xfce-i686
ISO:  Mageia-10-alpha1-Live-Xfce-i686.iso
DATE: Mon Dec  8 03:57:28 PM CET 2025
MD5:  OK     SHA3: OK     SHA512: OK

PATH: Mageia-10-alpha1-Live-Xfce-x86_64
ISO:  Mageia-10-alpha1-Live-Xfce-x86_64.iso
DATE: Mon Dec  8 05:06:54 PM CET 2025
MD5:  OK     SHA3: OK     SHA512: OK


------------------------------------

All tests passed OK. Yippee!
Please check the dates are correct.

------------------------------------


Do you want to dump one onto USB? [y/n]n
Looks OK to me.

Whiteboard: MGA9TOO => MGA9TOO, MGA9-64-OK
CC: (none) => herman.viaene

Comment 64 Morgan Leijström 2025-12-22 00:05:20 CET
Sorry, this should already have been set to developer in comment 53 onwards, assigning now.

Removing the OK as we should test the upcoming version.

Please, when new version is built, assign back to QA!

Whiteboard: MGA9TOO, MGA9-64-OK => MGA9TOO
Assignee: qa-bugs => LpSolit

Comment 65 Frédéric "LpSolit" Buclin 2025-12-26 20:14:37 CET
(In reply to Dan Fandrich from comment #62)
> lpsolit has now been added. You should now be able to commit directly to the
> dorsync repo.

This still doesn't work:

remote: FATAL: W refs/heads/master qa/dorsync lpsolit DENIED by fallthru
Comment 66 Frédéric "LpSolit" Buclin 2025-12-31 01:02:45 CET
(In reply to Frédéric "LpSolit" Buclin from comment #65)
> This still doesn't work:
> 
> remote: FATAL: W refs/heads/master qa/dorsync lpsolit DENIED by fallthru

@Dan: do you have a moment to debug what's wrong here?
Comment 67 Marja Van Waes 2025-12-31 14:05:38 CET
Adding the flag: affects_mga9 +
to all bugs with MGA9TOO on the whiteboard, without removing MGA9TOO (for now).

Flags: (none) => affects_mga9+

Comment 68 Dan Fandrich 2026-01-08 07:28:35 CET
(In reply to Frédéric "LpSolit" Buclin from comment #66)
> @Dan: do you have a moment to debug what's wrong here?

The gitolite configuration file containing the expanded list of group members hasn't been updated since June, so it doesn't see you as a member. I'll try to figure out why that is.
Comment 69 Dan Fandrich 2026-01-08 08:55:24 CET
The git permissions issue was due a bad precondition that meant group member changes were only reflected whenever somebody happened to change an SSH key. I've updated it manually so you should have access to dorsync now and I will fix it properly tomorrow.
Comment 70 Mageia Robot 2026-01-08 17:52:01 CET
commit b7527f083a630525b8df210386a47f9896f72e4c
Author: Dan Fandrich <danf@...>
Date:   Wed Jan 7 23:41:29 2026 -0800

    Separate running the SSH keys script from the mgagit script (mga#34826)
    
    The mgagit script was previously run after the SSH keys script, but only
    if an SSH key was updated. That meant that any group changes were not
    reflected until after someone changed an SSH key.
    
    Decouple the running of the two scripts since they are not (completely)
    dependent on each other.
---
 Commit Link:
   https://gitweb.mageia.org/infrastructure/puppet/commit/?id=b7527f083a630525b8df210386a47f9896f72e4c
Comment 71 Mageia Robot 2026-01-08 21:39:36 CET
commit 6a019527c73b72a92a1309ffd7ad566806c62806
Author: Frédéric Buclin <LpSolit@...>
Date:   Fri Dec 26 20:11:15 2025 +0100

    Improve the detection of USB drives, and let the user edit dorsync.skip
    (mga#34826)
---
 Commit Link:
   https://gitweb.mageia.org/qa/dorsync/commit/?id=6a019527c73b72a92a1309ffd7ad566806c62806
Comment 72 Frédéric "LpSolit" Buclin 2026-01-08 21:42:45 CET
(In reply to Bruno Cornec from comment #42)
> I have submitted a mga9 RPM for updates_testing. If you find it correct, you
> can move it to updates then.

Bruno, can you update the RPM with this last commit, please ?

Assignee: LpSolit => bruno

Comment 73 Frédéric "LpSolit" Buclin 2026-01-08 21:43:21 CET
(In reply to Dan Fandrich from comment #69)
> I've updated it manually so you should have access to dorsync now and I will
> fix it properly tomorrow.

It works now. Thank you!
Comment 74 Dan Fandrich 2026-01-08 22:02:04 CET
> It works now. Thank you!

Excellent!

You'll need to bump the version and push a new dorsync tag before making a new RPM. Remember to update the version embedded in the script to match.
Comment 75 Frédéric "LpSolit" Buclin 2026-01-08 22:04:31 CET
(In reply to Dan Fandrich from comment #74)
> You'll need to bump the version and push a new dorsync tag before making a
> new RPM. Remember to update the version embedded in the script to match.

IMO, we should drop the hardcoded version in the script and only use git tags. What do you think?
Comment 76 Dan Fandrich 2026-01-08 22:41:05 CET
It's nice to be able to tell what version you have on disk, but it's probably worse if it becomes out of sync with reality. And as long as it's a manual process, it's guaranteed to fail at some point. So, I wouldn't be upset if it were removed.
Comment 77 Bruno Cornec 2026-01-08 23:19:17 CET
(In reply to Frédéric "LpSolit" Buclin from comment #72)
> (In reply to Bruno Cornec from comment #42)
> > I have submitted a mga9 RPM for updates_testing. If you find it correct, you
> > can move it to updates then.
> 
> Bruno, can you update the RPM with this last commit, please ?

Yes, will do ASA the version and tags are correct.
Comment 78 Bruno Cornec 2026-01-08 23:26:13 CET
(In reply to Dan Fandrich from comment #76)
> It's nice to be able to tell what version you have on disk, but it's
> probably worse if it becomes out of sync with reality. And as long as it's a
> manual process, it's guaranteed to fail at some point. So, I wouldn't be
> upset if it were removed.

What about using a commit hook in all Mageia git infra repo, with inspiration from https://dev.to/elpalomo/automate-semantic-versioning-and-tagging-with-a-git-hook-577b

That would ease a lot the work on our side once it works.
Comment 79 Frédéric "LpSolit" Buclin 2026-01-08 23:36:20 CET
Done! I removed the hardcoded version and tagged the latest commit.
Comment 80 Dan Fandrich 2026-01-09 00:54:41 CET
(In reply to Bruno Cornec from comment #78)
> What about using a commit hook in all Mageia git infra repo, with
> inspiration from
> https://dev.to/elpalomo/automate-semantic-versioning-and-tagging-with-a-git-hook-577b

That would work, but only for users who have set up the hook, a manual process leading to another point of failure. I'm not too worried about dorsync since it's likely to have very few contributors and I suspect will be pretty stable again once the current churn is over. If someone put the effort in for a general solution we could install it server-wide on all our git repos then all Mageians could learn to use it and expect it to be everywhere.
Comment 81 Bruno Cornec 2026-01-09 02:22:22 CET
(In reply to Frédéric "LpSolit" Buclin from comment #79)
> Done! I removed the hardcoded version and tagged the latest commit.

Package v3.1 pushed to cauldron.
Comment 82 Frédéric "LpSolit" Buclin 2026-01-09 21:22:22 CET
(In reply to Bruno Cornec from comment #81)
> Package v3.1 pushed to cauldron.

Thanks! Could you update the RPM in Mageia 9 too, please? :)
Comment 83 Bruno Cornec 2026-01-10 00:11:30 CET
Just submitted to core/updates_testing
Frédéric "LpSolit" Buclin 2026-01-10 00:17:44 CET

Assignee: bruno => qa-bugs

Comment 84 katnatek 2026-01-10 03:07:14 CET
[S]RPM: mga-dorsync-3.1-1.mga9
katnatek 2026-01-10 03:08:46 CET

Version: Cauldron => 9
Status comment: (none) => Package in comment 84
Whiteboard: MGA9TOO => (none)

Comment 85 Morgan Leijström 2026-01-10 12:26:30 CET
Good

Comment regarding file checking:
Should it really check all ISO:s even when I choosed to only update one of them (all was downloaded previously).

Also, maybe there could be an option to not check all three checksums per ISO.  It is good that *someone* test that *all* checksums files are good though, but takes time for every user...
Comment 86 Herman Viaene 2026-01-12 14:14:48 CET
# dorsync 
/usr/bin/dorsync: line 21: /root/.config/Mageia/dorsync.conf: No such file or directory
Missing file /root/.config/Mageia/dorsync.conf has been created.
Please edit this file and then rerun dorsync.
There is no such folder under /root/.config/
Comment 87 Herman Viaene 2026-01-12 14:45:09 CET
MGA9-64 server Plasma Wayland on Compaq H000SB.
No installation issues.
Used settings from previous test above.
$ dorsync 

Do you want to Rsync (R), skip Rsync and just dump to usb (D) or quit (Q): R

dorsync.skip file found:

  Mageia-10-alpha1-Live-GNOME-x86_64  SKIPPED 
  Mageia-10-alpha1-Live-Plasma-x86_64  SKIPPED 
  Mageia-10-alpha1-Live-Xfce-i686  SELECTED 
  Mageia-10-alpha1-Live-Xfce-x86_64  SELECTED 
  Mageia-10-alpha1-i686  SKIPPED 
  Mageia-10-alpha1-x86_64  SKIPPED 

Do you want to continue with this selection?
Press Y to continue, or any other key to modify this list: Y
Starting rsync..
receiving incremental file list
./
Then load of download lines and at the end:
sent 1,010,296 bytes  received 2,901,228,100 bytes  4,000,328.60 bytes/sec
total size is 8,025,900,298  speedup is 2.77

Completed rsync

7.5G    Total

3.6G    Mageia-10-alpha1-Live-Xfce-i686
4.0G    Mageia-10-alpha1-Live-Xfce-x86_64

Starting checks, please ensure the dates are correct

PATH: Mageia-10-alpha1-Live-Xfce-i686
ISO:  Mageia-10-alpha1-Live-Xfce-i686.iso
DATE: Sat Dec 20 02:18:13 PM CET 2025
MD5:  OK     SHA3: OK     SHA512: OK

PATH: Mageia-10-alpha1-Live-Xfce-x86_64
ISO:  Mageia-10-alpha1-Live-Xfce-x86_64.iso
DATE: Sat Dec 20 03:23:01 PM CET 2025
MD5:  OK     SHA3: OK     SHA512: OK


------------------------------------

All tests passed OK. Yippee!
Please check the dates are correct.

------------------------------------


Do you want to dump one onto USB? [y/n]n
Looks good to me, but I didn't test the dumping to USB. I use a Ventoy stick.
Comment 88 Mageia Robot 2026-01-12 15:26:04 CET
commit 0eb150ba760b0ab9c286e2acd5105a762b90af1f
Author: Frédéric Buclin <LpSolit@...>
Date:   Mon Jan 12 15:24:47 2026 +0100

    Make sure the path exists before creating dorsync.conf (mga#34826)
---
 Commit Link:
   https://gitweb.mageia.org/qa/dorsync/commit/?id=0eb150ba760b0ab9c286e2acd5105a762b90af1f
Comment 89 Frédéric "LpSolit" Buclin 2026-01-12 15:54:13 CET
(In reply to Herman Viaene from comment #86)
> # dorsync 
> /usr/bin/dorsync: line 21: /root/.config/Mageia/dorsync.conf: No such file
> or directory
> Missing file /root/.config/Mageia/dorsync.conf has been created.
> Please edit this file and then rerun dorsync.
> There is no such folder under /root/.config/

Thanks for catching that. This is fixed in comment 88.



(In reply to Morgan Leijström from comment #85)
> Comment regarding file checking:
> Should it really check all ISO:s even when I choosed to only update one of
> them (all was downloaded previously).

I will try to improve this.
Comment 90 Frédéric "LpSolit" Buclin 2026-01-12 16:05:07 CET
I suddenly see new .torrent files which were not present before. Is that expected?
Comment 91 Morgan Leijström 2026-01-12 19:14:43 CET
The .torrent files in root folder is OK.
The are recently created.

I also see the three new .gpg files in each subfolder for the isos, they were created a couple days ago, after it was agreed to release.

All as it should be.

We should probably manually remove the .torrent and .gpg files before syncing next ISOs round (same time as when we rename the rest).
Comment 92 Frédéric "LpSolit" Buclin 2026-01-12 19:34:32 CET
Do we want dorsync to download .torrent files too? Or should it ignore them?
Comment 93 Morgan Leijström 2026-01-12 19:51:24 CET
We do not need them for testing, but it is a convenient way to get them today for us who started seeding.  Otherways we could get hem official from mirrors, but maybe hours later, but that probably do not make much difference...

/ For seeding, I copied the files to my archive files, then started torrenting from there so dorsync can update the files separately... /
Comment 94 Thomas Andrews 2026-01-13 16:09:21 CET
(In reply to Frédéric "LpSolit" Buclin from comment #92)
> Do we want dorsync to download .torrent files too? Or should it ignore them?

I think they should be ignored. 

Remember what the purpose of dorsync is, a tool to help QA iso testers to sync/download test isos. Torrents have nothing to do with that. They are for the public, after the testing is done and the isos are released.
Comment 95 Frédéric "LpSolit" Buclin 2026-01-13 18:03:20 CET
(In reply to Thomas Andrews from comment #94)
> > Do we want dorsync to download .torrent files too? Or should it ignore them?
> 
> I think they should be ignored. 

OK, perfect. I agree to exclude them too. I asked because the original code didn't exclude them, so I wondered if that was on purpose.
Comment 96 Thomas Andrews 2026-01-13 19:49:16 CET
You'd have to ask MrsB. that one.
katnatek 2026-01-17 02:11:43 CET

Source RPM: (none) => dorsync

katnatek 2026-01-17 02:12:54 CET

Source RPM: dorsync => mga-dorsync


Note You need to log in before you can comment on or make changes to this bug.