Bug 1461

Duplicate versions of which packages ? Duplicated where ?

CC: (none) => boklm

I think I already explained that, but I'll try again.  When you have release + updates, if you have a package in updates, there are versions of it in both media (the original release version and the updates version).  For most users there is no need to have access to the original release version and it's just a waste of disk space on a small organizational mirror.  So the idea is that in an 'updated' medium, it would have hard links to all of the packages in release that haven't been updated, and links to the ones in updates that have.  You would only have one version of each package, not duplicate versions like you can have now if you mirror release and updates, and therefore use considerably less disk space.  Again, remember I'm talking about helping out small organizational mirrors here, which would only mirror the updated media, while public mirrors would continue to mirror the whole tree.  Also, for reference, my Mandriva bug reminded me that Mandrake/Mandriva actually used to do something very similar to this with the "community" tree for 2006.0, before they switched to the new build system.

Ok. However this is not so simple :
 - it needs a script to find which rpm files should be in the repository, and make the hardlinks
 - generate the hdlists for this repository for each new package, which takes a few minutes on the server
 - add new repositories for core/nonfree/tainted.
 - explain to people what it is, as most people will not understand why there is a new repository with the same packages
 - some people will ask to have the same for backports

So I don't think it's worth it for saving a few gigabytes on only some servers.

None of those things look like an impediment to me.
- First remember this has already been done before as I mentioned.  I know there's something somewhere that indexes what RPMS are built out of which SRPMS, so I'm sure it wouldn't be that hard to implement.
- Generating hdlists may take a few minutes, but it's trivial to do and we have well defined processes for doing that already.
- Yes, it would add one more directory under core/nonfree/tainted, no big deal.
- What it is for could be mentioned on the mailing lists and explained on the wiki right where the other directories' purposes are explained, no big deal.
- backports doesn't have the release/updates distinction, so it's not relevant there.

It may not seem worth it to you, but those gigabytes are worth a lot more to some of us, hence the request.

If you want a repository with no duplicates between release and updates repository, I think it should be possible to write a script to download only the required packages from the mirrors, and generate the hdlists locally. I don't think it needs to be done on the mirrors.

It would be a lot easier to do this at the build system level where your script has access to all of the metadata about the packages.  At the remote level you would only have access to the filenames and would basically be guessing based on that.  Also it would require anyone that could use this to implement their own script (and that's assuming that they all could).  It would be a lot more useful to do this in the build system and have it available at that level.

(In reply to comment #6)
> It would be a lot easier to do this at the build system level where your script
> has access to all of the metadata about the packages.  At the remote level you
> would only have access to the filenames and would basically be guessing based
> on that.

You have access to the source rpm name from the hdlists. And most packages don't change name between release and updates, so the filename would probably be enough to remove 99% of unneeded packages, if you don't want to use hdlists.

> Also it would require anyone that could use this to implement their
> own script (and that's assuming that they all could).

They don't need to implement their own script, if someone create it and share it.

> It would be a lot more
> useful to do this in the build system and have it available at that level.

Why would it be more useful to do it there ?

It would be a lot of work to do it and maintain it, add complexity on the mirrors and build system, for something that most people will not use, so I don't think we should do it.

And if we start doing this, then some people will come and say "I only use gnome and not KDE, can you create a hardlinked repository with only gnome packages ?", then "I only use server packages, can you create a hardlinked repository with no X ?", etc ...

It would take some work to do it, but it would only need to be done once and it would be done.  It would take zero work to maintain and add no complexity to anything.  You're only assuming that nobody would use it.

Your claims about other things people might ask for are both irrelevant and downright silly.  If you don't want to implement this yourself, that's unfortunate, but of course it's your choice.  It would be nice if someone who had the power and ability to do it could.  Something that could result in as much as a 30% reduction in used space for those who made use of it would really be nice.

(In reply to comment #7)
> And if we start doing this, then some people will come and say "I only use
> gnome and not KDE, can you create a hardlinked repository with only gnome
> packages ?", then "I only use server packages, can you create a hardlinked
> repository with no X ?", etc ...

I hate to respond further to this, since it is totally irrelevant and off topic, but excluding things like that could be done very easily by people just using rsync excludes.  Doing what I'm asking for would not be anywhere near that simple from the outside end, but would not be that difficult or complicated to do upstream, and the cost to do it upstream would be quite minimal.

I wouldn't mess with mirrors like that. The more complicate you make them the more problems you get. Few gigabytes of disk space ain't worth it. But if that complicated system breaks, thousands of users are affected.

CC: (none) => sander.lepik

Sander, like I said before, this has already been done before and it worked great.  It's not complicated, and it doesn't require "messing with" anything, and it's not something that would just up and break for any specific reason.  Honestly it would only be a relatively minor change to the build system.  It's one thing to object to doing the implementation work, I can understand that.  Unreasonable objections that don't make any sense are another thing.  If the gigabytes of space weren't worth it, I wouldn't have made the request.

Let me try to explain my "this has already been done before" comment further.  I'm referring to MDV 2006.0, where the layout was a little different, but the exact same thing was done.  Back then, there was a 2006.0 tree under both official/ and under devel/.  The official/2006.0 tree was not changed after the release was made.  Updates were stored separately in their own tree in official/updates/2006.0.  In devel/2006.0, it looked just like the official/2006.0 tree, except in main and contrib where the packages were stored, the original packages were there, but when a package was updated, the older one was removed and only the updated version was stored there.  The devel/2006.0 tree looked like a pristine tree as if the release had just come out with all of the updates applied, and was implemented with hard links (I believe) to where they were stored separately under official/2006.0 and official/updates/2006.0, so the packages media under devel/2006.0 were exactly what I'm asking for with this updated medium.

It was after the 2006.0 release that the new build system and packages media organization was implemented, and it was actually that change that complicated things.  Before the change, rsyncing for small mirrors was really easy, as you could just mirror the devel tree, or if you wanted, you could download the official/2006.0 tree once, and then not have to run rsync over that large tree again, you would only need to keep running it over official/updates/2006.0.  You could run one rsync command per release that you wanted to mirror, and it required *no* exclude rules in your rsync commands, it was really simple!

The change to the new build system and packages media organization was what complicated things.  Now you need a list of rsync exclude at the least to exclude the debug, backports, and testing media.  What I'm asking for won't bring back the simplicity of mirroring before 2007.0, but it will at least make things a little better.

I'm still not 100% sure what you really want. Remove packages from release when there is newer version in the updates or what?

Or what do you want to hardlink then? You need to give better example.

Thanks for the comment Sander.  Let me try to clarify this.

With what I'm asking for, the contents under release/ and updates/ could be left as they are now.  What I'm asking for would add one more directory next to those two, called updated/.  Upon initial release of a distro version, its contents would initially look the same as release/.  Then, kind of like you were thinking in your question, every time a new version of a package is released, it is placed under updated/ and the older version is removed.  So what you would see under updated/ is every package in the distro, and you would only see one version of each package, the newest one.

As far as the hard links, the purpose of this is to make it so that adding this medium doesn't require any additional disk space to implement on the upstream and public mirrors.  The package files under updated/ would actually be hardlinks to wherever they are also stored, be it ../release/ (if they have not had any updates issued yet) or ../updates/ (if they have been updated).  This means that each package would only be physically stored once, even though they would appear under updated/ as well as either release/ or updates/.  The only additional disk space required on the main mirrors would be for updated/media_info/ which is really small.

What this would allow for, is a small organizational mirror to exclude release/ and updates/ from their mirroring, essentially only mirroring the updated/ subdirectory of each medium.  Installations could be done from such a mirror, where it would pull all of the packages from one place for each medium (instead of two if you install from release/ and updates/), and the storage required by the mirror would be less as it would not have to store multiple versions of packages that have been updated.  I hope this clarifies it, but don't hesitate to ask if you have any more questions.

As already said, I think it is too complicate to manage, confusing for users, and could be done locally with a script if really wanted. So I don't think we should do this.

As I said, I disagree.  It wasn't too complicated to manage when it was done before with MDV 2006.0.  It is not confusing for users, in fact *users* won't be affected by it in the least.  Only people who run mirrors will care, and that ones that don't care can safely ignore it.  I'm sure you understand that it would be significantly easier to do it upstream than to do it locally with a script.  Again, it would be really nice for some of us out here in the real world.  Nobody is forcing you to implement it, it would just be really nice, and if it could be done that would be really great.  The objections you've given to this are not reasonable.  As I said, if you just don't want to implement it, that's unfortunate, but that's your prerogative.

(In reply to comment #8)
> It would take some work to do it, but it would only need to be done once and it
> would be done.  It would take zero work to maintain and add no complexity to
> anything.  You're only assuming that nobody would use it.

youri would need to be updated to copy the new packages to two repositories, with the same files hardlinked. That's adding complexity. If we need to remove an updated package, we need to remove it from one repository, remove from the other and copy again the old version hardlinked. And what happens if there are inconsistencies between the two repositories ?

And if someone forget the -H rsync option ? They will download everything twice ...

I think the 2006.0 mirror was too messy so that's not really a good idea. And in fact, we tried to reduce the number of repository, so adding more repo is kinda defeating the purpose. 

And besides adding complexity, that's just adding I/Os to the server. Generating hdlist is currently a lengthy process ( until we switch to a better tool ), and for now, the tradeoff is IMHO not worth until there is lots of users asking for that. 

Now, a better solution would be to have a smarter synchronisation tool. For example, a script that would sync release as a first step, and then sync update with --no-delete on it, and remove old rpms and generate hdlist on the server.

This would satisfy everyone, the only downside is that we shift the complexity from our buildsystem to the mirroring script. And I thnk we could distribute this script in the distro ( or on the mirror ).

CC: (none) => misc

Hello Michael,

(In reply to comment #18)
> I think the 2006.0 mirror was too messy so that's not really a good idea.

I'd be interested in you explaining this further.  Really the 2006.0 mirror was a lot cleaner.  Each different repository was in its own tree.  After the change, now you've got multiple different repositories under the same tree which makes it more complicated from a mirroring standpoint.  It used to be you could simply rsync on the repositories you wanted.  Now you have to use an excludes list to leave out the ones that you don't.

Also, you may recall the rpmsync script that I wrote (still in MDV contrib).  It's really simple when you run it on one repository, and for packages that are updated, the old one is deleted and the new one is added, and in the output from the dryrun, it can easily see that.  Then it renames the old one to the new one before running the real rsync.  The amount of savings you get from this varies, but it's not insignificant.  So with the current mirror organization rpmsync still works, but at the least when something's updated for the first time you don't get that savings because nothing's deleted.  Another things that's nice about rpmsync is it's a simple drop-in replacement for rsync and doesn't have any repository-specific logic in it.

So from a mirroring standpoint, it would actually greatly simplify things to make it like 2006.0 and turn the organization on its head.  Under official/, the top level directories would be the things that are now under each medium, i.e. updates, debug, backports, and either release or the distro versions sitting at the top level as are now.  Then each tree would only house one repository.  I'm not suggesting this, just saying it would be ideal for those that are just trying to mirror certain things (unlike public mirrors that just run one huge rsync on mageia/).

> And
> in fact, we tried to reduce the number of repository, so adding more repo is
> kinda defeating the purpose. 

What do you mean?  When did you try to reduce the number of repositories?  Mageia has the same ones as Mandriva as far as I can tell (again that's release, updates, debug, and backports).  It does have one less medium (just core instead of main/contrib) but that's an entirely different thing.

> And besides adding complexity, that's just adding I/Os to the server.
> Generating hdlist is currently a lengthy process ( until we switch to a better
> tool ), and for now, the tradeoff is IMHO not worth until there is lots of
> users asking for that. 

If the added I/O upstream is that much of a problem, that's unfortunate.  Ideally the tool that builds the hdlists could build both of them in one pass rather than requiring a second pass, which would mitigate that problem.  This would be the one thing that would actually add complexity, admittedly.  So if at some point in the future a better tool for generating the metadata is developed, the complexity problem goes away.  Do you have any inclination as to whether it's possible to do better?

> Now, a better solution would be to have a smarter synchronisation tool. For
> example, a script that would sync release as a first step, and then sync update
> with --no-delete on it, and remove old rpms and generate hdlist on the server.
> 
> This would satisfy everyone, the only downside is that we shift the complexity
> from our buildsystem to the mirroring script. And I thnk we could distribute
> this script in the distro ( or on the mirror ).

Yeah, it's possible.  It certainly locks you into one mirroring method, but I would imagine almost everyone uses rsync anyway.  It could possibly lock you into some other choices too depending on how it's designed.  It would also have some complicated repository/mirror-specific logic in it, but it should be do-able.

By reducing repository, I mean that :

On Mandriva : 
- main
- contrib 
- non-free
- restricted ( powerpack )
- plf-free
- plf-nonfree
(  + debug )

( yes, plf is not part of mandriva but that's the basis we took for modeling mageia layout )
6 ( main, contrib, etc ) * 4 ( release updates, etc ) * 2 ( debug ) * $nb_ARCH
=> 96

On mageia 
- core
- non-free
- tainted
3 ( main, contrib, etc ) * 5 ( release updates, etc ) * 2 ( debug ) * $nb_ARCH
=> 60

I guess that I was not clear, as you would call core/etc a media, and release/etc a repository. 

And you are right, I was wrong, that the 10.2 that was a little bit messy.

For the hdlist issue, we need to do some test with gendistrib, but I would still be inclined to push complexity on the user side. It was already hard to find a consensus for the mirror layout, and I think we should not change it every 6 months. 

In fact, the issue boil down to the order of the various component of the url : 
- arch
- version
- media ( core, non-free, tainted )
- repository 
and the position of debug 

Basically, we had people that
- wanted to sync only one arch 
- wanting to sync only one version
- wanted to exclude some media ( tainted )
- wanted to exclude some repository ( testing, backport or debug )

 
arch/version must be the upper level, as we also host drakx code, etc.
Then we either have media/repo, or the reverse. And the question is wether people want 
to exclude media, or repo. I think both case are valid, even if a single user would likely prefer excluding repo ( since there is more repo, so more likely something to exclude ).
I think we could not satisfy all of them with simple solution ( and I am not sure that adding symlink on mirror would really be good ).

The problem of having complex regexp and exclude for rsync is valid, so that's why I propose to hide the complexity under a higher level script.

So what could be the features :

- something that only sync the most recent version of a rpm, for a given version/arch distro, based on several repository ( ie, release/update, or potentially release/update/backports )
- something that can remove some repository and some media without more than giving the name ( ie, be smart about this ).

For example,  :
script --version 1 --arch i586  --only-media core,tainted  --only-repository release,updates  --merge updated
  or 
script --version 1 --arch i586 --exclude-media non-free   --exclude-repository debug,testing 

this would greatly reduce the complexity ( and maybe default argument for version and arch ). Now, adding the same idea as rpmsync, and automatically fetching the mirror from the list, this would help people who want to do private mirror, I guess.

Being tied to rsync is IMHO not a big issue.

Michael, thank you for your constructive feedback.

I agree with you that we should change the layout as infrequently as possible.  I've always been disappointed that the current layout was the one that won for 2007.  With things that have been added (for instance I don't think even non-free existed in 2006.0), I suppose no matter what order you use, many people would still have to use rsync excludes for something now.  I still would have liked to have seen repository (testing, debug, backports, etc) be at the highest level, because I think that's what most people are likely to exclude, and mixing different repositories in the same tree causes more problems than anything.  For instance, having different media doesn't really matter as packages move between medias sometimes and media aren't meaningful from a mirroring standpoint, one set of packages is the union of all of the media.  It just gets problematic when you're mixing repositories together and you've got different versions in the same tree.  Actually now that I think of it, putting the repository first in the ordering (well, after arch) would probably be a better solution than the 'updated' repository that I proposed.  That being said, it's a major change that I wouldn't expect to implemented, even as nice as it would be, and it would probably be the best solution overall.

A more advanced mirroring script does sound interesting though, and I'm sure it would be useful.  I agree that being tied to rsync isn't really an issue.  Hopefully I'll have time to try to implement something, now that I'm finally done with grad school.

While the whole discussion is interesting here, is there a need to keep this bug report open or can we close it ? Reading it I'm not sure what it is about in the end so I fear it will stay in its current state forever. But I may be wrong :)

CC: (none) => stormi

Samuel, I started a new job shortly after the discussion ended which is why I didn't have the time to think about this that I thought I would, and the discussion died.  To summarize, I think these are the issues:

There are different media (core, non-free, tainted) and different repositories (release, updates, testing, backports) all under the same root in the mirror layout, so there's a lot of duplication (different versions of the same package in different media like core and tainted as well as in different repositories like release and updates).  For some organizational mirrors, it is desired to exclude some repositories, and possibly some media.  That is not very easy to do with the current mirror layout, but it is possible with rsync excludes.  Michael Scherer suggested making a higher-level mirroring script to make this easier, which was a good idea.

The problem that remains is that even with that, there is still the duplication between release and updates.  On a small mirror where disk space is not unlimited, it would be desirable to be able to mirror such that only the newest updated package is kept, and the older original release version is not, and furthermore, to be done in such a way that it is easy for a client script to know when a package is updated so that it can move the old version to the new version before running rsync (which can save some bandwidth sometimes).  The easiest way to do this is if there is an upstream repository where this is already done, so that an rsync dryrun will see that the older version was removed.  This was done in the past in Mandriva 2006.0 with the devel tree.

In this bug report, I suggested adding a repository called "updated" which would contain hard links to packages in release if they haven't been updated and hard links to packages in updates for those that have (so when a new package is added to updates, the link to the old package in release is removed).  The reason it would better to do this upstream on the mirrors is that when a new package is submitted to updates in the build system, it knows what SRPM is being submitted, what RPMs that is creating, and what RPMs came from the older version of the SRPM that was there before.  Having access too all of that metadata, it makes it a lot easier for the build system to manage this than a client script that only has package filenames to go on, which isn't foolproof.  So it would still be nice to address this upstream.

Some concern was given about how long it takes genhdlist to run for each repository, as adding another repository would require another run of this tool and increase the time needed to generate all of the hdlists.  It was mentioned that it is possible that the genhdlist tool could be improved so that it could make the hdlists for the different repositories in one run without having to make multiple passes over the same packages.

So there are still some things that I could do and some things that could be done upstream, which is why this is still open.

ping?

CC: (none) => marja11

(In reply to comment #23)
> 
> In this bug report, I suggested adding a repository called "updated" which
> would contain hard links to packages in release if they haven't been updated
> and hard links to packages in updates for those that have (so when a new
> package is added to updates, the link to the old package in release is
> removed).  The reason it would better to do this upstream on the mirrors is
> that when a new package is submitted to updates in the build system, it knows
> what SRPM is being submitted, what RPMs that is creating, and what RPMs came
> from the older version of the SRPM that was there before.  Having access too
> all of that metadata, it makes it a lot easier for the build system to manage
> this than a client script that only has package filenames to go on, which isn't
> foolproof.  So it would still be nice to address this upstream.

As already said, it's possible to get the information about which SRPM is creating which RPM from hdlists.

So I don't see any reason why it couldn't be done by a script on the client.

Sorry I've been unavailable and really busy, so I haven't been able to address the high-level script idea, but it's on my TODO list.  As for the upstream part of it, it really would be much better off being addressed upstream and would be a lot simpler.  I'm not convinced it really could conceivably be done client-side, but if so it'd be extraordinarily complicated.

All the needed infos and metadatas are available, so I don't see any reason why it would be more difficult to do it outside the build system.

If not, then explain what is missing, and maybe we can provide it.

Well, doing it upstream would be as simple as generating a hard-link in the updated repository and removing hard-links to the previous version when a new packages goes into updates.

I'm trying to imagine what it would take to do it client-side.  I guess it would be something like this.  First sync the media_info directory for each media's updates repository, then generate a list of SRPM names that generated their contents, then look at the media_info from the release repositories and generate a list of RPMs generated from those same SRPMS, move them into the updates repositories, then sync the media excluding everything but the updates repositories, and then sync the root of the distro version tree excluding everything under media except for media_info.  That sounds extremely complicated, cumbersome, inefficient, and fragile to me.  That's why it would be SO much nicer to have this addressed in a simple, stable manner upstream.

(In reply to comment #28)
> Well, doing it upstream would be as simple as generating a hard-link in the
> updated repository and removing hard-links to the previous version when a new
> packages goes into updates.

Well, we could also say the same about doing it outside the build system ...

> I'm trying to imagine what it would take to do it client-side.  I guess it
> would be something like this.  First sync the media_info directory for each
> media's updates repository, then generate a list of SRPM names that generated
> their contents, then look at the media_info from the release repositories and
> generate a list of RPMs generated from those same SRPMS, move them into the
> updates repositories, then sync the media excluding everything but the updates
> repositories, and then sync the root of the distro version tree excluding
> everything under media except for media_info.  That sounds extremely
> complicated, cumbersome, inefficient, and fragile to me.  That's why it would
> be SO much nicer to have this addressed in a simple, stable manner upstream.

Something similar would need to be done on build system.

(In reply to comment #29)
> (In reply to comment #28)
> > Well, doing it upstream would be as simple as generating a hard-link in the
> > updated repository and removing hard-links to the previous version when a new
> > packages goes into updates.
> 
> Well, we could also say the same about doing it outside the build system ...

With the information the build system automatically has access to that something outside it would not, it would not be as simple to do it outside.

> > I'm trying to imagine what it would take to do it client-side.  I guess it
> > would be something like this.  First sync the media_info directory for each
> > media's updates repository, then generate a list of SRPM names that generated
> > their contents, then look at the media_info from the release repositories and
> > generate a list of RPMs generated from those same SRPMS, move them into the
> > updates repositories, then sync the media excluding everything but the updates
> > repositories, and then sync the root of the distro version tree excluding
> > everything under media except for media_info.  That sounds extremely
> > complicated, cumbersome, inefficient, and fragile to me.  That's why it would
> > be SO much nicer to have this addressed in a simple, stable manner upstream.
> 
> Something similar would need to be done on build system.

No, doing in the build system wouldn't require any of that complication.  Let me reiterate that this has also already been done before (MDK 2006.0).  Also, with some updates it's partially already done by removing the previous version from updates when adding a newer update.

(In reply to comment #30)
> (In reply to comment #29)
> > (In reply to comment #28)
> > > Well, doing it upstream would be as simple as generating a hard-link in the
> > > updated repository and removing hard-links to the previous version when a new
> > > packages goes into updates.
> > 
> > Well, we could also say the same about doing it outside the build system ...
> 
> With the information the build system automatically has access to that
> something outside it would not, it would not be as simple to do it outside.

Then which information is available to the build system and not outside ?

First of all the build system knows when a new package is being built, because it's building it.  It knows explicitly which SRPM is being built.  It knows what RPMs are generated by the result of it.  It is at the time that the new package is built that things need to happen and can be done (links generated and removed).  Outside of the build system all that is immediately visible is the filenames that exist on the mirrors at one snapshot in time, with no deeper information or context.  Sure media_info information is there, but it has to be downloaded and parsed.  It still won't know what's changed since the last time it looked.  This is just one of those things that makes a LOT more sense to do in a "push" fashion than a "pull" fashion.  That should be pretty obvious.

You can easily get the list of new packages by comparing the list of files on mirrors last time the script was run and now. So you can easily download all new packages that were added.

You can easily see which SRPM built which RPM by looking at packages headers, or using hdlists. rpm -qp --qf '%{SOURCERPM}' file.rpm will tell you which SRPM was used to build the package.

So it should be quite easy to do :
- have a script to download all new packages added on updates repository and save them in the same directory
- have a script to keep only the latest SRPM of each package and corresponding binary packages
- generate hdlists

Nothing really complexe. It is some work to write this script, but it would also require some work to do it on build system. And it cannot be done on primary mirror for reasons already explained.

Our definitions of easily certainly differ, but yes the procedure would be a combination of what you described and what I said in Comment 28.  I stand by what I said about it being complicated, cumbersome, inefficient, and fragile.

Let me also reiterate the gain from this.  Currently 1 core/relase is 15G and 1 core/updates is 5.1G.  Considering most of the packages in updates are also in release and so the release copies are redundant, that's probably around a 20% reduction in space you could get from not having them, which is significant.

The only reasonable objection to doing it on primary mirror that's been given (besides actually doing the work to implement it) is the extra time to generate one set of hdlists, but a solution has been proposed that would solve that as well (again, assuming it was implemented).  It is really the only reasonable way to do this.

(In reply to comment #34)
> Our definitions of easily certainly differ, but yes the procedure would be a
> combination of what you described and what I said in Comment 28.  I stand by
> what I said about it being complicated, cumbersome, inefficient, and fragile.

We could also say that it would be complicated, cumbersome, inefficient, and fragile if done on the build system. So let's keep this outside the build system as most people don't need this, and it can be done outside as easily.

> 
> Let me also reiterate the gain from this.  Currently 1 core/relase is 15G and 1
> core/updates is 5.1G.  Considering most of the packages in updates are also in
> release and so the release copies are redundant, that's probably around a 20%
> reduction in space you could get from not having them, which is significant.

You're free to implement the script to make your own mirror.

> 
> The only reasonable objection to doing it on primary mirror that's been given
> (besides actually doing the work to implement it) is the extra time to generate
> one set of hdlists, but a solution has been proposed that would solve that as
> well (again, assuming it was implemented).  It is really the only reasonable
> way to do this.

No solution has been provided to avoid generating more hdlists. I don't think it would be simple to allow genhdlists to generate multiple hdlists at the same time, or that it would be efficient, and have seen no patch proposed to do that.

And the other objections given are also reasonable :
 - more complexity on build system, which is already not very simple
 - more complexity for users who will not know what is this new repository
 - more bandwidth used on all other mirrors because they will have to download those new hdlists. Looking at the size of cauldron hdlists, it would add around 300MB of hdlists to download each time the mirror is updated, for one distribution.
 - it can be done as easily outside build system
 - there is already a lot of things on TODO list

So I think we can close as WONTFIX, at least for now.

Status: NEW => RESOLVED
Resolution: (none) => WONTFIX