Created attachment 110 [details]
Patch to apply the proposed changes

You should see with guillomovitch

CC: (none) => misc

afaik --mirrorlist isn't also on urpmi.addmedia

CC: (none) => maarten.vanraes

The two first proposal are OK. The last one also, excepted for the use of /etc/sysconfig/bash-completion, because it is part of bash-completion package, whereas urpmi completion is handled in urpmi package.

BTW, instead of making package completion optional because it is slow, a better idea would be to make it faster. The support of /var/lib/urpmi/names.* files in urpmi was specifically done for making bash completion able to use grep instead of running urpmq --list, but the growing complexity of urpmi.cfg made it impossible to parse from shell code. Today, adding a small dedicated perl completion helper able to parse this configuration, and then parsing those names.* files would probably be an option. Otherwise, support for those useless files could get dropped from urpmi.

CC: (none) => guillomovitch

I just commited a new version of completion script in subversion, taking advantage of precomputed package names files, instead of urpmq --list. It is lightning fast, but they are two potential issues:
- urpmi media with spaces in their names (will likely get ignored)
- packages with regexp meta-caracters in their name

Please test and report.

Assignee: thierry.vignaud => guillomovitch

one bug https://bugs.mageia.org/show_bug.cgi?id=4937

*** Bug 4940 has been marked as a duplicate of this bug. ***

CC: (none) => mageia

Here is a fix: improves it by a factor of 10.

The bash-completion function:  "_urpmi_packages()"
is what is slow (8 sec), whereas the underlying "urpmq --list" is actually fast (0.15 sec).

Here is what I think is happening, using "apache" as an example:

1. The underlying  urpmq --list  is fundamentally fast (not perfect, but good):
   urpmq --list | grep apache        #takes about 0.6 sec


2. The compgen function is terrible. The essence of _urpmi_packages() is:
    compgen -W "$(urpmq --list)" --  apache     #takes 8.6 sec.


In comparison:
  compgen -W "$(urpmq --list | grep apache)" --    #takes 0.65 sec



So, I suggest changing the last line of _urpmi_packages.

#1    COMPREPLY=( $( compgen -W "$(urpmq $options --list)" -- $cur ) )
#2    COMPREPLY=( $( compgen -W "$(urpmq $options -y $cur )" -- ) )
#3    COMPREPLY=( $( compgen -W "$(urpmq $options --list | grep ^$cur )" -- ) )


#1 is what is currently there, and it's really slow. (8.6 s)
#2 uses urpmq's regex searching. It's slightly slower (0.75 s) than grep 
#3 is what I recommend. It's the fastest (0.65 s), and also only matches packages beginning with "apache", rather than containing "apache"



P.S. There might be a case for filing an upstream bug on "compgen": it shouldn't be this slow!

The original point of moving from old grep-based filtering to compgen-based filtering in every completions (not just urpmi) was to avoid side-issues due to potential meta-characters in the completed pattern. AFAIK, there is no way to have grep use '^' to anchor the pattern first, then ignore every other following metacharacters :/

However, it is clearly overkill here, I fully agree. Did you test the new completion commited in svn ? It is also grep-based, but using pre-computed package name files instead of urpmq --list. I'd be interested to have some benchmark value for this solution too.

Status: NEW => ASSIGNED

Sorry, I've not seen anything in svn; I'd (wrongly) assumed that everything would be in Cauldron as soon as it's ready. Can you point me in the right direction.

Incidentally, the main problem seems to be compgen -W. 

Compare:
  compgen -W "`seq 1 50000 | grep 1794`"     #fast:    0.019 s
  compgen -W "`seq 1 50000`" 1794            #slooow:  3.8 s



Lastly, why not first escape all the metachars in the string, then prepend the '^' ?  eg:

X='hel$l*o'
Y=^$(echo $X | sed 's/[][()\.^$?*+]/\\&/g')

> P.S. There might be a case for filing an upstream bug on "compgen": it
> shouldn't be this slow!


I just did that. Compgen is O(n^2) for the number of options. This makes it really slow as n exceeds 50000.


for n=50k:    (using 1794 as an arbitrary choice)

time compgen -W "`seq 1 50000`" 1794    #3.8 sec
time compgen -W "`seq 1 50000` | grep 1794"    #0.02 sec


for n=500k:

time compgen -W "`seq 1 500000`" 1794    #7 minutes, 16 sec  (!!!)
time compgen -W "`seq 1 500000` | grep 1794"    #0.2 sec


Given that urpmq --list produces 43031 options, we have to filter before the compgen...

You can browse the svn repository from http://svnweb.mageia.org/soft/rpm/urpmi/trunk

Anyway, my own implementation was:
# get active media list
_urpmi_get_medias $options

for media in $medias; do
    COMPREPLY+=( $( compgen -W \
	    "$(grep ^$cur /var/lib/urpmi/names.$media 2>/dev/null)" ) )
done

Which means:
- one call to perl for urpmq --list-media
- one call to grep for every active media

I doubt I can compete for speed with yours, which seems far more robust anyway. I just commited it. And it's time to kill support for this /var/lib/urpmi/names.* files hack.

Regarding the metacharacter, I think we're a bit distinct from general case, as most of these are prohibited in package names anyway, and the remaining ones ('+', for instance) are not metacharacter in basic regular expression dialect used by grep. Let's wait for actual problems before using this, hmmm, barely readable code.

Fix released in urpmi 6.48.

Status: ASSIGNED => RESOLVED
Resolution: (none) => FIXED