Bug 373 - add some more completions to urpmi
Summary: add some more completions to urpmi
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: Cauldron
Hardware: All Linux
Priority: Normal enhancement
Target Milestone: ---
Assignee: Guillaume Rousse
QA Contact:
Keywords: Junior_job, PATCH
: 4940 (view as bug list)
Depends on:
Reported: 2011-03-15 00:39 CET by Ahmad Samir
Modified: 2012-03-27 19:48 CEST (History)
6 users (show)

See Also:
Source RPM: urpmi
Status comment:

Patch to apply the proposed changes (1.46 KB, patch)
2011-03-15 00:39 CET, Ahmad Samir
Details | Diff

Description Ahmad Samir 2011-03-15 00:39:04 CET
I propose adding the following options to urpmi bash-completion:
- make --searchmedia use _urpmi_medias to complete media names (already used with --*media* options)
- add --replacefiles to urpmi completion, it's useful with --replacepkgs (which is already used with urpmi)

- make the _urpmi_packages() completion optional (I find it slow...), and leave it enabled by default by adding:
# enable urpmi packages completion

to /etc/sysconfig/bash-completion .
Comment 1 Ahmad Samir 2011-03-15 00:39:39 CET
Created attachment 110 [details]
Patch to apply the proposed changes
Ahmad Samir 2011-03-15 00:40:09 CET

Hardware: i586 => All

Comment 2 Michael Scherer 2011-03-15 00:41:26 CET
You should see with guillomovitch

CC: (none) => misc

Comment 3 AL13N 2011-03-15 00:46:53 CET
afaik --mirrorlist isn't also on urpmi.addmedia

CC: (none) => maarten.vanraes

Marja Van Waes 2011-10-05 21:11:49 CEST

CC: (none) => marja11
Assignee: bugsquad => thierry.vignaud

Thierry Vignaud 2011-10-10 19:20:10 CEST

Keywords: (none) => Junior_job

Thierry Vignaud 2011-12-06 10:34:50 CET

Keywords: (none) => PATCH

Comment 4 Guillaume Rousse 2011-12-06 15:17:25 CET
The two first proposal are OK. The last one also, excepted for the use of /etc/sysconfig/bash-completion, because it is part of bash-completion package, whereas urpmi completion is handled in urpmi package.

BTW, instead of making package completion optional because it is slow, a better idea would be to make it faster. The support of /var/lib/urpmi/names.* files in urpmi was specifically done for making bash completion able to use grep instead of running urpmq --list, but the growing complexity of urpmi.cfg made it impossible to parse from shell code. Today, adding a small dedicated perl completion helper able to parse this configuration, and then parsing those names.* files would probably be an option. Otherwise, support for those useless files could get dropped from urpmi.

CC: (none) => guillomovitch

Dan Joita 2012-03-07 11:18:07 CET

CC: (none) => djmarian4u
Summary: Add some more completions to urpmi => add some more completions to urpmi

Comment 5 Guillaume Rousse 2012-03-07 22:35:54 CET
I just commited a new version of completion script in subversion, taking advantage of precomputed package names files, instead of urpmq --list. It is lightning fast, but they are two potential issues:
- urpmi media with spaces in their names (will likely get ignored)
- packages with regexp meta-caracters in their name

Please test and report.

Assignee: thierry.vignaud => guillomovitch

Comment 6 Manuel Hiebel 2012-03-14 01:27:33 CET
one bug https://bugs.mageia.org/show_bug.cgi?id=4937
Comment 7 Guillaume Rousse 2012-03-14 12:49:39 CET
*** Bug 4940 has been marked as a duplicate of this bug. ***

CC: (none) => mageia

Comment 8 Richard Neill 2012-03-14 16:22:49 CET
Here is a fix: improves it by a factor of 10.

The bash-completion function:  "_urpmi_packages()"
is what is slow (8 sec), whereas the underlying "urpmq --list" is actually fast (0.15 sec).

Here is what I think is happening, using "apache" as an example:

1. The underlying  urpmq --list  is fundamentally fast (not perfect, but good):
   urpmq --list | grep apache        #takes about 0.6 sec

2. The compgen function is terrible. The essence of _urpmi_packages() is:
    compgen -W "$(urpmq --list)" --  apache     #takes 8.6 sec.

In comparison:
  compgen -W "$(urpmq --list | grep apache)" --    #takes 0.65 sec

So, I suggest changing the last line of _urpmi_packages.

#1    COMPREPLY=( $( compgen -W "$(urpmq $options --list)" -- $cur ) )
#2    COMPREPLY=( $( compgen -W "$(urpmq $options -y $cur )" -- ) )
#3    COMPREPLY=( $( compgen -W "$(urpmq $options --list | grep ^$cur )" -- ) )

#1 is what is currently there, and it's really slow. (8.6 s)
#2 uses urpmq's regex searching. It's slightly slower (0.75 s) than grep 
#3 is what I recommend. It's the fastest (0.65 s), and also only matches packages beginning with "apache", rather than containing "apache"

P.S. There might be a case for filing an upstream bug on "compgen": it shouldn't be this slow!
Comment 9 Guillaume Rousse 2012-03-14 16:43:18 CET
The original point of moving from old grep-based filtering to compgen-based filtering in every completions (not just urpmi) was to avoid side-issues due to potential meta-characters in the completed pattern. AFAIK, there is no way to have grep use '^' to anchor the pattern first, then ignore every other following metacharacters :/

However, it is clearly overkill here, I fully agree. Did you test the new completion commited in svn ? It is also grep-based, but using pre-computed package name files instead of urpmq --list. I'd be interested to have some benchmark value for this solution too.


Comment 10 Richard Neill 2012-03-14 16:57:20 CET
Sorry, I've not seen anything in svn; I'd (wrongly) assumed that everything would be in Cauldron as soon as it's ready. Can you point me in the right direction.

Incidentally, the main problem seems to be compgen -W. 

  compgen -W "`seq 1 50000 | grep 1794`"     #fast:    0.019 s
  compgen -W "`seq 1 50000`" 1794            #slooow:  3.8 s

Lastly, why not first escape all the metachars in the string, then prepend the '^' ?  eg:

Y=^$(echo $X | sed 's/[][()\.^$?*+]/\\&/g')
Comment 11 Richard Neill 2012-03-14 18:49:53 CET
> P.S. There might be a case for filing an upstream bug on "compgen": it
> shouldn't be this slow!

I just did that. Compgen is O(n^2) for the number of options. This makes it really slow as n exceeds 50000.

for n=50k:    (using 1794 as an arbitrary choice)

time compgen -W "`seq 1 50000`" 1794    #3.8 sec
time compgen -W "`seq 1 50000` | grep 1794"    #0.02 sec

for n=500k:

time compgen -W "`seq 1 500000`" 1794    #7 minutes, 16 sec  (!!!)
time compgen -W "`seq 1 500000` | grep 1794"    #0.2 sec

Given that urpmq --list produces 43031 options, we have to filter before the compgen...
Comment 12 Guillaume Rousse 2012-03-14 20:09:22 CET
You can browse the svn repository from http://svnweb.mageia.org/soft/rpm/urpmi/trunk

Anyway, my own implementation was:
# get active media list
_urpmi_get_medias $options

for media in $medias; do
    COMPREPLY+=( $( compgen -W \
	    "$(grep ^$cur /var/lib/urpmi/names.$media 2>/dev/null)" ) )

Which means:
- one call to perl for urpmq --list-media
- one call to grep for every active media

I doubt I can compete for speed with yours, which seems far more robust anyway. I just commited it. And it's time to kill support for this /var/lib/urpmi/names.* files hack.

Regarding the metacharacter, I think we're a bit distinct from general case, as most of these are prohibited in package names anyway, and the remaining ones ('+', for instance) are not metacharacter in basic regular expression dialect used by grep. Let's wait for actual problems before using this, hmmm, barely readable code.
Comment 13 Guillaume Rousse 2012-03-27 19:48:47 CEST
Fix released in urpmi 6.48.

Resolution: (none) => FIXED

Note You need to log in before you can comment on or make changes to this bug.