Bug 17706 - wpasupplicant and iwlist report no access points if there are too many of them (we're using deprecated iwlist & wext instead of iw & nl80211)
Summary: wpasupplicant and iwlist report no access points if there are too many of the...
Status: NEW
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: Cauldron
Hardware: x86_64 Linux
Priority: Normal major
Target Milestone: ---
Assignee: Mageia tools maintainers
QA Contact:
URL:
Whiteboard: MGA8TOO, MGA9TOO
Keywords:
Depends on: 32130 32422
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-07 03:39 CET by w unruh
Modified: 2023-10-22 13:05 CEST (History)
7 users (show)

See Also:
Source RPM: drakx-net-2.58-1.mga9
CVE:
Status comment:


Attachments
iwlist s output (155 bytes, text/plain)
2016-02-09 20:18 CET, w unruh
Details
iw dev wlp3s0 scan (508.05 KB, text/plain)
2016-02-09 20:21 CET, w unruh
Details

Description w unruh 2016-02-07 03:39:55 CET
Description of problem:
wpa_supplicant and iwlist report no Access Points if there are too many of them.

Version-Release number of selected component (if applicable):


How reproducible: always


Steps to Reproduce:
At my university in one of the lecture rooms there are more than 90 access points at times. This causes both iwlist ( which is used to report the available access points in net-applet/Network Center,) and wpa_supplicant to report no access points at all. 

On wpa_supplicant if I change to using nl80211  driver instead of WEXT, wpa_supplicant works and can connect (using ifup-eth) However, the Mageia Network Center tools use iwlist to find the available access points. It however reports nothing making it impossible to use network center to find which access points are available for me to connect to . 

Now it seems that the replacement for iwlist and related tools is "ip" 

Can Network Center be changed so as to use ip as well as (instead of) iwlist?

As I say, iwlist does not work in some situations. 
(I think in the class some of the computers of the students have been set up to act as Access Points, leading the huge number reported.)

wext is deprecated now anyway, so Mageia should be switching over to the nl80211 driver anyway.




Reproducible: 

Steps to Reproduce:
Comment 1 Marja Van Waes 2016-02-07 22:32:49 CET
In that lecture room, please run as root (after, of course, changing "wlp5s0" to whatever that should be for your wlan card):

  iwlist wlp5s0 scan > iwlist.txt

and

  iw dev wlp5s0 scan > iw_full.txt

and attach the resulting files to this bug report

Thanks :-)

Keywords: (none) => NEEDINFO
Source RPM: wireless-tools-30-0.pre9.9.mga5, drakx-net-applet-2.17-1.mga5, libdrakx-net-2.17-1.mga5 => wireless-tools-30-0.pre9.9.mga5, drakx-net-2.17-1.mga5
CC: (none) => marja11, thierry.vignaud, tmb

Comment 2 w unruh 2016-02-09 20:18:08 CET
Created attachment 7440 [details]
iwlist s output

This is the output to iwlist scan Note that it sees no wireless AP since "argument list too long"
Comment 3 w unruh 2016-02-09 20:21:50 CET
Created attachment 7441 [details]
iw dev wlp3s0 scan

iw dev wlp3s0 scan output. Note that there are about 250 BSS entries (and .5 MB of data) No wonder iwlist ran out of space.
Comment 4 Marja Van Waes 2016-02-09 21:35:59 CET
Thanks for the attachments :-)

@ Thierry

Is it OK to assign to you, now that blino stopped maintaining drakx-net?

If not, then please reassign to pkg-bugs@ml.mageia.org

Whiteboard: (none) => MGA5TOO
Keywords: NEEDINFO => (none)
Version: 5 => Cauldron
Source RPM: wireless-tools-30-0.pre9.9.mga5, drakx-net-2.17-1.mga5 => drakx-net-2.17-1.mga5
Assignee: bugsquad => thierry.vignaud

Comment 5 w unruh 2016-08-22 18:34:51 CEST
I certainly do not have the perl expertise to take up the maintainance.

Note that using the new nl80211 instead of the wext as the WPA_WIRELESS_DRIVER seems to work much better. I have been using it for the last half year.
Unfortunately network center makes wext as the default when it connects to a new site.
Comment 6 w unruh 2017-06-19 16:53:03 CEST
In  Mageia 6, wext instead of nl80211 is still made the default by Network Center in connecting to a new ESSID. This should be changed.
(the only place I can find wext is in
/usr/lib/libDrakX/network/connection/wireless.pm
Not sure if that is the only place. I donot know perl well enough.
Comment 7 Marja Van Waes 2017-06-19 17:15:09 CEST
Thanks for the reminder.

Apart from needing to drop wext in favour of nl80211, we're still using deprecated iwlist, too and should move to iw instead.

@ Frédéric, Thierry

Should there be separate bug reports for wext => nl80211 and for iwlist => iw ?

CC: (none) => LpSolit
Assignee: thierry.vignaud => mageiatools

Comment 8 w unruh 2017-07-09 19:21:21 CEST
I AFAICS wext is still the default in MGA6 official.
/usr/lib/libDrakX/network/connection/wireless.pm
change wext to nl80211
 It seems like the change from iwlist to iw is a lot more complicated.
The format has changed and they have an explicit warning against screenscraping.

Finally, the whole of ifup-eth needs a bit of a rewrite since wpa_supplicant does not honour the settings and basically picks the strongest source for any listed in /etc/wpa_supplicant.conf. I have rewritten ifup-eth so as to extract only the desired stanza from wpa_supplicant.conf and have wpa_supplicant just use that altered config file. This way network center does actually try to connect to the desired ssid. But this is a different issue.
Comment 9 w unruh 2017-09-23 18:44:44 CEST
Another reminder that this really really needs fixing.
I am stuck in Sao Paulo airport getting that "Argument list too long" error and netcenter does not list any items and iwlist still gives that same error.
Comment 10 Marja Van Waes 2017-10-11 12:05:55 CEST
(In reply to w unruh from comment #9)
> Another reminder that this really really needs fixing.
> I am stuck in Sao Paulo airport getting that "Argument list too long" error
> and netcenter does not list any items and iwlist still gives that same error.

CC'ing inktvis75, who is willing to help with this bug and related stuff.

CC: (none) => inktvis75

Comment 11 Frederik Vos 2017-10-11 12:52:13 CEST
Actually this bug is very complicated to fix (not the problem itself),
and it's depending on many other components.

And maybe we shouldn't fix it but make an important decision:
- make the service file for wpa_supplicant interface dependend, see enhancement request https://bugs.mageia.org/show_bug.cgi?id=21828
- switch the wireless configuration to systemd-networkd, ditching ifup/ifdown scripts and commands, ditching ifplugd
- remove /etc/sysconfig/wpa_supplicant
- use the iw command instead of iwlist

This will fix all of the described problems.
I am willing to help, but I do think that such a major rewrite should be targeted for Mageia 7
Comment 12 Frederik Vos 2017-10-11 12:58:36 CEST
in the mean time, read:
- https://wiki.mageia.org/en/Systemd-networkd
- create the service file as in https://bugs.mageia.org/show_bug.cgi?id=21828
- create a /etc/systemd/network/wireless.network:

[Match]
Name = wl*

[Network]
Description = Wireless Network
DHCP=yes


create/etc/wpa_supplicant/wpa_supplicant-<device name>.conf: 

wpa_passphrase 'ssid' 'passpharase' > /etc/wpa_supplicant/wpa_supplicant-<device>.conf

find out the ssid with the iw command
Marja Van Waes 2017-10-11 13:14:56 CEST

Depends on: (none) => 21828

Frédéric "LpSolit" Buclin 2017-10-11 13:41:15 CEST

CC: LpSolit => (none)

Comment 13 w unruh 2018-06-28 21:51:20 CEST
And another reminder. 
I was in tel Aviv airport and again, no connection possible because there were too many access points. And at the U Edmonton, connection impossible because too many access points (375 according to iw)

Network Center still has advantages over the other possibilities, but more and more those are vitiated by not being able to connect at all.
Comment 14 w unruh 2018-07-12 09:36:46 CEST
Again, at U Edmonton and UBC in large lecture theatres this again does not work.

I do not think going to systemd networkd is the way to go. It seems all this needs is that iw be used in /lib/libDrakX/network/monitor.pm, together with the 
extraction of the equivalent stuff from the output of iw. 
If I know what the SSID is of the network I want to connect to, then I can use the ifup programs to connect to the network without trouble. Ie, wpa_supplicant has no problem with finding the network (using the nl80211 module). It is just the display in network center using iwlist that is the problem. 

It does not seem to me that it should be so much trouble to rewite monitor.pm so that iw is used instead of iwlist (says the person who does not know perl, and thus cannot rewrite it himself).

Note that things like systemd-networkd may be useful for people who have one wireless network they always connect to. For me who travels around the world regularly and has to connect to various and sundry networks ( about 10 different ones in various places around the world in the past month) having the gui so I can see and easily tell it to connect to various networks is absolutely essential. As is the ability, which network center gives and none of the other options really do, to set up all of the configuration options easily.

Whiteboard: MGA5TOO => MGA5TOO MGA6TOO

w unruh 2018-07-12 09:37:56 CEST

Source RPM: drakx-net-2.17-1.mga5 => drakx-net-2.17-1.mga5 drakx-net-2.32.2-1

Comment 15 Thomas Backlund 2018-07-12 10:04:43 CEST
We are not going to systemd-networkd.

If you stop + disable network service, and and enable + start NetworkManager, does that one find your wireless ?

(if you use plasma, iirc you need to install plasma-nm too)
Comment 16 w unruh 2018-09-29 23:21:21 CEST
It seems that one way out of this impass is to start up wpa_supplicant before Network Center begins to as for the access points. Then the wpa_cli option in 
monitor.pm (in /lib/libDrakx/network) will ask wpa_supplicant for the list of access points. Since (we assume) wpa_supplicant is using nl80211 driver, it will deliver the full list of available ssid even if it is huge (eg, at airports). 

As an additional thing, one can then sort the BSSID according to signal strenths, and tell wpa_supplicant to use the strongest BSSID. It is stupid to use a BSSID with 35% strength for a given SSID when one  with 85% strength is available.
Comment 17 Marja Van Waes 2018-09-30 17:27:15 CEST
CC'ing Zézinho, who bravely touched the drakx-net code recently :-)

CC: (none) => lists.jjorge
Whiteboard: MGA5TOO MGA6TOO => MGA6TOO

Comment 18 Marja Van Waes 2018-09-30 17:31:37 CEST
@ w unruh

Can you please test whether Thomas Backlund's suggestion works (see paste below) when you hit this again (to make sure this bug only happens when using drakx-net)

(In reply to Thomas Backlund from comment #15)

> If you stop + disable network service, and and enable + start
> NetworkManager, does that one find your wireless ?
> 
> (if you use plasma, iirc you need to install plasma-nm too)
Marja Van Waes 2018-09-30 17:45:51 CEST

Summary: wpasupplicant and iwlist report no access points if there are too many of them. => wpasupplicant and iwlist report no access points if there are too many of them (we're using deprecated iwlist & wext instead of iw & nl80211)

Comment 19 w unruh 2018-09-30 20:22:48 CEST
The problem is that that just runs Network Manager which I do not like. The other problem is that I need to be in an area with too many access points so that Network Center/iwlist  fails. Most places that is not true fortunately.
Except of course for testing purposes.

If I get a chance and find somewhere convenient to test Network Manager, I will try.
Comment 20 w unruh 2019-05-16 08:56:56 CEST
Once again I have been screwed by this bus . I was in hospital and could not get any list of AP in Network Manager. iwlist s gave the error that there were too many entries. iw gave 330.

This really really is a severe bug. Anywhere where there are a large number af APs
Airports, hospitals (where every thermometer seems to act as an AP), universities,...

If nothing else writing a little program which puts iw scan output into the same form as iwlist output would be a huge step. (unfortunately iw writer's disclaiming any permanace to the output -- no screenscrapping-- does not help. They have had this warning for 5 years now, at which point it starts tolook like incompetence.
Comment 21 Marja Van Waes 2019-05-16 19:25:26 CEST
Could you please, please test what tmb asked in comment #15

> 
> If you stop + disable network service, and and enable + start
> NetworkManager, does that one find your wireless ?
> 
> (if you use plasma, iirc you need to install plasma-nm too)

If you hate NetworkManager too much to test that, then please take a look at our drakx-net code http://gitweb.mageia.org/software/drakx-net/ 

http://gitweb.mageia.org/software/drakx-net/tree/lib/network/connection/wireless.pm#n920 and following read:

    sub wpa_supplicant_get_driver {
        my ($module) = @_;
        $module =~ /^hostap_/ ? "hostap" :
        $module eq "prism54" ? "prism54" :
        $module =~ /^ath_/ ? "madwifi" :
        $module =~ /^at76c50|atmel_/ ? "atmel" :
        "wext";
    }

You can find that file on your system, it is
  
  /usr/lib/libDrakX/network/connection/wireless.pm

You could test what happens if you replace "wext" with "nl80211"

To test using iw instead of iwlist, you could have a look at 
http://gitweb.mageia.org/software/drakx-net/tree/lib/network/monitor.pm#n74 and following lines:

   } else {
        #- else use iwlist
        require network::connection::wireless;
        my ($current_essid, $current_ap) = network::connection::wireless::get_access_point($o_intf);
        if ($o_intf && !$> && !`$::prefix/sbin/ip link show $o_intf up`) {
            system("$::prefix/sbin/ip link set $o_intf up");
        }
        my @list = `$::prefix/sbin/iwlist $o_intf scanning 2>/dev/null`;
        my $net = {};
        my $quality_match = qr/Quality[:=](\S*)/;
        my $eval_quality = sub {
            my ($qual) = @_;
            $qual =~ s!/0+$!/255!; #- prism54 reports quality with division by zero
            $qual =~ m!/! ? eval($qual)*100 : $qual;
        };

On your system, that file is 

   /usr/lib/libDrakX/network/monitor.pm


The first iwlist, in line 75, will be easy to replace, the second one is harder. Just replacing "prefix/sbin/iwlist" with "prefix/sbin/iw dev" won't work, additional changes will be needed (maybe even in different files), but you're likely smarter than me, so don't hesitate to play with the code and report your results
Comment 22 w unruh 2019-07-19 16:08:47 CEST
I still have not had a chance to try NM in a place where there are too many access point for iwlist. (my transits through large airports tend to be fleeting and time constrained, so lots of wireless testing is not in the cards).

I have replaces wext by nl80211 in the /usr/lib/libDrakX/network/connection/wireless.pm
for quite a while and it works fine, except it does not solve the problem of too many access points. 
Monitor has three methods it tries to use to find access points. One is to to use iwlist-- this seems to be the default in that it is used most often. Another is to use wpa_cli to ask wpa_supplicant, but this tends not to produce many of the access points, and needs wpa_supplicant to be running, which it is not when one is trying to select an access point. It really seems that on seems to be required to replace iwlist with iw. However the output of iw is very different from that of iwlist which means that figuring out how to massage the output of iw to the the form that Network Center (NC) requires is highly non-trivial, especially to somone like me who dies not understand the wireless nomenclature very well. (eg, WPA2 is RSN in iw, but I do not know what wpa is, or wep, or how I tell it has no encryption. And while most AP by now use wpa2, enough do not to make my lack of knowledge an impediment.

My other even greater impediment is that I do not know perl, so figuring how to alter a perl program is fraught. 

I keep hoping that there is someone amongst the Mageia volunteer that is both familair with wireless technology, and with perl, that they could rewrite Network Center to use iw, rather than iwlist.
Comment 23 w unruh 2019-11-15 17:25:06 CET
Just a reminder that Network Center is still using the deprecated iwlist, rather than iw to figure out which access points (BSSID/SSID) are available which means if there are too many access points, networking ceases to work and iwlist reports 0 access points available.

monitor.pm needs to be rewritten so as to use iw rather than iwlist to determine the list of APs. (Since the output of iw is very different from that of iwlist, this means it needs someone who is familiar with what Network Center needs, with what the report by iw means in its various accronyms, and with perl.
Comment 24 w unruh 2019-11-24 21:31:21 CET
Re<a href="show_bug.cgi?id=17706#c15">comment #15</a>)  : I finally got to a place with loads of access points (Sao Paulo GRU airport) and network center reported no access points, and Network Manager reported about 12 (actual number about 200 as seen by iw-- I think NM simply reports the strongest
one if more than one. )

Whiteboard: MGA6TOO => MGA7TOO

Comment 25 Morgan Leijström 2022-09-20 08:31:21 CEST
Still the same on mga8?
Any news in testing?

CC: (none) => fri

Comment 26 w unruh 2022-09-20 21:15:21 CEST
I do not know. With the new wpa_supplicant which seems to handle many situations much better than the old one did, and with my using wpa_cli  and sudo to look up the current access point, there is hope that it is better. But I have not gotten yto places where the number of access points is as huge as it is say in the Taiwan airport or other places with many access points (in part of course because I have not traveled much in the past few years. )

It is hard to find places with too many access points.
Comment 27 Daniel Tartavel 2023-02-17 18:08:12 CET
Hi,

Already valid on mageia 8 up to date, on many computers.

CC: (none) => contact

Morgan Leijström 2023-02-17 18:43:35 CET

Whiteboard: MGA7TOO => MGA8TOO

Marja Van Waes 2023-10-21 22:14:07 CEST

Source RPM: drakx-net-2.17-1.mga5 drakx-net-2.32.2-1 => drakx-net-2.58-1.mga9
Whiteboard: MGA8TOO => MGA8TOO, MGA9TOO

Marja Van Waes 2023-10-21 22:15:12 CEST

Depends on: (none) => 32130

Marja Van Waes 2023-10-21 22:24:21 CEST

Depends on: 21828 => (none)

Marja Van Waes 2023-10-22 13:05:16 CEST

Depends on: (none) => 32422


Note You need to log in before you can comment on or make changes to this bug.