Bug 1113 - glibc / libc-2.12.1.so causes sshd and httpd to segfault
Summary: glibc / libc-2.12.1.so causes sshd and httpd to segfault
Status: RESOLVED FIXED
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: 1
Hardware: All Linux
Priority: High critical
Target Milestone: ---
Assignee: Mageia Bug Squad
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-05-03 12:22 CEST by DariuszSki
Modified: 2011-06-18 15:55 CEST (History)
3 users (show)

See Also:
Source RPM: glibc
CVE:
Status comment:


Attachments

Description DariuszSki 2011-05-03 12:22:25 CEST
Description of problem:
Running the Beta of Mageia, I can use console to SSH another machine on internal / internet, but if I SSH to the Mageia machine I get a "connection refused" message, log file shows a .so segfaults.

Likewise, if I use SFTP to another machine internally / internet, there is no problem, but if I try to SFTP to the Mageia machine, the connection is refused, same .so segfaults.

From syslog:
SSH using console.
03/05/2011 10:39:38	localhost	kernel	sshd[25991]: segfault at bf711864 ip b73009dc sp bf711854 error 6 in libc-2.12.1.so[b7248000+15f000]
03/05/2011 10:40:02	localhost	kernel	===>rt_ioctl_giwscan. 7(7) BSS returned, data->length = 888
03/05/2011 10:40:53	localhost	kernel	sshd[26576]: segfault at bf576744 ip b73c49dc sp bf576734 error 6 in libc-2.12.1.so[b730c000+15f000]

SFTP using Filezilla GUI.
03/05/2011 10:49:27	localhost	kernel	sshd[30273]: segfault at bf6fcf34 ip b73db9dc sp bf6fcf24 error 6 in libc-2.12.1.so[b7323000+15f000]
03/05/2011 10:49:33	localhost	kernel	sshd[30278]: segfault at bf6d24c4 ip b74369dc sp bf6d24b4 error 6 in libc-2.12.1.so[b737e000+15f000]
03/05/2011 10:49:38	localhost	kernel	===>rt_ioctl_giwscan. 7(7) BSS returned, data->length = 886


Version-Release number of selected component (if applicable):
libc-2.12.1.so

How reproducible:
Attempt to login to a Mageia machine using shell SSH or GUI SFTP.
Ahmad Samir 2011-05-03 20:41:18 CEST

CC: (none) => arnaud.patard
Assignee: bugsquad => tmb
Source RPM: libc-2.12.1 => kernel

Comment 1 Thomas Backlund 2011-05-03 20:48:07 CEST
I have no problems sshing to my Mageia systems,so:

Is this a clean install or an upgrade ?

Have you checked the hw with memtest ?
Ahmad Samir 2011-05-03 21:30:11 CEST

CC: arnaud.patard => misc
Source RPM: kernel => openssh-server

Comment 2 DariuszSki 2011-05-04 16:31:12 CEST
The install method I tried was the one Windows users may try, update from (working) Mandriva 2010.x to Mageia. Will post separately on test request.

There was no problem in Mandriva 2010.x with SSH / SFTP access, so something has gone wrong in the Mageia package.
Comment 3 Michael Scherer 2011-05-04 16:48:17 CEST
It work fine here too. 

Can you try to get a backtrace for that ?
Comment 4 DariuszSki 2011-05-15 12:14:44 CEST
I have not got backtrace to work, must be doing something wrong. However, I ran the verbose command for accessing the Mageia install, and it comes back with the following......

> ssh usename@192.xxx.xxx.xxx -pxxxxxxx -v
OpenSSH_5.5p1, OpenSSL 1.0.0a 1 Jun 2010
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to 192.xxx.xxx.xxx [192.xxx.xxx.xxx] port xxxxxx.
debug1: Connection established.
debug1: identity file /home/someuser/.ssh/id_rsa type -1
debug1: identity file /home/someuser/.ssh/id_rsa-cert type -1
debug1: identity file /home/someuser/.ssh/id_dsa type 2
debug1: identity file /home/someuser/.ssh/id_dsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.8
debug1: match: OpenSSH_5.8 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.5
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host '[192.xxx.xxx.xxx]:xxxxxx' is known and matches the RSA host key.
debug1: Found key in /home/someuser/.ssh/known_hosts:1
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
Connection closed by 192.xxx.xxx.xxx


Don't know it that helps.
Comment 5 Michael Scherer 2011-05-15 12:21:56 CEST
It doesn't help much. Is this a fresh install ? Can you try to upgrade to latest cauldron, check if there is no problem on rpm side ( with rpm -V ), check hardware ?

Do you have some crypto-hardware acceleration stuff ? What is the server running, vm, real hardware, what processor ?
Comment 6 DariuszSki 2011-05-25 13:26:06 CEST
Slightly different command to last debug test..
/usr/sbin/sshd -d -pxxxxx

debug1: sshd version OpenSSH_5.8p1
debug1: private host key: #0 type 0 RSA1
debug1: read PEM private key done: type RSA
debug1: private host key: #1 type 1 RSA
debug1: read PEM private key done: type DSA
debug1: private host key: #2 type 2 DSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
debug1: rexec_argv[2]='-pxxxxx'
Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port xxxxx on 0.0.0.0.
Server listening on 0.0.0.0 port xxxxx.
debug1: Bind to port xxxxx on ::.
Server listening on :: port xxxxx.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 192.168.xxx.xxx port xxxxx
debug1: Client protocol version 2.0; client software version OpenSSH_5.5
debug1: match: OpenSSH_5.5 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8
debug1: permanently_set_uid: 487/482
debug1: list_hostkey_types: ssh-rsa,ssh-dss
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received
debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT
debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: KEX done
debug1: userauth-request for user xxxxxxx service ssh-connection method none
debug1: attempt 0 failures 0
debug1: do_cleanup
Segmentation fault


I have installed the debug rpm for glibc but don't know how to run it, it is what is causing the last line "seg fault", and stopping me logging into Mageia via SSH.

I installed Mageia on top of Mandriva 2011.1 to test that install method (most likely to be used for ex-Windows users). It is on a netbook, so definitely not in a virtual machine, âIntel(R) Atom(TM) CPU N270.

Mandriva 2011.1 had no problem with SSH into the netbook, something changed in Mageia Beta / pre-release.
Comment 7 DariuszSki 2011-05-25 14:33:28 CEST
After more testing of the Mageia install, looking through the kernel.log file, I have also found that Apache also fails to load due to the same libc-2.12.1.so problem, httpd does not start on boot as it's supposed to.

Summary: Cannot SSH / SFTP as libc-2.12.1.so segfaults => glibc / libc-2.12.1.so causes sshd and httpd to segfault
Source RPM: openssh-server => glibc

Comment 8 Michael Scherer 2011-05-26 10:49:11 CEST
Could you try to use gdb to get the backtrace ?  gdb -p should do the trick , either for apache or sshd.
Comment 9 DariuszSki 2011-05-26 14:04:18 CEST
I hope I've run gdb right, but here is what I got from trying process ID for sshd (as root).

http://pastebin.com/v6X1eNhx
Comment 10 Michael Scherer 2011-05-26 14:29:56 CEST
You also need to type "c" ( shorthand for continue ), once you have the gdb prompt, do something to make it crash, and type "bt" ( for "backtrace" ) once it crashed.  See  http://wiki.mandriva.com/en/Development/Howto/Software_Crash for background information.
Comment 11 DariuszSki 2011-05-26 17:10:41 CEST
After typing "c", tried to SSH into my Mageia install from another machine, eventually below is what got generated.


(gdb) c
Continuing.
Detaching after fork from child process 26688.
bt

                    
Program received signal SIGTERM, Terminated.
0xffffe424 in __kernel_vsyscall ()
(gdb) bt
#0  0xffffe424 in __kernel_vsyscall ()
#1  0xb744faad in ___newselect_nocancel () from /lib/i686/libc.so.6
#2  0x080506f8 in server_accept_loop (ac=1, av=0x0) at sshd.c:1110
#3  main (ac=1, av=0x0) at sshd.c:1792
(gdb) 
#0  0xffffe424 in __kernel_vsyscall ()
#1  0xb744faad in ___newselect_nocancel () from /lib/i686/libc.so.6
#2  0x080506f8 in server_accept_loop (ac=1, av=0x0) at sshd.c:1110
#3  main (ac=1, av=0x0) at sshd.c:1792
(gdb)
Comment 12 DariuszSki 2011-06-08 23:43:32 CEST
Changed the status of this bug as it now affects the released version of Mageia, in both 32bit and 64bit versions.

Is there a fix for this, without ssh and Apache can't do any work.

Priority: Normal => High
Component: Installer => RPM Packages
Hardware: i586 => All
Version: Cauldron => 1
Assignee: tmb => bugsquad

Comment 13 Christiaan Welvaart 2011-06-09 20:28:25 CEST
Is your /etc/hosts corrupt maybe? I googled a bit and found a similar ubuntu bug, but for glibc 2.10.1 . Your gdb log shows a SIGTERM which is not a segfault (that would be called SIGSEGV), so maybe the backtrace is not the right one (it looks too generic, sshd wasn't doing anything there).

CC: (none) => cjw

Comment 14 DariuszSki 2011-06-10 22:04:42 CEST
I also googled but didn't find what you did.. anyway, I tried your suggestion, and unbelievably it worked. It is strange as my hosts file has not been changed in a long time and ran without problem in Mandriva.

I can confirm that the edit / change to the hosts file fixed the problem in both 32bit and 64bit Mageia, both SSH and Apache now start without problem.

Thank you for your finding of this solution. Will make a mental note on SIGTERM and SIGSEGV.

Closing this bug report.

Status: NEW => RESOLVED
Resolution: (none) => FIXED

Comment 15 Michael Scherer 2011-06-14 12:39:48 CEST
Explaining use what you changed in /etc/hosts would help to diagnose the problem in the futur.
Comment 16 DariuszSki 2011-06-18 13:49:33 CEST
The initial problem on my netbook which I filed this bug report for occurred when I upgraded from Mandriva 2011.2 to the beta 1 of Mageia for testing. And on my main machine I copied a hosts backup when I did a clean install of Mageia1. In both circumstances, there were no problems with the hosts file in Mandriva 2011.2.

The hosts file that caused the problem looked something like:

127.0.0.1 machine1
127.0.0.1 localhost
127.0.0.1 site1.mylocal
127.0.0.1 site2.mylocal
127.0.0.1 site3.mylocal
127.0.0.1 www.doubleclick.net ad.doubleclick.net *.paypal.com


I changed it to look like:

127.0.0.1 machine1
127.0.0.1 localhost
127.0.0.1 site1.mylocal
127.0.0.1 site2.mylocal
127.0.0.1 site3.mylocal

ie., removing the last line that I did not want any of the sites access to. This then allowed sshd and httpd to start. Putting that last line back stopped sshd and httpd working. So there was something it did not like about that line, but I can't figure out what. Same hosts file used in my Win7 test setup.


I since changed the hosts (via MCC gui) to look like:

127.0.0.1 machine1
127.0.0.1 localhost
127.0.0.1 machine1 site1.mylocal
127.0.0.1 machine1 site2.mylocal
127.0.0.1 machine1 site3.mylocal

And experience no problems.


The line
127.0.0.1 www.doubleclick.net ad.doubleclick.net *.paypal.com
I moved into the file hosts.deny , this file did not exist in my Mandriva setup, so had no knowledge of its existence.
Comment 17 Sander Lepik 2011-06-18 15:08:50 CEST
You can't use wildcard in /etc/hosts file.

CC: (none) => sander.lepik

Comment 18 DariuszSki 2011-06-18 15:55:14 CEST
I checked the backup hosts, and it had one "*" entry. But importing the problem line without the "*" entry got the same results, sshd and httpd don't start.

Note You need to log in before you can comment on or make changes to this bug.