Bug 12995 - Segmentation Fault with apache-mod_php after httpd reload when opcache is enabled
Summary: Segmentation Fault with apache-mod_php after httpd reload when opcache is ena...
Status: RESOLVED FIXED
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: Cauldron
Hardware: All Linux
Priority: Normal critical
Target Milestone: ---
Assignee: Oden Eriksson
QA Contact:
URL:
Whiteboard: MGA4TOO MGA5TOO
Keywords:
: 13233 (view as bug list)
Depends on:
Blocks:
 
Reported: 2014-03-11 17:22 CET by Marc Krämer
Modified: 2015-10-09 22:07 CEST (History)
2 users (show)

See Also:
Source RPM: php-5.5.8-2.mga4.src.rpm
CVE:
Status comment:


Attachments
trace of apache with segfault (107.07 KB, application/x-gzip)
2014-04-01 02:37 CEST, Marc Krämer
Details

Description Marc Krämer 2014-03-11 17:22:06 CET
With the latest packages of apache, maybe mod_php it happens quite frequent, that the server crashes with a segementation fault in log:

 [core:notice] [pid 31937] AH00052: child pid 7665 exit signal Segmentation fault (11)


Sth. is special about this:
1. if you log in to that server, you can see apache is still "running":
ps aux|grep apa
apache    7010  0.1  2.3 554392 94784 ?        S    16:48   0:02 /usr/sbin/httpd -DFOREGROUND
apache    7668  0.0  0.2 472912 10708 ?        S    17:12   0:00 /usr/sbin/httpd -DFOREGROUND
apache    7669  0.0  0.2 472912 10708 ?        S    17:12   0:00 /usr/sbin/httpd -DFOREGROUND
apache    7670  0.0  0.2 472912 10708 ?        S    17:12   0:00 /usr/sbin/httpd -DFOREGROUND
apache    7718  0.0  0.2 472912 10708 ?        S    17:12   0:00 /usr/sbin/httpd -DFOREGROUND
root      7722  0.0  0.0  14608   944 pts/2    S+   17:12   0:00 grep --color apa


but every request to the server results in an seg-fault.

2. systemctl stop httpd-prefork.service didn't kill all instances:
ps aux|grep htt
apache    7773  0.0  0.2 175700 10272 ?        S    17:13   0:00 /usr/sbin/httpd -DFOREGROUND
apache    7776  0.0  0.2 472912 10996 ?        S    17:13   0:00 /usr/sbin/httpd -DFOREGROUND
root      7784  0.0  0.0  14608   944 pts/2    S+   17:13   0:00 grep --color htt


I should mention, that opcache.enable was loaded & enabled, what I did change now.

Reproducible: 

Steps to Reproduce:
Comment 1 Marc Krämer 2014-03-11 20:34:18 CET
disabling opcache cache didn't help.

maybe it comes some request after I did a reload to make some changes known to apache
David Walser 2014-03-12 15:24:20 CET

Assignee: bugsquad => guillomovitch

Comment 2 Marc Krämer 2014-03-16 09:25:09 CET
verified:
a simple systemctl reload httpd will cause a segmentation fault by child processes and the server is unreachable. It looks like all ne spawned processes have the same problem.

This problem is new in mageia 4. apache in mageia 3 was very reliable.
Comment 3 Guillaume Rousse 2014-03-19 20:35:35 CET
Without a reproductible test case, it will be quite difficult to investigate this issue.

Status: NEW => UNCONFIRMED
Ever confirmed: 1 => 0

Comment 4 Marc Krämer 2014-03-19 23:04:18 CET
you're right. Since I can't reproduce this on a simple clean install of apache on my local machine. I did a few more tests by copying the settings from the server, but it still works here.

I'll do a few more tests, hope I find it until sunday (logrotate), cause it gets annoying getting called early in the morning.
Comment 5 Marc Krämer 2014-03-23 21:26:15 CET
I found one issue which causes my server instance to fail (but not on my local machine).

running /usr/sbin/httpd -k graceful (as it is stated with $OPTIONS in httpd.service)
causes my apache to produce the named Segmentation faults wheras
apachectl graceful
works fine!


Btw. I would suggest to add a config check before reload/restart with
apachectl -t

This would help to decrease downtime due to configuration failure on reload.
Comment 6 Marc Krämer 2014-04-01 02:33:41 CEST
found: by using strace on apache I found the segemenation fault lies in memory region of php in extension /usr/lib64/php/extensions/filter.so. So this bug is filed against apache with mod_php activated.
Deactivating this extens prevents segemenation faults, but since this extension is needed for almost all webapplications this was just a prove.

As you can see on the attached strace, reload (SIGHUP) occurs at around line 18135. First segmentation fault on line 22152 at adress 0x7fad02dcbd40, which is within region of filter.so (see line 17658)

If you tell me how to make apache write core files, I can give you an core file, if this makes it easier. But I think having the startadress and the segmentation location this must be enough to find the reason for this crash.
Comment 7 Marc Krämer 2014-04-01 02:37:43 CEST
Created attachment 5086 [details]
trace of apache with segfault
Marc Krämer 2014-04-01 02:38:35 CEST

Source RPM: apache-2.4.7-5.mga4.src.rpm => php-5.5.8-2.mga4.src.rpm

Comment 8 David Walser 2014-04-04 01:54:00 CEST
Unfortunately apache in mga4 refuses to allow itself to be debugged in any way other than strace (won't dump core, won't give a backtrace in gdb).

Does the PHP update candidate in Bug 12842 help at all?

CC: (none) => guillomovitch, luigiwalser
Assignee: guillomovitch => oe

Comment 9 David Walser 2014-04-18 19:44:39 CEST
Even with the newest PHP update for Mageia 4, I can reproduce this problem on my Moodle server.  For some reason, it isn't reproducible with my MediaWiki server.  Perhaps it's because the Moodle server actually has clients actively connected to it.

Essentially, after systemctl reload httpd.service, any attempts to load a PHP page causes the child httpd process handling the connection to segfault.  Doing a systemctl restart httpd.service fixes the problem.
David Walser 2014-04-18 19:44:53 CEST

Status: UNCONFIRMED => NEW
Ever confirmed: 0 => 1

David Walser 2014-04-18 19:45:04 CEST

Hardware: x86_64 => All
Severity: major => critical

Comment 10 David Walser 2014-04-18 19:45:31 CEST
*** Bug 13233 has been marked as a duplicate of this bug. ***
Comment 11 Marc Krämer 2014-04-18 20:26:26 CEST
since this problem was to serious for me, I moved to php-fpm.

For me this looks like a pointer to a shared value is not updated (reset) after reload. This might be caused by the new opcache (just a guess).
Comment 12 David Walser 2014-04-18 20:43:24 CEST
Yes, it very well could be related to opcache, as I am using that on my Moodle server, but not on my MediaWiki server.

Summary: Segementation Fault on apache, systemd didn't kill all instances => Segmentation Fault with apache-mod_php after httpd reload, systemd didn't kill all instances

Comment 13 David Walser 2014-04-18 20:49:58 CEST
Indeed, opcache is the culprit.  I installed it on my MediaWiki server and enabled it in php.ini, and then the issue becomes reproducible there.

Summary: Segmentation Fault with apache-mod_php after httpd reload, systemd didn't kill all instances => Segmentation Fault with apache-mod_php after httpd reload when opcache is enabled

Guillaume Rousse 2014-05-07 09:37:50 CEST

CC: guillomovitch => (none)

David Walser 2015-01-12 13:43:16 CET

Version: 4 => Cauldron
Whiteboard: (none) => MGA4TOO

Comment 14 Morgan Leijström 2015-04-22 08:43:59 CEST
This was last updated a year ago.  Maybe it is OK now?
What is the upstream bug report?

CC: (none) => fri

Comment 15 David Walser 2015-04-22 13:44:47 CEST
Interesting, maybe it was fixed.  I can't reproduce it now (mga4 or mga5).  To be sure, I'll see what happens when the current php update (Bug 15721) gets released and installed on my Moodle server at work (with php-opcache now installed again).  If it doesn't cause an issue, I'll remove the conflicts in mga5.
Comment 16 Morgan Leijström 2015-04-22 14:58:07 CEST
IMO the (bogus)conflicts Bug 15743 is a separate issue and should be resolved regardless.
Comment 17 David Walser 2015-04-27 14:32:29 CEST
Saw this problem once the PHP 5.5.24 update was installed on our Moodle server, so at least in PHP 5.5.x it's still not fixed.
Comment 18 Morgan Leijström 2015-04-28 10:17:52 CEST
It seems Owncloud was patched in an attempt not to trig this (or similar) bug in opcache https://github.com/owncloud/core/issues/9885
Comment 19 Marc Krämer 2015-04-28 11:27:47 CEST
is this bug concerning apache + php with opcache known on upstream?
Fixing a script as a workaround won't last long - and since not all interpreted scripts can be patched, this is a really worse work-around.

Apache_mod_php has some drawbacks and only works single threaded in apache.
I think I've read it is advised to use php as (f)cgi - so maybe it is better to give the people an easier way to migrate to fpm-php instead of mod_php.
Comment 20 Morgan Leijström 2015-04-28 16:28:34 CEST
I do not know much about this, i just try to optimise my simple owncloud setup...
I asked now in that owncloud issue about upstream.

Web search turns up several use cases where workaround is to disable opcache.

Several issues like this described in zend-opcache bug tracker:
https://github.com/zendtech/ZendOptimizerPlus/issues

Changelog: https://pecl.php.net/package-changelog.php?package=ZendOpcache
- What version do we have?

For a real workaround today, what caching are recommended on Mageia 5 for PHP data and opcode?
Before, APC was used to cache both, but it is deprecated, right?
Many places i have seen recommendations to use zend opcache for opcode caching and APCu for data caching... removing opcache there need to be something else...
Or APC is still a good alternative?
Comment 21 David Walser 2015-04-28 16:38:05 CEST
php-apc has been obsoleted by php-apcu in Mageia 5, as php-apc does not work with PHP 5.6.  We still have php-xcache for opcode caching, but I haven't tried it.
Comment 22 Marc Krämer 2015-04-28 18:13:20 CEST
If you don't use the apache module, I recommend using the built-in opcache. Running this with php-fpm in mageia 4 since i wrote this bug report.

Using php-apc & php-xcache with earlier versions of php as an apache module, resulted in lower performance. The only reason (in my opinion) to enable this, is to have an extra data cache. But using memcached is more flexible (can be combined with mysql - maybe soon mariadb too) than using a special php-cache.
Comment 23 Morgan Leijström 2015-04-28 18:56:03 CEST
OK thanks.  I try to keep things simple. Memcached seem not to be that.

On the Owncloud trigged issue, AdamW posted https://bugs.php.net/bug.php?id=67687
Comment 24 David Walser 2015-04-28 18:58:37 CEST
I'm of the same mind that memcached is too complicated.  As for AdamW's issue, that's unrelated to this issue.
Comment 25 Morgan Leijström 2015-04-28 19:03:00 CEST
Ok.  Seem safest to avoid zend opcache anyway.
Going to try, but why do php-apcu and php-xcache conflict?
Comment 26 David Walser 2015-04-28 19:11:39 CEST
Technically they don't, but it's for the same reason opcache does.  xcache conflicts with php-apc (which was appropriate, of course), which is now provided by php-apcu.  So the conflicts will have to be dropped from php-xcache as well.
Comment 27 Morgan Leijström 2015-04-28 19:13:49 CEST
So I should post a bug on mageia php-xcache package?
Comment 28 David Walser 2015-04-28 19:16:04 CEST
(In reply to Morgan Leijström from comment #27)
> So I should post a bug on mageia php-xcache package?

Nope, I already fixed it in SVN and sent a freeze push request.

As for php-opcache conflicting with php-apc, I'll fix that in the next PHP update, which may be after Mageia 5 is released.  Due to the bug(s) in php-opcache, fixing it immediately is not a priority.
Comment 29 Morgan Leijström 2015-04-28 19:32:41 CEST
Great, thanks David :)
Samuel Verschelde 2015-06-02 10:34:43 CEST

Whiteboard: MGA4TOO => MGA4TOO MGA5TOO

Comment 30 David Walser 2015-10-09 22:07:14 CEST
The PHP 5.6.14 update we just released confirmed for me that this issue no longer exists.  Marking as FIXED.

Status: NEW => RESOLVED
Resolution: (none) => FIXED


Note You need to log in before you can comment on or make changes to this bug.