Bug 30924

Summary:	httpd fails to restart (randomly) after logrotate at 04:02h
Product:	Mageia	Reporter:	Barry Jackson <zen25000>
Component:	RPM Packages	Assignee:	Thomas Backlund <tmb>
Status:	NEW ---	QA Contact:
Severity:	major
Priority:	Normal	Keywords:	UPSTREAM
Version:	8
Target Milestone:	---
Hardware:	All
OS:	Linux
Whiteboard:
Source RPM:	apache-2.4.54-1.1.mga8.src.rpm	CVE:
Status comment:

Description Barry Jackson 2022-10-02 12:15:55 CEST

Description of problem:
logrotate is set to run daily.
At 04:02h on most days the server restarts normally, like:

[Sat Oct 01 04:02:03.100993 2022] [mpm_prefork:notice] [pid 1411] AH00171: Graceful restart requested, doing restart
[Sat Oct 01 04:02:05.697565 2022] [mpm_prefork:notice] [pid 1411] AH00163: Apache/2.4.54 (Unix) OpenSSL/1.1.1q PHP/8.0.23 mod_perl/2.0.10 Perl/v5.32.1 configured -- resuming normal operations
[Sat Oct 01 04:02:05.697612 2022] [core:notice] [pid 1411] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

However around twice a week on random days it fails to re-start with:

[Sun Oct 02 04:02:02.442675 2022] [mpm_prefork:notice] [pid 1411] AH00171: Graceful restart requested, doing restart
httpd: Syntax error on line 54 of /etc/httpd/conf/httpd.conf: Syntax error on line 1 of /etc/httpd/conf/modules.d/70_mod_php.conf: Cannot load modules/mod_php.so into server: /etc/httpd/modules/mod_php.so: cannot allocate memory in static TLS block

Restarting httpd later in the day shows:

[Sun Oct 02 10:35:17.304602 2022] [core:warn] [pid 1773416] AH00098: pid file /run/httpd/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Sun Oct 02 10:35:17.312408 2022] [mpm_prefork:notice] [pid 1773416] AH00163: Apache/2.4.54 (Unix) OpenSSL/1.1.1q PHP/8.0.23 mod_perl/2.0.10 Perl/v5.32.1 configured -- resuming normal operations
[Sun Oct 02 10:35:17.312465 2022] [core:notice] [pid 1773416] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

Any ideas for debugging this?

Comment 1 Thomas Backlund 2022-10-02 12:34:52 CEST

(In reply to Barry Jackson from comment #0)

> 
> [Sun Oct 02 04:02:02.442675 2022] [mpm_prefork:notice] [pid 1411] AH00171:
> Graceful restart requested, doing restart
> httpd: Syntax error on line 54 of /etc/httpd/conf/httpd.conf: Syntax error
> on line 1 of /etc/httpd/conf/modules.d/70_mod_php.conf: Cannot load
> modules/mod_php.so into server: /etc/httpd/modules/mod_php.so: cannot
> allocate memory in static TLS block
> 

I'd say this it the one that kills your restart.

So that would be related to:
https://bugs.php.net/bug.php?id=80814
https://bz.apache.org/bugzilla/show_bug.cgi?id=65918

Comment 2 Barry Jackson 2022-10-02 14:17:30 CEST

Thanks Thomas,
I did not find those when I searched.

Obviously an issue way over my head. ;)

Will flag as UPSTREAM and keep open.

For now I will go back to my workaround of stopping httpd at 03.59 and restarting at 04:01 and keep checking back in those bug reports from time to time.

Severity: normal => major
Keywords: (none) => UPSTREAM

Comment 3 Thomas Backlund 2022-10-02 14:33:33 CEST

As another workaround you could try:


-	/bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true
+	/bin/systemctl restart httpd.service > /dev/null 2>/dev/null || true



as restarting it might help clearing the used memory slots..

Comment 4 Barry Jackson 2022-10-02 23:40:22 CEST

Thanks, Done.
 
I will report back after a week or earlier if it happens again.

Comment 5 Lewis Smith 2022-10-03 21:48:56 CEST

Thanks to Barry for the report; and Thomas for dealing with it. Assigning to you since it is already in your hands.

> cannot allocate memory in static TLS block
Looks unhealthy.

Assignee: bugsquad => tmb

Comment 6 Barry Jackson 2022-10-10 12:53:16 CEST

Thanks Thomas,

It's been a week now without a problem so 'restart' rather than 'reload' appears to have done the trick.

Whether this would be a suitable default is less clear so I will leave it for you and others to decide

My thought is that a brief disconnect is better than an apparently random, complete shutdown.

Cheers,
Barry