Bug 14427 - httpd fails to start
Summary: httpd fails to start
Status: RESOLVED FIXED
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: Cauldron
Hardware: i586 Linux
Priority: Normal normal
Target Milestone: ---
Assignee: Colin Guthrie
QA Contact:
URL:
Whiteboard: 5beta1
Keywords:
Depends on:
Blocks:
 
Reported: 2014-11-01 20:49 CET by William Kenney
Modified: 2014-11-07 15:42 CET (History)
1 user (show)

See Also:
Source RPM:
CVE:
Status comment:


Attachments
httpd failing to start in M5B1 (3.97 KB, text/plain)
2014-11-02 16:49 CET, William Kenney
Details

Description William Kenney 2014-11-01 20:49:10 CET
After a successful install of .....
an update of the install resulted in httpd not starting. httpd did work
before the update.
[root@localhost wilcal]# service httpd start
Redirecting to /bin/systemctl start httpd.service
Job for httpd.service failed. See "systemctl status httpd.service" and "journalctl -xe" for details.
[root@localhost wilcal]# ^C
[root@localhost wilcal]# systemctl status httpd.service
â httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
   Active: failed (Result: exit-code) since Sat 2014-11-01 12:22:23 PDT; 45s ago
  Process: 2195 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
 Main PID: 2195 (code=exited, status=1/FAILURE)

Nov 01 12:22:23 localhost httpd[2195]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'S...is message
Nov 01 12:22:23 localhost systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Nov 01 12:22:23 localhost systemd[1]: Failed to start The Apache HTTP Server.
Nov 01 12:22:23 localhost systemd[1]: Unit httpd.service entered failed state.
Nov 01 12:22:23 localhost systemd[1]: httpd.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

Reproducible: 

Steps to Reproduce:
Comment 1 William Kenney 2014-11-01 20:54:50 CET
For some reason httpd worked fine as a Vbox client but when I went
to real hardware, then updated the install, httpd stopped working
and I could not get it to start. Proftp works just fine.
Comment 2 David Walser 2014-11-01 21:08:17 CET
Do you see anything in the logs in /var/log/httpd, or /var/log/messages or /var/log/syslog or the systemd journal (journalctl) that indicate what the problem is?  You haven't provided any information that can be used to debug this.

Keywords: (none) => NEEDINFO

Comment 3 William Kenney 2014-11-01 21:14:58 CET
(In reply to David Walser from comment #2)

> Do you see anything in the logs in /var/log/httpd, or /var/log/messages or
> /var/log/syslog or the systemd journal (journalctl) that indicate what the
> problem is?  You haven't provided any information that can be used to debug
> this.

Thanks for the response David. I'm still work'n this one.
This test install was on an x86-64 machine.
I'm gonna try a HW install on an i586 machine later today
and see if it does the same thing. For now lemme just tinkerwith it.
Any help, like look'n at /var/log/httpd /var/log/messages
or /var/log/syslog, is helpful.
Comment 4 William Kenney 2014-11-02 16:49:44 CET
Created attachment 5556 [details]
httpd failing to start in M5B1

Looking at this again this morning I'm finding that httpd is failing on all of
my M5 test platforms. I have attached httpd_fail.txt to this bug with copys
of what I think is the the relevant text from service httpd start,
/var/log/httpd/error_log, /var/log/messages and /var/log/syslog
Comment 5 William Kenney 2014-11-02 16:50:39 CET
Looking at this again this morning I'm finding that httpd is failing on all of
my M5 test platforms. I have attached httpd_fail.txt to this bug with copys
of what I think is the the relevant text from service httpd start,
/var/log/httpd/error_log, /var/log/messages and /var/log/syslog
William Kenney 2014-11-02 18:15:00 CET

Whiteboard: (none) => 5beta1

Comment 6 David Walser 2014-11-03 03:57:18 CET
Comment on attachment 5556 [details]
httpd failing to start in M5B1

The only thing in that attachment that appears to say anything is the lines like:
"[Sun Nov 02 07:25:18.808960 2014] [core:emerg] [pid 1183] (2)No such file or directory: AH00023: Couldn't create the rewrite-map mutex (file /run/httpd/rewrite-map.1183)
AH00016: Configuration Failed"

What is the output of "ls -ld /run/httpd" ?
Comment 7 William Kenney 2014-11-03 05:03:16 CET
(In reply to David Walser from comment #6)

> What is the output of "ls -ld /run/httpd" ?

[root@localhost wilcal]# ls -ld /run/httpd
ls: cannot access /run/httpd: No such file or directory

And indeed in /run there is no httpd anything.
In this M4 workstation that is running httpd there is:

/run/httpd/authdigest_shm.1459
/run/httpd/httpd.pid
Comment 8 Frank Griffin 2014-11-03 12:46:42 CET
Just for giggles, try "mkdir /run/httpd; chown apache:apache /run/httpd" and retest.

CC: (none) => ftg

Comment 9 David Walser 2014-11-03 14:23:49 CET
We don't need any giggles, of course it'll work if /run/httpd exists.

systemd-tmpfiles isn't working, and that's clearly the issue.

Keywords: NEEDINFO => (none)
Assignee: bugsquad => mageia

Comment 10 Colin Guthrie 2014-11-03 14:29:30 CET
Can you test:

1. stop apache
2. rm -rf /run/httpd
3. /usr/bin/systemd-tmpfiles --create
4. ls -ld /run/httpd

This works fine for me here on Cauldron.
Comment 11 Frank Griffin 2014-11-03 14:49:58 CET
(In reply to David Walser from comment #9)
> We don't need any giggles, of course it'll work if /run/httpd exists.
> 
> systemd-tmpfiles isn't working, and that's clearly the issue.

I wasn't aware that creation of /run/httpd was the responsibility of systemd-tmpfiles.  

The intent of the test was to see if, in trying to create /run/httpd/rewrite-map.1183, apache wasn't creating the needed directory structure itself, and whether it ought to be.
Comment 12 Colin Guthrie 2014-11-03 15:31:09 CET
As apache is run by root at this stage (due to it needing to be root to get port 80) it may as well create the folder internally rather than rely on external stuff and/or packaging to do it for us, so IMO, apache itself could be fixed instead to do the necessary mkdir stuff.
Comment 13 William Kenney 2014-11-03 16:08:49 CET
(In reply to Colin Guthrie from comment #10)

> Can you test:
> 
> 1. stop apache
> 2. rm -rf /run/httpd
> 3. /usr/bin/systemd-tmpfiles --create
> 4. ls -ld /run/httpd
> 
> This works fine for me here on Cauldron.

This is getting very interesting. Yes Colin, infact running through
the process you described does get httpd going again, and it works
just fine. But, it does not survive a reboot. Upon a reboot the
/run/httpd directory is gone again and service httpd does not start.
Comment 14 Colin Guthrie 2014-11-03 16:27:10 CET
(In reply to William Kenney from comment #13)
> This is getting very interesting. Yes Colin, infact running through
> the process you described does get httpd going again, and it works
> just fine. But, it does not survive a reboot. Upon a reboot the
> /run/httpd directory is gone again and service httpd does not start.

After a fresh boot, what does the following command (run as root) say:

systemctl status systemd-tmpfiles-setup.service
Comment 15 William Kenney 2014-11-03 16:47:59 CET
(In reply to Colin Guthrie from comment #14)

> After a fresh boot, what does the following command (run as root) say:
> 
> systemctl status systemd-tmpfiles-setup.service

[root@localhost wilcal]# systemctl status systemd-tmpfiles-setup.service
â systemd-tmpfiles-setup.service - Create Volatile Files and Directories
   Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-setup.service; static)
   Active: inactive (dead)
     Docs: man:tmpfiles.d(5)
           man:systemd-tmpfiles(8)
Comment 16 William Kenney 2014-11-03 16:49:12 CET
I'm here for about another hour.
Comment 17 Colin Guthrie 2014-11-03 16:54:04 CET
OK, so that looks like it's not even being tried at all (as it says "inactive (dead)").


Smells very much like there is some kind of early boot ordering cycle problems and this is just a symptom of it bubbling up.

Please reboot with "systemd.log_level=7 systemd.log_target=journal" on the kernel command line and post the output of "journalctl -b"

Thanks!
Comment 18 William Kenney 2014-11-03 17:00:13 CET
Note that both this and:
https://bugs.mageia.org/show_bug.cgi?id=14436
seem to have occured both during the same update last week.
The NFS mount thing has also been attributed historically
to boot ordering sequence.
Comment 19 William Kenney 2014-11-03 17:06:41 CET
(In reply to Colin Guthrie from comment #17)

> Please reboot with "systemd.log_level=7 systemd.log_target=journal" on the
> kernel command line and post the output of "journalctl -b"

Your gonna have to coach me on how to do that. I'm using GRUB2
on the system under test. Also I've got two Vbox M5 installs that
are about a week old that don't exhibit this problem. Both the
NFS and HTTPD thing surfaced at the same time on new M5 installs
about Thursday last week.
Comment 20 Colin Guthrie 2014-11-03 17:34:52 CET
I'm afraid I don't know anything about GRUB2 (I have avoided it like the plague). In older grub you just hit esc, then e for edit, then edited it and pressed return and then hit b to boot it.
Comment 21 William Kenney 2014-11-03 17:40:36 CET
(In reply to Colin Guthrie from comment #20)

> I'm afraid I don't know anything about GRUB2 (I have avoided it like the
> plague). In older grub you just hit esc, then e for edit, then edited it and
> pressed return and then hit b to boot it.

Thanks. I'll have to let you guys take this from here.
Comment 22 Colin Guthrie 2014-11-03 18:15:14 CET
Hmm, this looks like a likely fix:

http://cgit.freedesktop.org/systemd/systemd/commit/?id=919699ec301ea507edce4a619141ed22e789ac0d

I'll apply it in my next build
Comment 23 William Kenney 2014-11-07 15:42:12 CET
Resolved in latest builds.
Comment 24 William Kenney 2014-11-07 15:42:43 CET
Resolved

Status: NEW => RESOLVED
Resolution: (none) => FIXED


Note You need to log in before you can comment on or make changes to this bug.