Bug 14106 - runlevel unknown, no network after update
Summary: runlevel unknown, no network after update
Status: RESOLVED DUPLICATE of bug 15670
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: Cauldron
Hardware: x86_64 Linux
Priority: Normal normal
Target Milestone: ---
Assignee: Colin Guthrie
QA Contact:
URL:
Whiteboard:
Keywords: NEEDINFO
Depends on:
Blocks:
 
Reported: 2014-09-17 08:13 CEST by Martin Schmitz
Modified: 2015-06-06 09:43 CEST (History)
7 users (show)

See Also:
Source RPM: systemd
CVE:
Status comment:


Attachments
urpmi --auto-update (console output) (62.12 KB, text/plain)
2014-09-17 11:01 CEST, Martin Schmitz
Details
jounaltctl -b directly after reboot (97.46 KB, text/plain)
2014-09-17 11:02 CEST, Martin Schmitz
Details
jounalctl -b after run: init 3 (102.47 KB, text/plain)
2014-09-17 11:04 CEST, Martin Schmitz
Details
Dead services after reboot (10.62 KB, text/plain)
2015-04-22 14:34 CEST, Meg Skywalker
Details
systemd-update-utmp-runlevel.status after reboot (270 bytes, text/plain)
2015-04-22 14:35 CEST, Meg Skywalker
Details
systemd-update-utmp.status after reboot (495 bytes, text/plain)
2015-04-22 14:36 CEST, Meg Skywalker
Details

Description Martin Schmitz 2014-09-17 08:13:07 CEST
Description of problem:

runlevel unknown, no network after first "urpmi --auto-update"

Version-Release number of selected component (if applicable):

net-tools-2.0-0.20140707git.4.mga5 (I think it's the init scripts)

How reproducible:

I've installed Cauldron R2 as server (package-choice: minimal (with grumpi (2nd menupoint in installation)) without dokumentation (about 700MB)).
- installed sshd, shorewall
- reboot => runlevel 3, network-up, access per ssh perfect
Then I did a "urpmi --auto-update" directly after installation.
- reboot => runlevel unknown, no network, no access per ssh
Logged-in as root
- type "init 3" => runlevel 3, network-up, access per ssh perfect

Steps to Reproduce:
1. repeat "How reproducible:"


Reproducible: 

Steps to Reproduce:
Comment 1 Marja Van Waes 2014-09-17 10:05:26 CEST
I'm not sure I understand what you mean by "runlevel unknown", do you mean it ended up in rescue mode?

Can you please, as root, run: 
journalctl -b > journalctl-b.txt

and attach journalctl-b.txt to this bug report?

CC: (none) => marja11
Component: Backports => RPM Packages

Marja Van Waes 2014-09-17 10:05:45 CEST

Whiteboard: (none) => NEEDINFO

Comment 2 Martin Schmitz 2014-09-17 11:01:24 CEST
Created attachment 5419 [details]
urpmi --auto-update (console output)
Comment 3 Martin Schmitz 2014-09-17 11:02:56 CEST
Created attachment 5420 [details]
jounaltctl -b directly after reboot
Comment 4 Martin Schmitz 2014-09-17 11:04:00 CEST
Created attachment 5421 [details]
jounalctl -b after run: init 3
Comment 5 Martin Schmitz 2014-09-17 11:09:31 CEST
WOW!! fast reaction!

by "runlevel unknown" I mean:

- restart server
- login as "root"
- run: "runlevel"
- runlevel throws "unknown" to console (network unreachable)
- run "init 3"
- run: "runlevel"
- runlevel throws "N 3" to console (network is up)

so it's not possible to update and reboot the server remote, because after reboot, I have to type "init 3" manually. After "init 3" is done, everythings running fine
Comment 6 Martin Schmitz 2014-09-17 11:39:50 CEST
p.s. in "jounaltctl-b_runlevel_unknown.txt" is "network-up" missing.
Comment 7 Marja Van Waes 2014-09-17 11:47:02 CEST
Thx for all the information.

CC'ing Colin (who knows all about initscripts and systemd) and Thomas (who was the last to push net-tools)

CC: (none) => mageia, thomas
Whiteboard: NEEDINFO => (none)

Comment 8 David Walser 2014-09-17 20:58:17 CEST
What does net-tools have to do with any of this?

Also, what does this show?
ls -l /etc/systemd/system/default.target
Comment 9 Martin Schmitz 2014-09-18 08:49:58 CEST
Hi David

sorry about my mentioned "net-tools", but to create a bug, you have to choose a package. For I don't know where the bug is, I chose this one (seems plausible to me, that missing network have sth. to do with a package with network-stuff).
But as I wrote in my first post "(I think it's the init scripts)", because network runs, if running "init 3" manually.

ls -l /etc/systemd/system/default.target points to
/usr/lib/system.d/system/multi-user.target

I forgot, sth. to say:
As I setup the server, the was a "small" update first (5/6 packages including clib). After this, the system is telling "please restart couse of new clib". The the second "big" update (see attachment 5419 [details]) was done behind.

If you want to, I can setup the server again and do a
- journalctl -b > journalctl-b.txt
between "small" and "big" update. Or is the infos' I sent you enough to reproduce the bug?
Comment 10 Martin Schmitz 2014-09-22 12:45:28 CEST
Hi Marja,

it seems to be a shorewall problem. I reinstalled the server:

starts up and displays:
> started IP-tables for firewall -> OK
run "runlevel" => "N 3"

reboot after the update:
> xt_addrtype ipv6 does not support broadcast matching
run "runlevel" => unknown
run "init 3"
run "runlevel" => N 3

after first installation I did a
> urpmi.removemedia -a
> urpmi.addmedia --distrib --mirror-list
> urpmi.removemedia (all 32bit)
> urpmi --auto-update
- aria2
- glibc
- lib64archive13
- lib64rpm3
- locales
- locales-de
- meta-task
- perl-MDV-Distribconf
- perl-URPM
- rpm
- urpmi

reboot => till here everything is OK

after 2nd installation (342 packages) the error occures.
Comment 11 Meg Skywalker 2014-09-24 16:39:16 CEST
Same here. Since some update via --auto-update the network not starting (correct) anymore after reboot.

Same here, tell me that runlevel is "unknown".

CC: (none) => identity.mageia.org

Comment 12 David Walser 2014-10-08 01:09:57 CEST
Sounds to me like systemd isn't correctly setting the runlevel in utmp anymore.

Assignee: bugsquad => mageia
Source RPM: net-tools-2.0-0.20140707git.4.mga5 => systemd

Comment 13 Martin Schmitz 2014-10-08 09:26:00 CEST
Maybe (my knowledge isn't in this enough to discuss). I only know how to reproduce this but easily:

- install minimum installation (ca: 750GB)
- 1. update
- 2. update
- Error occures
Comment 14 Morgan Leijström 2014-11-05 19:16:09 CET
One of my cauldron 64 bit systems, Nvidia, fail to boot with latest kernel 3.17.2-desktop-3.mga5

The boot messages on screen stops at 
 " Starting Update UTMP about System Runlevel Changes"

I also experience Bug 14457 - Missing hwdb.bin at start of, boot me at #5

I can ctrl-alt-Fx to other consoles if someone (want me to check or try something)

For now, we boot on kernel 3.17.1-desktop-2.mga5 and it works nice.

CC: (none) => fri

Comment 15 Martin Schmitz 2014-11-06 10:23:24 CET
@Morgan: does this have anything to do with the missing network / runlevel? I can't find any coherence to the threadtheme.
Comment 16 Morgan Leijström 2014-11-06 11:01:31 CET
No idea, I am just an "advanced user"/reporter trying to bring information :)

I can log and try things if i get specific instructions.

Beware though this is a production machine although running cauldron, because neither mga4 nor fedora20 handle its suspend.
Comment 17 Morgan Leijström 2014-11-11 21:26:39 CET
Whatever it was, that problem one of my copmuters had is gone with latest kernel 3.17.2-desktop-4.mga5 and other updates :)
Comment 18 Martin Schmitz 2014-11-12 07:41:00 CET
Dear forumadmin,

can you please move the comments from "Morgan Leijström" to the correct thread, because they have nothing to do with the "unknown runlevel/missing network" bug.
After the last comment, people may think, that the bug is solved, but it isn't. The biggest problem is, that the bug occures if one tries to setup a computer (mostly used for server-services) with minimal software (like described in the first post). Clients with X-server and more rpm's installed does not have this problem. I did not found out, which rpm(s) is(are) needed to solve this problem.
Comment 19 Christiaan Welvaart 2014-11-12 08:26:27 CET
Bugzilla is not a forum. 

Do you still have this problem with current cauldron? In your logs I see:

shorewall[604]: /usr/lib64/ipset/ipset_bitmap_port.so: /usr/lib64/ipset/ipset_bitmap_port.so: undefined symbol: ipset_parse_tcp_udp_port

This problem is supposed to be fixed now.

---------------------------------------------------------------------------
Anyway, what you would want to know at first is:

  systemctl status multi-user.target

(runlevels 2,3,4 are aliases of multi-user.target; runlevel 5 is an alias of graphical.target)

On your system default.target is the same as multi-user.target as you said it is symlinked in /etc (I guess this determines your "runlevel" on boot up).

If you don't see a problem there, the status of
   network.service
   network-up.service
may be useful, and finally sshd.service but that's not where the problem is AFAICT.

CC: (none) => cjw

Comment 20 Meg Skywalker 2014-11-12 12:53:10 CET
This problem is supposed to be fixed now.
> ---------------------------------------------------------------------------
> Anyway, what you would want to know at first is:

  systemctl status
> multi-user.target

(runlevels 2,3,4 are aliases of multi-user.target;
> runlevel 5 is an alias of graphical.target)

On your system default.target
> is the same as multi-user.target as you said it is symlinked in /etc (I
> guess this determines your "runlevel" on boot up).

If you don't see a
> problem there, the status of
   network.service
   network-up.service
may be
> useful, and finally sshd.service but that's not where the problem is AFAICT.

Not fixed.
After reboot:
multi-user.target active
network.service inactive (dead)
network-up.service inactive (dead)
sshd.service active (running)
Comment 21 Meg Skywalker 2015-04-21 14:26:27 CEST
***PUSH***
Comment 22 Martin Schmitz 2015-04-22 07:41:08 CEST
...and it's so easiliy to reproduce :-). I can't imagine, that solving this problem is such a big task, because if you run "init 3" manually after rebooting, everything is running fine.

In my opinion, this entry is missing somewhere in the (self-generating?) start-script after the first update.
Comment 23 Marja Van Waes 2015-04-22 08:45:10 CEST
(In reply to Martin Schmitz from comment #22)
> ...and it's so easiliy to reproduce :-). I can't imagine, that solving this
> problem is such a big task, because if you run "init 3" manually after
> rebooting, everything is running fine.

You're probably right: For someone with plenty of free time and the needed knowledge it won't be hard.

However, while Colin (the assignee) does have that knowledge, he does not have much time: he has a living to earn, a private life, is involved in at least one upstream project (if not more), has sysadmin tasks here, a team to lead, 58 more open bugs assigned to him etc.

> 
> In my opinion, this entry is missing somewhere in the (self-generating?)
> start-script after the first update.

If you do happen to have time to inquire further, David's suggestion sounds like a good starting point

(In reply to David Walser from comment #12)
> Sounds to me like systemd isn't correctly setting the runlevel in utmp
> anymore.

Maybe systemd-update-utmp-runlevel.service or systemd-update-utmp.service is involved? 

I lack both the needed knowlegde and the needed time to look further
Comment 24 Marja Van Waes 2015-04-22 10:29:37 CEST
@ Martin

Did you only hit this bug on the very first updates you did, and never again on later updates?

When Meg reported he had hit it, it was about a week after the first Mageia 5 mass rebuild, in September.

However, did it occur again when there were less updates at the same time? That might help to find the culprit (to me, with my less-than-basic understanding of how such things work, even initscripts and sysvinit-legacy-tools could be the cause)
Comment 25 Meg Skywalker 2015-04-22 12:04:35 CEST
For me, I hit the bug on every reboot ;)

Didn't set up a new instance of mageia to check if I can run a "clean" system into the bug. (but comment #13 looks like that).
Comment 26 Marja Van Waes 2015-04-22 14:06:24 CEST
Ah, if it happens on _every_ reboot, and not only on the first reboot after  updating for the first time, then I misunderstood :-/
Comment 27 Meg Skywalker 2015-04-22 14:34:58 CEST
Created attachment 6324 [details]
Dead services after reboot
Comment 28 Meg Skywalker 2015-04-22 14:35:40 CEST
Created attachment 6325 [details]
systemd-update-utmp-runlevel.status after reboot
Comment 29 Meg Skywalker 2015-04-22 14:36:08 CEST
Created attachment 6326 [details]
systemd-update-utmp.status after reboot
Comment 30 Martin Schmitz 2015-04-23 11:42:26 CEST
sth. about time: I'm working fulltime in the university as admin, so I do not have much time either. I know, that you do you jobs for free and also in your free time, and I appreciate this. But the Mageia team asks for helping by bugreports. That's what I've done. Then it was written, it can't be this and that and the thread falls unsoluted in sleepmode.
I can't even install a server, make an update, and then it is to be reinstalled again, because no other update does fix the network (same problem as you have: too less time). So I stopped trying it again.
That was the state, when I reported this bug (installed it nearly 10 times new, always the same error). Now I live with an updateless server, that is rebootable. For me it's the best solution in the moment. I will setup a new Server, when 5 is finished. If the bug occures again, I will stay in this state, if not a bir security problem occures.

But thanks at all for your work. I really like Mageia.
Comment 31 Samuel Verschelde 2015-06-06 02:33:47 CEST
Could it be a duplicate to 15670 (which was fixed very recently)?
Samuel Verschelde 2015-06-06 02:34:12 CEST

Keywords: (none) => NEEDINFO

Comment 32 Thomas Backlund 2015-06-06 09:43:01 CEST

Yep, by all the symptoms above it looks that way

*** This bug has been marked as a duplicate of bug 15670 ***

Status: NEW => RESOLVED
CC: (none) => tmb
Resolution: (none) => DUPLICATE


Note You need to log in before you can comment on or make changes to this bug.