Bug 16086 - preload service causes ordering cycles with rc.local or network-auth services
Summary: preload service causes ordering cycles with rc.local or network-auth services
Status: RESOLVED FIXED
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: 5
Hardware: All Linux
Priority: Normal major
Target Milestone: ---
Assignee: QA Team
QA Contact:
URL:
Whiteboard: advisory MGA5-64-OK MGA5-32-OK
Keywords: validated_update
Depends on:
Blocks:
 
Reported: 2015-06-06 01:15 CEST by David Walser
Modified: 2015-09-14 14:06 CEST (History)
4 users (show)

See Also:
Source RPM: systemd-217-9.mga5.src.rpm, preload
CVE:
Status comment:


Attachments
journalctl -ab with systemd.log_level=debug with network-auth enabled (501.34 KB, text/plain)
2015-07-03 01:09 CEST, David Walser
Details

Description David Walser 2015-06-06 01:15:03 CEST
Booting after enabling the network-auth service resulted in a red ERROR message in the console right at the beginning of the boot process.  It goes by fast, but it said the cycle started at systemd-*something* or something with session (?) and ended at network-auth and network-auth was disabled.

Reproducible: 

Steps to Reproduce:
David Walser 2015-06-06 01:15:09 CEST

Whiteboard: (none) => MGA5TOO

Comment 1 David Walser 2015-06-20 17:00:46 CEST
I also needed to add an /etc/rc.d/rc.local script just for one boot and saw it say it was dropping plymouth-quit-wait (I think) and getty@tty1 because of an ordering cycle.  I think something's wrong with our systemd version.

Severity: normal => major

Comment 2 Colin Guthrie 2015-07-01 09:56:39 CEST
FWIW, the getty@tty1/stop job is pretty harmless. It's not started, so the stop job being removed doesn't matter much - it's more a technicality because it conflicts with the graphical manager which wants to claim tty1.

Anyway, any such issues shouldn't crop up in the normal case so there must be some kind of issue somewhere.

I'll try an install and see what happens. Note that I updated systemd in cauldron not long ago, so is this bug really about MGA5's systemd? (or does it still happen in cauldron too?)
Comment 3 David Walser 2015-07-01 14:47:37 CEST
Ahh thanks, I will not be running Cauldron again for several months.  I am preparing to deploy this Mageia 5 install in the classrooms at work.

Version: Cauldron => 5
Whiteboard: MGA5TOO => (none)

Comment 4 Florian Hubold 2015-07-02 15:16:53 CEST
(In reply to David Walser from comment #1)
> I also needed to add an /etc/rc.d/rc.local script just for one boot and saw
> it say it was dropping plymouth-quit-wait (I think) and getty@tty1 because
> of an ordering cycle.  I think something's wrong with our systemd version.

Exactly the same description as in bug 16260
I also think something is pretty wrong there.

Can you please attach a journal as described in https://bugs.mageia.org/show_bug.cgi?id=16260#c5

CC: (none) => doktor5000

Comment 5 David Walser 2015-07-03 01:09:37 CEST
Created attachment 6800 [details]
journalctl -ab with systemd.log_level=debug with network-auth enabled

Looks like a legit ordering cycle, but still surprising that it wasn't seen in Mageia 4.  Also, systemd should do like RPM and delete *dependencies* when it finds ordering loops, not *services.*
Comment 6 Colin Guthrie 2015-07-03 11:26:18 CEST
(In reply to David Walser from comment #5)
> Also, systemd should do like RPM and delete *dependencies*
> when it finds ordering loops, not *services.*

It doesn't delete *services* it deletes *jobs*. Jobs are what allows systemd to transition from one state to another and it's the jobs that conflict and are removed to break ordering cycles.

I'll setup a container to see if I can reproduce and fix.

As a quick hunch, I see preload in both the reports here. Can you try disabling this and see if it helps.

I strongly suspect we should remove preload from the distro. The systemd folks upstream removed their one because no one was properly maintaining it and I doubt very much anyone has maintained ours for a long, long time. I suspect it does either more harm than good or does very limited good. If it's to blame, I suggest we simply kill it!
Comment 7 David Walser 2015-07-03 15:21:16 CEST
(In reply to Colin Guthrie from comment #6)
> It doesn't delete *services* it deletes *jobs*. Jobs are what allows systemd
> to transition from one state to another and it's the jobs that conflict and
> are removed to break ordering cycles.

It deletes instances of a service.  If you want to call that jobs, that's fine, but you know what I mean.  It will pick a service that's enabled and refuse to start it.  It should not do that.
Comment 8 David Walser 2015-07-03 15:24:47 CEST
(In reply to Colin Guthrie from comment #6)
> I strongly suspect we should remove preload from the distro. The systemd
> folks upstream removed their one because no one was properly maintaining it
> and I doubt very much anyone has maintained ours for a long, long time. I
> suspect it does either more harm than good or does very limited good. If
> it's to blame, I suggest we simply kill it!

I'll try disabling preload when I get back to work next week.  Our package indeed hasn't been touched since it was imported.  I could have sworn I had updated it at some point, but I guess I'm thinking about something else.  I have no idea what benefit it really gives.
Comment 9 David Walser 2015-07-07 04:18:26 CEST
Good catch Colin.  Disabling preload.service yields no problem with either rc.local or network-auth.

Summary: enabling network-auth service results in ordering cycle which disables network-auth => preload service causes ordering cycles with rc.local or network-auth services
Source RPM: systemd-217-9.mga5.src.rpm => systemd-217-9.mga5.src.rpm, preload

Comment 10 Colin Guthrie 2015-07-07 12:27:41 CEST
(In reply to David Walser from comment #7)
> (In reply to Colin Guthrie from comment #6)
> > It doesn't delete *services* it deletes *jobs*. Jobs are what allows systemd
> > to transition from one state to another and it's the jobs that conflict and
> > are removed to break ordering cycles.
> 
> It deletes instances of a service.  If you want to call that jobs, that's
> fine, but you know what I mean. 

"jobs" is the term used. (see "systemctl list-jobs" for example - although it's output is almost always uninteresting - unless something is taking a long time to start/stop!)

> It will pick a service that's enabled and refuse to start it.
> It should not do that.

What do you suggest it does? I mean, "you" (either the admin or a distro) can create an ordering conflict easily enough. When you try to transition to a state that includes that conflict, what should systemd do? AFAIK there are only two options: break the conflict somehow or just give up. I guess you're suggesting it should just give up? There is a whole bunch of code that tries to remove jobs in a sensible way and it's not *that* common a problem really, so I'm not sure how much effort should be focused on making this bit more complex (adding the emergency shell etc). Or do you have another suggestion?



(In reply to David Walser from comment #9)
> Good catch Colin.  Disabling preload.service yields no problem with either
> rc.local or network-auth.

Good! Nice to know where the problem lies. I'd suggest we just drop it. I doubt it really gives any real times savings on boot (it'll start too late to do any real good I guess and either we should focus proper dev time on making a genuinely useful readahead system work or just drop it - I vote for the latter as the upstream systemd code (which is probably the nicest implementation) was dropped due to lack of proper maintainer, and I really don't think we'll be able to get someone suitably qualified here - it's not the kind of package you can simple "drop in" and expect to magically work!)

So if we do drop it, what package should we update to Obsolete it? Should we obsolete it in a systemd update? Or is there something else more suitable?
Comment 11 David Walser 2015-07-07 13:27:53 CEST
(In reply to Colin Guthrie from comment #10)
> What do you suggest it does? I mean, "you" (either the admin or a distro)
> can create an ordering conflict easily enough. When you try to transition to
> a state that includes that conflict, what should systemd do? AFAIK there are
> only two options: break the conflict somehow or just give up. I guess you're
> suggesting it should just give up? There is a whole bunch of code that tries
> to remove jobs in a sensible way and it's not *that* common a problem
> really, so I'm not sure how much effort should be focused on making this bit
> more complex (adding the emergency shell etc). Or do you have another
> suggestion?

Like I said in Comment 5, do like RPM does.  Delete the dependency, not the job (or the package in RPM's case).  In that case, systemd would be doing the best it can to try to do the right thing for the user.  It would still be starting all of the services that are supposed to start.  Obviously if there's an ordering cycle, (at least) one of the dependencies is invalid.  There's no way for it to know which one, so it just has to remove one of the dependencies and start the services.  That's the best it can do.  It's also still respecting all of the dependencies but one.  If that doesn't quite work right, then systemd can't be blamed.  By refusing to start one of the services completely, it's like it's punishing the user for a developer's mistake, as if it's hoping it's making it easier for the dependency issue to be caught, reported, and fixed.  That's OK in a development environment, but not in production.

> (In reply to David Walser from comment #9)
> So if we do drop it, what package should we update to Obsolete it? Should we
> obsolete it in a systemd update? Or is there something else more suitable?

Probably obsolete it in systemd-units.  Nothing else comes to mind for me that would be more logical.
Comment 12 Colin Guthrie 2015-09-08 12:05:01 CEST
Just to prevent this dragging on, I've created a new "version" of preload that nukes itself. People with preload installed will upgrade and get this, those without it, will not be affected.

Should be in updates_testing soon.

preload-0.6.4-7.1.mga5.src.rpm
preload-0.6.4-7.1.mga5.x86_64.rpm


Advisory text can just be pretty blunt:

This package is deprecated in Mageia. It has not been sufficiently
integrated and is known to cause problems and is thus now disabled.
Colin Guthrie 2015-09-08 12:07:33 CEST

Assignee: mageia => qa-bugs

Comment 13 Thierry Vignaud 2015-09-08 12:33:11 CEST
Wouldn't have it been cleaner to push a package obsoleting it?
(preun of former package would have done the chkconfig --del part)

CC: (none) => thierry.vignaud

Kristoffer Grundström 2015-09-08 17:53:02 CEST

CC: (none) => kristoffer.grundstrom1983

Comment 14 Kristoffer Grundström 2015-09-08 17:54:08 CEST
I see this in x86_64 as well.

Hardware: i586 => All

Comment 15 James Kerr 2015-09-09 13:39:35 CEST
Testing on mga-5-64

Before update:

 rpm -q preload
preload-0.6.4-7.mga5

# ll /usr/sbin/preload
-rwxr-xr-x 1 root root 50288 Oct 15  2014 /usr/sbin/preload*

After installing update from Testing:

 rpm -q preload
preload-0.6.4-7.1.mga5

# ll /usr/sbin/preload
ls: cannot access /usr/sbin/preload: No such file or directory

OK for mga-5-64

Whiteboard: (none) => MGA5-64-OK

Comment 16 James Kerr 2015-09-10 16:36:27 CEST
Testing on mga5-32

Before update:

# systemctl status preload
â preload.service - LSB: Adaptive readahead daemon
   Loaded: loaded (/etc/rc.d/init.d/preload)
   Active: active (running) since Thu 2015-09-10 15:09:36 IST; 13min ago
   CGroup: /system.slice/preload.service
           ââ828 /usr/sbin/preload --verbose 1

After updating to the package in testing:

# systemctl status preload
â preload.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

OK for mga5-32

Whiteboard: MGA5-64-OK => MGA5-64-OK MGA5-32-OK

Comment 17 James Kerr 2015-09-10 16:40:49 CEST
Proposed Advisory
*********************************************
Update removes deprecated preload service

The preload package is deprecated in Mageia. It has not been sufficiently
integrated and is known to cause problems and is thus now disabled.

Reference: https://bugs.mageia.org/show_bug.cgi?id=16086

Source rpm:  preload-0.6.4-7.1.mga5
******************************************************************
Comment 18 James Kerr 2015-09-10 16:44:08 CEST
This update is now validated

A QA committer needs to upload the advisory to SVN

The packages can then be pushed to updates

Keywords: (none) => validated_update
CC: (none) => sysadmin-bugs

Comment 19 claire robinson 2015-09-13 22:41:09 CEST
Advisory uploaded.

Whiteboard: MGA5-64-OK MGA5-32-OK => advisory MGA5-64-OK MGA5-32-OK

Comment 20 Mageia Robot 2015-09-13 23:59:21 CEST
An update for this issue has been pushed to Mageia Updates repository.

http://advisories.mageia.org/MGAA-2015-0121.html

Status: NEW => RESOLVED
Resolution: (none) => FIXED

Comment 21 Kristoffer Grundström 2015-09-14 14:06:45 CEST
(In reply to Mageia Robot from comment #20)
> An update for this issue has been pushed to Mageia Updates repository.
> 
> http://advisories.mageia.org/MGAA-2015-0121.html

Thanks.

Note You need to log in before you can comment on or make changes to this bug.