Bug 20255 - NFS server service is not starting at boot
Summary: NFS server service is not starting at boot
Status: RESOLVED FIXED
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: Cauldron
Hardware: All Linux
Priority: Normal major
Target Milestone: ---
Assignee: Guillaume Rousse
QA Contact:
URL: https://bugzilla.redhat.com/show_bug....
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-02-07 20:43 CET by Davy Defaud
Modified: 2017-07-06 14:53 CEST (History)
4 users (show)

See Also:
Source RPM: nfs-utils-1.3.4-1.mga6.src.rpm
CVE:
Status comment:


Attachments

Description Davy Defaud 2017-02-07 20:43:28 CET
Description of problem:
The NFS server service is enabled but fails to start at boot time.
Starting manually after boot works.

How reproducible: At every boot.


Steps to Reproduce:
1. install nfs-utils package
2. configure the exported directory in /etc/exports and run exportfs -a
3. start the NFS server service: systemctl start nfs-server
4. enable the service at boot time: systemctl enable nfs-server
5. reboot, the service wonât start. :-(

Redâ¯Hat bugzilla #970595 is reporting a very similar issue.
People are talking about an nfs.target and nfs-lock.service that should be enabled, but that Mageia doesnât provide:

[root@darkvador ~]# systemctl status nfs.target nfs-lock
Failed to dump process list, ignoring: Unit nfs.target not found.
Failed to dump process list, ignoring: Unit nfs-lock.service not found.
â nfs.target
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

â nfs-lock.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)
Comment 1 Bit Twister 2017-02-08 04:37:08 CET
FYI: rather than modifying /etc/exports,
mkdir -p /etc/exports.d and place your settings in something like 
/etc/exports.d/local.exports.

As for your target/service items, here is a chopped down copy of mine with 
loaded active words removed.

$ systemctl list-units | grep nfs
proc-fs-nfsd.mount              mounted   NFSD configuration filesystem
var-lib-nfs-rpc_pipefs.mount    mounted   RPC Pipe File System
nfs-idmapd.service              running   NFSv4 ID-name mapping service
nfs-mountd.service              running   NFS Mount Daemon
nfs-server.service              exited    NFS server and services
rpc-statd.service               running   NFS status monitor for NFSv2/3 locking.
nfs-client.target               active    NFS client services

Verified mine is active on boot. You will need to look through the journal to see if you can find something to do with your problem.

Only change to /etc/nfsmount.conf is around line 147

# disable ipv6 changed by /local/bin/nfsmount_conf_changes Fri 03 Feb 09:54 2017
# https://forums.mageia.org/en/viewtopic.php?f=25&t=10081#p58482
[ Server "localhost" ]
Defaultvers=3

Entirely possible I have tweaked something else which allows it run on my setup.
For instance my network is started via systemd-networkd.service instead of network or NetworkManager. This is a static ip address setup running named.

Could be because I have gssproxy active but have it disabled on boot.
I have also disabled ipv6 on boot and every conf file using it.

This is a clean install of Mageia-6-sta2-x86_64-DVD.iso of all package groups in  Server and Workstation plus updates which means I might have packages installed which you do not.

CC: (none) => bittwister2

Comment 2 Marja Van Waes 2017-02-08 10:33:49 CET
Assigning to the registered maintainer.

CC: (none) => marja11
Assignee: bugsquad => guillomovitch

Comment 3 Guillaume Rousse 2017-02-08 20:50:32 CET
I can't reproduce the reported issue, on current cauldron.

The forementioned systemd units (nfs.target and nfs-lock.service) don't exist anymore since the switch to upstream-provided systemd units (nfs-utils 1.3.3).
Comment 4 Frank Griffin 2017-02-08 21:01:17 CET
I've been reproducing this for months.  I thought I had opened a bug report, but I can't find it at the moment.  It might just have been a post on dev asking for confirmation

Basically, I have to do a systemctl restart nfs-server on all of my cauldron systems to get NFS directories exported.  My network is controlled by NM.

CC: (none) => ftg

Comment 5 Guillaume Rousse 2017-02-08 21:14:05 CET
It seems like a network readiness issue at boot time. The current nfs-server service requires network.target, but I suspect it is not enough to ensure the network is actually working.

Can you try to add an additional dependency on network-online.target ? The easiest way is to use a 
/etc/systemd/system/nfs-server.service.d/local.conf drop-in file, with following content:
Requires=network-online.target
Comment 6 Frank Griffin 2017-02-08 21:47:33 CET
(In reply to Guillaume Rousse from comment #5)
> It seems like a network readiness issue at boot time. The current nfs-server
> service requires network.target, but I suspect it is not enough to ensure
> the network is actually working.
> 
> Can you try to add an additional dependency on network-online.target ? The
> easiest way is to use a 
> /etc/systemd/system/nfs-server.service.d/local.conf drop-in file, with
> following content:
> Requires=network-online.target

That does the trick.  What's the equivalent if the activity is a target ?  I've long had problems with remote-fs.target running before the network is up and not mounting NFS volumes, but I tried symlinking network-online.target into remote-fs.target.wants and that didn't do the trick.  Any hints ?
Comment 7 Guillaume Rousse 2017-02-08 23:22:19 CET
According to systemd.unit(5), /etc/systemd/system/remote-fs.target.d/local.conf should also work for remote-fs.target.
Comment 8 Frank Griffin 2017-02-08 23:29:13 CET
(In reply to Guillaume Rousse from comment #7)
> According to systemd.unit(5),
> /etc/systemd/system/remote-fs.target.d/local.conf should also work for
> remote-fs.target.

Didn't.  Got an error at boot saying that local.conf couldn't be included and would be ignored.
Comment 9 Davy Defaud 2017-02-09 12:48:49 CET
I use Network Manager too, Bit Twister donât use it and I suppose that Guillaume neither.
So, if I try to summarize, the problem resides in the use Network Manager jointly with the NFS server service.
So, *eventually*, what would be the proper fix?

Modifying /usr/lib/systemd/system/nfs-server.service by adding:
Requires=network-online.target
Comment 10 Guillaume Rousse 2017-02-09 15:42:01 CET
I think the problem comes from NFS server using an insufficient 'network' dependency, which is not enough to ensure proper scheduling at boot time. The relationship with network manager is difficult to establish. And indeed, the fix seems to use 'network-online' systemd target instead of 'network' one.
Comment 11 Davy Defaud 2017-02-09 16:56:49 CET
Iâve just tried to replace the Requires=network.target by Requires=network-online.target.
Unfortunately, it doesnât work! :-(
I can see that thereâs also âAfter= network.targetâ, perhaps it should be replaced by âAfter= network-online.targetâ too?

Hereâs my current nfs-server.service file :

[Unit]
Description=NFS server and services
DefaultDependencies=no
Requires= network-online.target proc-fs-nfsd.mount rpcbind.target
Requires= nfs-mountd.service
Wants=rpcbind.socket
Wants=rpc-statd.service nfs-idmapd.service
Wants=rpc-statd-notify.service

After= local-fs.target
After= network.target proc-fs-nfsd.mount rpcbind.service nfs-mountd.service
After= nfs-idmapd.service rpc-statd.service
Before= rpc-statd-notify.service

# GSS services dependencies and ordering
Wants=auth-rpcgss-module.service
After=rpc-gssd.service gssproxy.service rpc-svcgssd.service

# start/stop server before/after client
Before=remote-fs-pre.target

Wants=nfs-config.service
After=nfs-config.service

[Service]
EnvironmentFile=-/run/sysconfig/nfs-utils

Type=oneshot
RemainAfterExit=yes
ExecStartPre=/usr/sbin/exportfs -r
ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS
ExecStop=/usr/sbin/rpc.nfsd 0
ExecStopPost=/usr/sbin/exportfs -au
ExecStopPost=/usr/sbin/exportfs -f

ExecReload=/usr/sbin/exportfs -r

[Install]
WantedBy=multi-user.target
Comment 12 Frank Griffin 2017-02-09 17:38:55 CET
Davy, you didn't use exactly the workaround mentioned by Guillaume in comment#5.  It wasn't to change the Requires but to add an additional file.
Comment 13 Guillaume Rousse 2017-02-09 20:06:55 CET
Using an additional file was just supposed to be easier than to modify original one, the result should be the same. With one small difference tough: when using an drop-in file, the additional dependency is added, as 'requires' directive is multivalued, whereas the modification used by Davy replaced it.

Davy, could you add back 'network.target' dependency, in addition to 'network-online.target', and see if it make any difference ?
Comment 14 Davy Defaud 2017-02-10 12:50:32 CET
@Frank: I didnât tried the proposed workaround because it is⦠a workaround. Iâd like to find a proper fix for this bug.

@Guillaume:
I tried to add back network.target in addition to network-online.target in the Requires. Still no success. I even tried to add network-online.target in the âAfter=â rule, but it still doesnât help. I donât think that the order matters, but hereâs exactly how Iâve changed my unit file:

[Unit]
Description=NFS server and services
DefaultDependencies=no
Requires= network.target network-online.target proc-fs-nfsd.mount rpcbind.target
Requires= nfs-mountd.service
Wants=rpcbind.socket
Wants=rpc-statd.service nfs-idmapd.service
Wants=rpc-statd-notify.service

After= local-fs.target
After= network.target network-online.target proc-fs-nfsd.mount rpcbind.service nfs-mountd.service
After= nfs-idmapd.service rpc-statd.service
[â¦]
Comment 15 Guillaume Rousse 2017-02-13 19:24:40 CET
Try passing --debug and --syslog options to nfs server, using RPCNFSDARGS variable in /etc/sysconfig/nfs, as I don't have any better idea.
Comment 16 Davy Defaud 2017-02-14 11:27:34 CET
I tried last night but, unfortunately, it doesnât provide useful information.
Comment 17 Davy Defaud 2017-02-16 03:44:07 CET
I have some good news. Iâve found the solution! :-)
Well, actually not me ;-), but the Ubuntu maintainer from Canonical. And, he has even reported the problem upstream.

IIUC, the problem is a circular dependency on the rpcbind daemon between the NFS server (nfs-server.service) and the NFS client (nfs-mountd.service)â¦

We were focused on the nfs-server.service, but itâs an nfs-mountd.service requirement that is preventing nfs-server.service to start at boot.

The solution (that Iâve just validated) is to add the following requirement in nfs-mountd.service:

After=rpcbind.socket


Hereâs the Ubuntu bug report with the whole discussion:
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1590799

And the upstream answer about the proposed solution that *should* (âI would not be against addingââ¦) be merged:
http://marc.info/?l=linux-nfs&m=148650983616845&w=2
But no yet included, AFAICS: http://git.linux-nfs.org/?o=age


So, Guillaume, I donât know if you prefer an immediate rebuilding including the patch, or wait for the next nfs-utils release which would contain the patch and other fixes/enhancementsâ¦

TIA,

Davy
Comment 18 Guillaume Rousse 2017-02-16 08:14:30 CET
I just submitted a new release including the fix, as the next upstream release (2.1.1) has already been published, but is slightly too invasive for me right now.
Comment 19 Davy Defaud 2017-02-16 21:26:24 CET
I confirm that nfs-utils-1.3.4-2.mga6 is fixing the bug.

Thank you Guillaume for this quick update,

Davy

Status: NEW => RESOLVED
Resolution: (none) => FIXED

Comment 20 Guillaume Rousse 2017-02-16 21:30:55 CET
Thanks, you did the hardest part :)
Comment 21 Davy Defaud 2017-07-06 14:53:15 CEST
*** Bug 21181 has been marked as a duplicate of this bug. ***

CC: (none) => ghibomgx


Note You need to log in before you can comment on or make changes to this bug.