Does it happens with latest updates?

Just updated and still having the same problem.

Ok, the following test-case reproduces the problem sometimes:

# systemctl stop proc-sys-fs-binfmt_misc.mount
# rmmod binfmt_misc
# /bin/mount binfmt_misc /proc/sys/fs/binfmt_misc -t binfmt_misc

The mount command hangs for a long time in 'D' state..

This testcase reproduces the problem reliably:

# systemctl stop proc-sys-fs-binfmt_misc.mount 
# systemctl stop proc-sys-fs-binfmt_misc.automount 
# rmmod binfmt_misc
# systemctl start proc-sys-fs-binfmt_misc.automount 
# ll /proc/sys/fs/binfmt_misc/
  (hang)


During the hang, this is what ps axww shows:
 4008 ?        D      0:00 /bin/mount binfmt_misc /proc/sys/fs/binfmt_misc -t binfmt_misc
 4009 ?        S      0:00 [kworker/u:1]
 4010 ?        S      0:00 /sbin/modprobe -q -- binfmt_misc
 4011 ?        S      0:00 sh -c /sbin/modprobe --first-time --ignore-install binfmt_misc && { /bin/mount -t binfmt_misc none /proc/sys/fs/binfmt_misc > /dev/null 2>&1 || :; }
 4013 ?        S      0:00 /bin/mount -t binfmt_misc none /proc/sys/fs/binfmt_misc

Perhaps a race condition between 2 mount operations? Don't know where to look from here, so I CC Colin, maybe he has some ideas.

CC: (none) => mageia

For what it's worth, this doesn't cause any hangs for me on boot up or via following your instructions here.

This is on real h/w. Not tried on vbox. I guess this could be h/w specific and actually more related to the binfmt_misc module loading than anything else? Is this even loaded on a typical non-systemd install? (I don't know enough about what misc binfmt actually provides (it sounds quite essential tho...) :s

It happened for me on real hw once (but then I switched this real HW to sysvinit), and in virtualbox.

binfmt_misc is responsible for handling non-known executable file types in Linux. For example, it is possible to configure it to run wine when you run ./something.exe; or batch processor when running ./something.bat, and so on. I don't know how essential this is, at least I have never used it. Perhaps we could disable it by default?

What is the latest systemd release you reproduced this bug with?

Anyway, this subsystem is only needed to make java, mono or windows programs directly executable (see http://0pointer.de/blog/projects/the-new-configuration-files for details), whereas we usually ship shell wrappers for them. It is perfectly non-critical to have it disabled by default.

CC: (none) => guillomovitch

Well, there's no reason for it to take any significant amount of time, especially when run in parallel with the rest of the boot stuff?

In any case, comment #4 looks rather suspect. The 'ps' output looks to me like there is an

install binfmt_misc /sbin/modprobe --first-time --ignore-install
binfmt_misc && { /bin/mount -t binfmt_misc none /proc/sys/fs/binfmt_misc >
/dev/null 2>&1 || :; }

entry in /etc/modprobe.conf or /etc/modprobe.d/*.conf, which will cause issues if one does the "mount" command before binfmt_misc is loaded, causing mount to run "modprobe binfmt_misc" which starts a second mount command for the same directory in parallel.

The cleanest solution in that case would be to drop the 'install' entry from modprobe configuration.
However, I have no such entry on my cauldron system (though I'm not 100% up-to-date at the moment).

CC: (none) => anssi.hannula

For the record, our wine package by default registers windows executables to be run via wine using binfmt_misc.

(In reply to comment #7)
> What is the latest systemd release you reproduced this bug with?

Still having the problem with systemd-34. Will try again when -35 lands in the mirrors. But i don't think it will change much.

Also i don't have install entries in modprobe files.

Sander, can you confirm the findings of comment #4?
I.e. that the hang is reproducable with those commands, and also provide the output of "ps faxww" (note added 'f') during the hang.

Created attachment 776 [details]
# ps faxww

2 mounts and a hang reproduced.

Same problem here, with a fully updated system.

Solution is to comment out line 138 of /lib/module-init-tools/modprobe.default

The line has
install binfmt_misc /sbin/modprobe --first-time --ignore-install binfmt_misc && { /bin/mount -t binfmt_misc none /proc/sys/fs/binfmt_misc > /dev/null 2>&1 || :; }

I thought I'd reported it, but can't find it right now.  I normally boot to
run level 3, and then startx.  Currently, trying to boot to run level 3, mgetty
does not get started.

I'll see if I can find that bug report, or file a new one for it.

CC: (none) => davidwhodgins

Duh, I was intending to ask to check for /lib/module-init-tools/modprobe.default in comment #12, but forgot it :)

Thanks Dave.

The reason I couldn't reproduce is that on 64-bit it is in /lib64/, which is wrong and causes it to not be loaded by modprobe at all.

So either systemd has to be modified to use "modprobe binfmt_misc" instead of trying to mount the binfmt_misc fs, or the line removed or altered in modprobe.default (and modprobe.default moved under /lib/ instead of /%_lib/).

Unless someone looks at this first, I plan to look at this this evening.

Ahh this will be why I don't see it either (64 bit here too).

I would suggest just removing the line in module-init-tools as the systemd way of working uses automounts, not a generic system, and thus modifiying it ot automatically modprobe is kinda going backwards.

Also, I have my suspicious that modprobe itself will kinda die sometime soon and the necessary stuff (which is actually quite trivial) merged into udev... but not sure when (or 100%, if) that will happen, but I think it should be considered when picking the solution to this problem.

Also thinking about this a bit more, perhaps the NFS automount stuff will give us problems at some point too? Perhaps worth considering removing that too if there are other, neater ways to solve this?

I've committed the removal of the binfmt_misc wrappers to module-init-tools.

However, I didn't submit it yet as there were unsubmitted changes by tv in module-init-tools that looked work-in-progress-ish (i.e. might break ldetect), I've asked him about the status of those.

Status: NEW => ASSIGNED
Assignee: bugsquad => anssi.hannula

He had forgotten about it, but said he'll probably be able to finish it on Monday or so.

OK, apparently it is ok to submit it after all :)

Closing as fixed in module-init-tools-3.16-1.mga2.

As for NFS automount, seems the nfsd and sunrpc mount wrappers could probably be removed (the nfs-server script will mount them anyway)... I guess the aliases should be kept, though, to allow auto-modprobing by mount. Colin, do you agree?

Status: ASSIGNED => RESOLVED
Resolution: (none) => FIXED

(In reply to comment #20)
> As for NFS automount, seems the nfsd and sunrpc mount wrappers could probably
> be removed (the nfs-server script will mount them anyway)... I guess the
> aliases should be kept, though, to allow auto-modprobing by mount. Colin, do
> you agree?

I guess so, the aliases seem pretty harmless and could be useful in some setups.