Bug 21080 - ssh login disabled by /run/nologin after a reboot
Summary: ssh login disabled by /run/nologin after a reboot
Status: RESOLVED OLD
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: 6
Hardware: All Linux
Priority: Normal normal
Target Milestone: ---
Assignee: Base system maintainers
QA Contact:
URL:
Whiteboard:
Keywords: NEEDINFO
Depends on:
Blocks:
 
Reported: 2017-06-12 22:21 CEST by Stig-Ørjan Smelror
Modified: 2020-08-02 22:39 CEST (History)
5 users (show)

See Also:
Source RPM: systemd
CVE:
Status comment:


Attachments
Journal after boot with nologin present (141.35 KB, text/plain)
2017-06-18 15:22 CEST, Stig-Ørjan Smelror
Details

Description Stig-Ørjan Smelror 2017-06-12 22:21:28 CEST
Hi.

I don't know what could make this happen, but I can't ssh into my machine after a reboot because a /run/nologin exists.

At first I thought it had something to do with my machine crashing (I overclocked my CPU, nothing to do with MGA), but now it has happened again.

I tried to restart the sshd server. Didn't work. Only thing working is to delete /run/nologin.

$ rpm -qa | grep ssh                                                                                                                                                                             130 ↵
lxqt-openssh-askpass-0.11.1-1.mga6
openssh-clients-7.5p1-2.mga6
libssh2_1-1.7.0-2.mga6
lib64ssh2-devel-1.7.0-2.mga6
openssh-server-7.5p1-2.mga6
autossh-1.4e-2.mga6
lib64ssh-devel-0.7.5-1.mga6
openssh-askpass-common-7.5p1-2.mga6
openssh-askpass-qt5-2.0.3-1.mga6
sshfs-fuse-2.5-4.mga6
lib64ssh4-0.7.5-1.mga6
qemu-block-ssh-2.8.1.1-4.mga6
openssh-7.5p1-2.mga6
lib64ssh2_1-1.7.0-2.mga6

$ journalctl -f
Jun 12 22:04:58 monster sshd[15785]: fatal: Access denied for user stig by PAM account configuration [preauth]
Jun 12 22:05:11 monster sudo[16400]:     stig : TTY=pts/2 ; PWD=/home/stig ; USER=root ; COMMAND=/bin/rm -f /run/nologin
Jun 12 22:05:14 monster sshd[16529]: Accepted publickey for stig from 192.168.10.190 port 44466 ssh2: RSA SHA256:<snip>
Comment 1 Marja Van Waes 2017-06-13 10:16:51 CEST
Does /run/nologin get recreated again on each reboot?

If so, can you trigger it by restarting a specific service, or avoid it by disabling starting a specific service at boot time?

CC: (none) => marja11
Keywords: (none) => NEEDINFO

Comment 2 Stig-Ørjan Smelror 2017-06-13 11:53:06 CEST
Hi.

No, it doesn't happen every time but I've seen it happen once during the last 4 reboots just completed during the past half hour.

I can't see any pattern to it...
Comment 3 Marja Van Waes 2017-06-13 12:05:53 CEST
(In reply to Stig-Ørjan Smelror from comment #2)
> Hi.
> 
> No, it doesn't happen every time but I've seen it happen once during the
> last 4 reboots just completed during the past half hour.
> 
> I can't see any pattern to it...

When you see /run/nologin again, then what is the output of:

  ls --full-time -a /run | grep nologin

and then also attach log.txt that is the result of running, as root:

  journalctl -ab > log.txt
Comment 4 Stig-Ørjan Smelror 2017-06-18 15:21:27 CEST
# ls --full-time -a /run | grep nologin
-rw-r--r--  1 root    root      42 2017-06-18 15:16:05.088834882 +0200 nologin
Comment 5 Stig-Ørjan Smelror 2017-06-18 15:22:25 CEST
Created attachment 9426 [details]
Journal after boot with nologin present
Comment 6 David Walser 2017-06-18 23:59:34 CEST
Actually the nologin thing is supposed to happen at boot until the system is ready for someone to log in.  The bug would be if it never deletes the nologin file when it's ready.
Comment 7 Stig-Ørjan Smelror 2017-06-19 08:30:14 CEST
Hi David.

Yes, I understand. But after over 30 minutes after the reboot, the nologin file is still present.

I noticed this when I tried to connect to this machine from another and got a message about nologin_pam (from the client).
Comment 8 Marja Van Waes 2017-06-19 16:44:24 CEST
(In reply to David Walser from comment #6)
> Actually the nologin thing is supposed to happen at boot until the system is
> ready for someone to log in.  The bug would be if it never deletes the
> nologin file when it's ready.


Stig's nologin file was, going by the time stamp, indeed created at boot time, but the man page of systemd-user-sessions.service says it's created when shutting down :-/ 

> systemd-user-sessions.service is a service that controls user logins
> through pam_nologin(8). After basic system initialization is complete,
> it removes /run/nologin, thus permitting logins. Before system
> shutdown, it creates /run/nologin, thus prohibiting further logins.

Anyway, the attached logs don't show something like:

... systemd[1]: Starting Permit User Sessions...
... systemd[1]: Started Permit User Sessions.

So it seems systemd-user-sessions-service didn't get started

Stig, what does (as root) 

    systemctl status systemd-user-sessions.service

return when the nologin file wasn't removed?

Assigning to our base system maintainers and CC'ing our systemd maintainer.

Assignee: bugsquad => basesystem
Source RPM: (none) => systemd
CC: (none) => mageia

Comment 9 Stig-Ørjan Smelror 2017-07-30 15:25:08 CEST
Hi.

One way to "fix" this remotely was to log in using Nomachine on nx port 4000. I was then able to remove the lock-file and log in with ssh.

As a side note, this means that Nomachine doesn't honor the nologin setting.
Comment 10 Florian Hubold 2017-08-08 22:02:43 CEST
(In reply to Stig-Ørjan Smelror from comment #9)
> One way to "fix" this remotely was to log in using Nomachine on nx port
> 4000. I was then able to remove the lock-file and log in with ssh.

The question is not really how to fix the immediate effect of this, but to find the cause. Please answer the earlier question from Marja so we can get some more details about this.


(In reply to Marja van Waes from comment #8)
> Stig, what does (as root) 
> 
>     systemctl status systemd-user-sessions.service
> 
> return when the nologin file wasn't removed?


Please also add the output of

systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After systemd-user-sessions.service --no-pager

CC: (none) => doktor5000

Comment 11 Stig-Ørjan Smelror 2017-08-09 12:36:30 CEST
@Florian.

I know. I posted the "solution" for others to see if it should happen to anybody else while we're looking to fix the underlying issue.

Will post the output requested the next time I experience this issue again.

Cheers,
Stig
Comment 12 Stig-Ørjan Smelror 2017-08-12 22:55:41 CEST
$ sudo systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After systemd-user-sessions.service --no-pager
Requires=sysinit.target system.slice
Requisite=
Wants=
BindsTo=
PartOf=
Before=prefdm.service session-c8.scope plymouth-quit-wait.service user@978.service getty@tty1.service session-c6.scope session-c3.scope shutdown.target multi-user.target display-manager-failure.service user@1000.service crond.service user-978.slice plymouth-quit.service session-c1.scope systemd-ask-password-wall.service user-1000.slice
After=network.target system.slice systemd-journal-flush.service systemd-journald.socket nss-user-lookup.target remote-fs.target basic.target sysinit.target
Comment 13 Stig-Ørjan Smelror 2017-10-10 23:03:40 CEST
(In reply to Marja van Waes from comment #8)
> (In reply to David Walser from comment #6)
> > Actually the nologin thing is supposed to happen at boot until the system is
> > ready for someone to log in.  The bug would be if it never deletes the
> > nologin file when it's ready.
> 
> 
> Stig's nologin file was, going by the time stamp, indeed created at boot
> time, but the man page of systemd-user-sessions.service says it's created
> when shutting down :-/ 
> 
> > systemd-user-sessions.service is a service that controls user logins
> > through pam_nologin(8). After basic system initialization is complete,
> > it removes /run/nologin, thus permitting logins. Before system
> > shutdown, it creates /run/nologin, thus prohibiting further logins.
> 
> Anyway, the attached logs don't show something like:
> 
> ... systemd[1]: Starting Permit User Sessions...
> ... systemd[1]: Started Permit User Sessions.
> 
> So it seems systemd-user-sessions-service didn't get started
> 
> Stig, what does (as root) 
> 
>     systemctl status systemd-user-sessions.service
> 
> return when the nologin file wasn't removed?
> 
> Assigning to our base system maintainers and CC'ing our systemd maintainer.

Got this again on October 5th. Didn't catch it until now because I haven't had the need to ssh into the machine.

$ ll /run/nologin
-rw-r--r-- 1 root root 42 Oct  5 20:06 /run/nologin

# systemctl status systemd-user-sessions.service
● systemd-user-sessions.service - Permit User Sessions
   Loaded: loaded (/usr/lib/systemd/system/systemd-user-sessions.service; static; vendor preset: ena
   Active: inactive (dead)
     Docs: man:systemd-user-sessions.service(8)
Comment 14 Kevin Bulgrien 2018-03-06 18:15:55 CET
I have started experiencing this condition now for a couple months, and it is very frequent.  As this machine is an important part of my infrastructure, it is a critical failure as now a reference system in a multi-system network is no longer reachable.

$ ls -l /run/nologin 
-rw-r--r-- 1 root root 42 Mar  6 10:11 /run/nologin
$ cat /run/nologin
"System is booting up. See pam_nologin(8)"
$ date
Tue Mar  6 11:10:38 CST 2018
$ uptime
 11:15:10 up  1:04,  0 users,  load average: 0.07, 0.07, 0.08
$ systemctl status systemd-user-sessions.service
● systemd-user-sessions.service - Permit User Sessions
   Loaded: loaded (/usr/lib/systemd/system/systemd-user-sessions.service; static
   Active: inactive (dead)
     Docs: man:systemd-user-sessions.service(8)

CC: (none) => kbulgrien

Comment 15 Kevin Bulgrien 2018-03-06 18:18:21 CET
Including additional information based on what I see was previously requested from other users:

$ systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After systemd-user-sessions.service --no-pager
Requires=system.slice sysinit.target
Requisite=
Wants=
BindsTo=
PartOf=
Before=getty@tty1.service prefdm.service crond.service multi-user.target plymouth-quit-wait.service session-c2.scope display-manager-failure.service systemd-ask-password-wall.service session-c1.scope user@983.service shutdown.target user@1000.service user-983.slice user-1000.slice plymouth-quit.service
After=system.slice systemd-journald.socket remote-fs.target network.target systemd-journal-flush.service sysinit.target nss-user-lookup.target basic.target
Comment 16 Kevin Bulgrien 2018-03-06 18:20:37 CET
I also just noted that this is posted against Cauldron, but I am seeing it in Mageia 6 (Production).
Comment 17 Kevin Bulgrien 2018-03-06 18:28:23 CET
I temporarily workaround the issue with:

$ sudo systemctl start systemd-user-sessions.service

The /run/nologin file is no longer present after doing this, and I can SSH from another system, however, this is not reliable as sometimes I do not have access to the console of the affected system.
Comment 18 Kevin Bulgrien 2018-03-06 18:51:43 CET
This bug appears not to be specific to Mageia.  For example:

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1650634
Comment 19 Kevin Bulgrien 2018-03-06 18:58:22 CET
A lot of related information going back to a 2014 issue here in this RedHat bug that was re-opened and has activity as of December 2017:

https://bugzilla.redhat.com/show_bug.cgi?id=1043212
Stig-Ørjan Smelror 2018-03-06 21:24:22 CET

Version: Cauldron => 6

Comment 20 Stig-Ørjan Smelror 2018-03-06 21:40:01 CET
Kevin.

Thanks for your reports and the link to b.r.c.
I've read through the whole report and it seems like an old issue, but I have no idea if any of the fixes mentioned there are relevant to systemd 230 that MGA6 is running.

Me, personally, haven't had this issue in a while though.

Cheers,
Stig
Comment 21 Aurelien Oudelet 2020-08-02 22:39:12 CEST
This message is a reminder that Mageia 6 is end of life.

Mageia stopped maintaining and issuing updates for Mageia 6. At that time this bug will be closed as OLD (EOL).

Package Maintainer: If you wish for this bug to remain open because you plan to 
fix it in a currently maintained version, simply change the 'version' to a later 
Mageia version prior to Mageia 6's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that we cannot 
be able to fix it before Mageia 6 was end of life.
If you would still like to see this bug fixed and are able to reproduce it against a later version of Mageia, you are encouraged to click on "Version" and change it against that version of Mageia.

Although we aim to fix as many bugs as possible during every release's lifetime, 
sometimes those efforts are overtaken by events. Often a more recent Mageia 
release includes newer upstream software that fixes bugs or makes them obsolete.

--
Mageia Bugsquad

Status: NEW => RESOLVED
Resolution: (none) => OLD
CC: (none) => ouaurelien


Note You need to log in before you can comment on or make changes to this bug.