Bug 18290 - / and /usr partitions are mounted as read only ("ro") right after the system boots
Summary: / and /usr partitions are mounted as read only ("ro") right after the system ...
Status: NEW
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: Cauldron
Hardware: x86_64 Linux
Priority: Normal normal
Target Milestone: ---
Assignee: Colin Guthrie
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-04-28 20:44 CEST by Shlomi Fish
Modified: 2020-04-05 04:00 CEST (History)
2 users (show)

See Also:
Source RPM: systemd-229-3.mga6.src.rpm
CVE:
Status comment:


Attachments
Output of journalctl -ab after being compressed with XZ. (27.55 KB, application/octet-stream)
2016-05-01 10:29 CEST, Shlomi Fish
Details
My desktop machine's /etc/fstab file (731 bytes, text/plain)
2016-05-01 18:29 CEST, Shlomi Fish
Details
new journalctl -ab output with no repeated BOOT_IMAGE (21.46 KB, application/x-xz)
2020-02-21 09:21 CET, Shlomi Fish
Details

Description Shlomi Fish 2016-04-28 20:44:06 CEST
Description of problem:

Right after my system (running Cauldron x86-64) boots my / and /usr partitions are mounted as read only. /home and other partitions are fine. After that, I am able to login as an underprivileged user, sudo -i into root and do "mount -o remount,rw /" and "mount -o remount,rw /usr" and the system appears to work fine after that.

I can provide more info about my setup and I'm on a Core i3 with a 2TB hard disk machine.

Version-Release number of selected component (if applicable):

Cauldron
Comment 1 Marja Van Waes 2016-04-30 16:14:25 CEST
@ Colin

What is needed to debug this? 

@ Shlomi

Unless Colin says that's _not_ needed, I guess it won't harm to attach the output of 
    journalctl -ab

CC: (none) => marja11
Assignee: bugsquad => mageia

Comment 2 Shlomi Fish 2016-05-01 10:29:56 CEST
Created attachment 7727 [details]
Output of journalctl -ab after being compressed with XZ.

This is the output of journalctl -ab per Marja's request.
Comment 3 Marja Van Waes 2016-05-01 13:26:45 CEST
(In reply to Shlomi Fish from comment #2)
> Created attachment 7727 [details]
> Output of journalctl -ab after being compressed with XZ.
> 

Thanks, Shlomi .... maybe /etc/fstab would be good to have, too.

For the root partition, it all looked similar to the journalctl -ab output for my root partition (without the issue). For /usr there's nothing to compare, because I don't have that partition

I'm wondering what this in your log means:
May 01 10:09:46 telaviv1.shlomifish.org systemd[1]: usr.mount: Unit is bound to inactive unit dev-disk-by\x2duuid-7963d799\x2d5f21\x2d
4e34\x2d9687\x2d142a5bc764be.device. Stopping, too.
Comment 4 Marja Van Waes 2016-05-01 13:44:12 CEST
ah, wait, now I do see a difference: after initially being mounted ro, my root partition is remounted after one second.... for you that takes 3 minutes 

May 01 10:12:59 telaviv1.shlomifish.org kernel: EXT4-fs (sda6): re-mounted. Opts: acl

(Or maybe that was when you manually remounted.. didn't check
Comment 5 Shlomi Fish 2016-05-01 18:27:46 CEST
(In reply to Marja van Waes from comment #4)
> ah, wait, now I do see a difference: after initially being mounted ro, my
> root partition is remounted after one second.... for you that takes 3
> minutes 
> 
> May 01 10:12:59 telaviv1.shlomifish.org kernel: EXT4-fs (sda6): re-mounted.
> Opts: acl
> 

that may have indeed been my manual remounting.

> (Or maybe that was when you manually remounted.. didn't check
Comment 6 Shlomi Fish 2016-05-01 18:29:00 CEST
Created attachment 7731 [details]
My desktop machine's /etc/fstab file

This is my fstab file.
Comment 7 David Walser 2016-06-07 14:24:26 CEST
What is your "lsblk -o +UUID" output?

You need to make sure the UUIDs in your /etc/fstab are correct.

/ is always read-only right after boot, it's only remounted rw during the boot process.  This can sometimes not happen if there's a bad service script that doesn't exit and makes the boot process get stuck.  As for /usr being ro, that's weird.
Comment 8 Shlomi Fish 2016-06-07 14:50:48 CEST
(In reply to David Walser from comment #7)
> What is your "lsblk -o +UUID" output?
> 
> You need to make sure the UUIDs in your /etc/fstab are correct.
> 

See this:

« QUOTE »
root@telaviv1:~$ lsblk -o +UUID
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT UUID
sda       8:0    0   1.8T  0 disk            
ââsda1    8:1    0    16G  0 part /boot      4a5d4de6-01cb-4f22-9a9e-f36f6a3867d0
ââsda2    8:2    0     1K  0 part            
ââsda5    8:5    0   1.4T  0 part /home      cd407dc5-f6b9-4e60-ab3b-b9d012f899c6
ââsda6    8:6    0    32G  0 part /          d320c338-58a2-4aa6-9d47-d2c0bc60524b
ââsda7    8:7    0  61.8G  0 part [SWAP]     9be1d836-b396-4fc3-970d-f9b4d82140e8
ââsda8    8:8    0  86.2G  0 part /var       21a21e54-a2e2-4b4b-b764-6c8db176cbec
ââsda9    8:9    0  92.4G  0 part /usr       7963d799-5f21-4e34-9687-142a5bc764be
ââsda10   8:10   0 111.8G  0 part            3d02a130-d1e1-452f-b701-8c7ce212c2a4
ââsda11   8:11   0  44.9G  0 part            e16c253a-9f1f-4fe5-92cc-e8ef41a75a0e
root@telaviv1:~$ cat /etc/fstab
# Entry for /dev/sda6 :
UUID=d320c338-58a2-4aa6-9d47-d2c0bc60524b / ext4 acl,noatime 1 1
# Entry for /dev/sda1 :
UUID=4a5d4de6-01cb-4f22-9a9e-f36f6a3867d0 /boot ext3 acl,noatime 1 2
# Entry for /dev/sda5 :
UUID=cd407dc5-f6b9-4e60-ab3b-b9d012f899c6 /home ext4 acl,noatime 1 2
/dev/sr0 /media/cdrom auto umask=0,users,iocharset=utf8,noauto,ro,exec 0 0
# Entry for /dev/sda10 :
UUID=3d02a130-d1e1-452f-b701-8c7ce212c2a4 /mnt/debian ext3 defaults 1 2
none /proc proc defaults 0 0
# Entry for /dev/sda9 :
UUID=7963d799-5f21-4e34-9687-142a5bc764be /usr ext4 acl,noatime 1 2
# Entry for /dev/sda8 :
UUID=21a21e54-a2e2-4b4b-b764-6c8db176cbec /var ext4 acl,noatime 1 2
# Entry for /dev/sda7 :
LABEL=sf_unix_swap swap swap defaults 0 0
root@telaviv1:~$ 

« / QUOTE »

Seems fine and I still have to remount as rw.


> / is always read-only right after boot, it's only remounted rw during the
> boot process.  This can sometimes not happen if there's a bad service script
> that doesn't exit and makes the boot process get stuck.  As for /usr being
> ro, that's weird.
Comment 9 Shlomi Fish 2016-06-11 18:56:39 CEST
This is still happening now (11 June 2016 - latest v6/Cauldron) and is still annoying the heck out of me. Does anyone have an idea?
Comment 10 Shlomi Fish 2019-06-07 23:07:57 CEST
I was able to fix this issue by adding these files:

```
root@telaviv1:~ # cat /etc/systemd/system/shlomif--afterboot.service 
[Unit]
Description=after boot cleanups

[Service]
Type=oneshot
ExecStart=/root/after-boot-rw.bash

[Install]
WantedBy=multi-user.target
root@telaviv1:~ # cat /r
root/ run/  
root@telaviv1:~ # cat /root/after-boot-rw.bash 
#!/bin/sh
mount -o remount,rw /sys/fs/cgroup
mount -o remount,rw /
mount -o remount,rw /usr
# /etc/init.d/mandrake_everytime
# cp -f /etc/resolv.conf.orig /etc/resolv.conf
# service postfix restart
# service httpd restart
# systemctl restart udisks2.service
root@telaviv1:~ # 
```
Comment 11 Dave Hodgins 2020-02-20 23:05:11 CET
Just saw this mentioned in konversation. The only things I see that seem strange
is that the kernel command line has BOOT_IMAGE specified twice, with the second
being BOOT_IMAGE=linux, and there's the message ...
systemd[1]: usr.mount: Unit is bound to inactive unit dev-disk-by\x2duuid-7963d799\x2d5f21\x2d4e34\x2d9687\x2d142a5bc764be.device. Stopping, too.

On my Mageia 7 system, the cmdline has
# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.5.4-desktop-1.mga7 root=UUID=486a2c6a-c5e3-4028-bde4-0a10bdf5dbb7 ro noiswmd rd.luks=0 rd.lvm=0 rd.md=0 rd.dm=0 ipv6.disable=1 audit=0

ipv6 is not available for me, so I've disabled it. I also don't use raid or
lvm on this install, so disable checking for those to speed up the boot.
The audit=0 is there to stop the useless messages from adding clutter to the
journal.

Try removing the BOOT_IMAGE=linux option.
Also, what's the output of "systemctl status usr.mount"?

CC: (none) => davidwhodgins

Comment 12 Shlomi Fish 2020-02-21 08:43:27 CET
(In reply to Dave Hodgins from comment #11)
> Just saw this mentioned in konversation. The only things I see that seem
> strange
> is that the kernel command line has BOOT_IMAGE specified twice, with the
> second
> being BOOT_IMAGE=linux, and there's the message ...
> systemd[1]: usr.mount: Unit is bound to inactive unit
> dev-disk-by\x2duuid-7963d799\x2d5f21\x2d4e34\x2d9687\x2d142a5bc764be.device.
> Stopping, too.
> 
> On my Mageia 7 system, the cmdline has
> # cat /proc/cmdline
> BOOT_IMAGE=/vmlinuz-5.5.4-desktop-1.mga7
> root=UUID=486a2c6a-c5e3-4028-bde4-0a10bdf5dbb7 ro noiswmd rd.luks=0 rd.lvm=0
> rd.md=0 rd.dm=0 ipv6.disable=1 audit=0
> 
> ipv6 is not available for me, so I've disabled it. I also don't use raid or
> lvm on this install, so disable checking for those to speed up the boot.
> The audit=0 is there to stop the useless messages from adding clutter to the
> journal.
> 
> Try removing the BOOT_IMAGE=linux option.

Thanks, I'll try.

> Also, what's the output of "systemctl status usr.mount"?

It is:

```
● usr.mount - /usr
   Loaded: loaded (/etc/fstab; generated)
   Active: active (mounted) since Thu 2020-02-20 10:38:37 IST; 23h ago
    Where: /usr
     What: /dev/sda9
     Docs: man:fstab(5)
           man:systemd-fstab-generator(8)
   Memory: 0B
   CGroup: /system.slice/usr.mount

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
```
Comment 13 Shlomi Fish 2020-02-21 09:21:03 CET
Created attachment 11513 [details]
new journalctl -ab output with no repeated BOOT_IMAGE

new journalctl -ab output with no repeated BOOT_IMAGE. There is still a mounted-as-ro issue after I move away the /etc/systemd/system file.
Comment 14 Dave Hodgins 2020-02-22 06:08:01 CET
Still has the very strange msg ...
usr.mount: Unit is bound to inactive unit dev-sda9.device. Stopping, too.
What's the output of (as root)
systemctl status dev-sda9.device

Also the msg ...
systemd[581]: usr.mount: Failed to connect stdout to the journal socket, ignoring: No such file or
 directory
is strange. No idea off hand about that one, though it may be tied to the first
message, or may be because the journal hasn't started yet.
Comment 15 Dave Hodgins 2020-04-05 04:00:39 CEST
Is this still happening? If so, what's the output of
systemctl status dev-sda9.device
df -h
df -i

Note You need to log in before you can comment on or make changes to this bug.