Bug 17092 - journalctl doesn't limit log size
Summary: journalctl doesn't limit log size
Status: RESOLVED FIXED
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: 5
Hardware: x86_64 Linux
Priority: Normal normal
Target Milestone: ---
Assignee: Colin Guthrie
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-11-05 20:42 CET by Reinout van Schouwen
Modified: 2019-04-09 10:04 CEST (History)
2 users (show)

See Also:
Source RPM: systemd-217-11.1.mga5.src.rpm
CVE:
Status comment:


Attachments

Description Reinout van Schouwen 2015-11-05 20:42:54 CET
Description of problem:
I got warnings about disk space on my root partition.
The cause was /var/log/journal with gigabytes of logs.
 
All options in /etc/systemd/journald.conf are commented out so journalctl doesn't keep a certain maximum size. We should be smarter about this so that logs don't fill up the root partition.

Version-Release number of selected component (if applicable):
217.11.1

How reproducible:
Observed it on my own PC

Steps to Reproduce:
1. Use pc normally, watch logs grow slowly and steadily
2.
3.


Reproducible: 

Steps to Reproduce:
Comment 1 David Walser 2015-11-06 15:18:21 CET
I thought systemd would only allow it to use a certain % of disk space on its partition.  If it's really unbounded, that's a problem.

Assignee: bugsquad => mageia

Comment 2 Colin Guthrie 2015-11-06 17:21:35 CET
The commented out options show the built in defaults - do not assume that this means there are no defaults!!

Please see the various journalctl options to list the storage usage (not at a computer right now so cannot look up the commands) and the defaults. Note that when journals starts it logs a line showing the current calculations.
Comment 3 Reinout van Schouwen 2015-11-09 08:27:59 CET
(In reply to Colin Guthrie from comment #2)
> The commented out options show the built in defaults - do not assume that
> this means there are no defaults!!

Right, but whatever the defaults are, they allowed the journal to grow to > 1GB whereas the remaining space on my root partition was < 1GB.
Comment 4 Colin Guthrie 2015-11-09 10:44:45 CET
So in order to investigate this any further, I'll need information about your system. What is your partition size, what is the free space etc. A listing of the files and folders inside /var/log/journal would also be useful (including their full path, dates and sizes). Also the output from "journalctl --disk-usage" (as root) and the output from "journalctl -b MESSAGE_ID=ec387f577b844b8fa948f33cad9a75e6" (also as root).

As you'll already have read in journald.conf man page, the default is to use 10% of disk space for log data, but also to leave 15% free disk space. Journal files should then be rotated (and dropped) accordingly automatically. This kinda implies that the partition is ~10Gb in size (which seems kinda normal), but the fact they do not leave ~1.5Gb free is the troubling part. Of course depending on the actual size of the /var partition, the approximate sizes you give in your report could make this all work. If, however, the sizes really do not compute, I'm wondering if there are stale, non-journal files in the /var/log/journal tree, that are messing up the stats (hence why the --disk-usage output would be useful). This can happen if e.g. the machine-id has somehow been changed or you use nspawn with --link-journal=host mode.

Anyway, it's very hard to tell for now without more info as the report assumes a problem but doesn't provide any size or metrics to verify it in any way. Once you provide this further information, we can look into whether there is a bigger problem.

Cheers!
Comment 5 Reinout van Schouwen 2015-11-09 21:59:54 CET
Yes, I should've documented it better. Unfortunately it's too late for that now, because I already deleted the files that were causing the problem. 

Anyway, here's some of the info you asked for (everything is looking pretty normal ATM):

$ ls -lh /var/log/journal/1d86337a64994802ad13fa599557d420/
totaal 136M
-rw-r-----  1 root systemd-journal  56M nov  6 14:01 system@000523ded7b5d907-f77e6791156d44db.journal~
-rw-r-----  1 root systemd-journal  24M nov  9 21:47 system.journal
-rw-r-----+ 1 root systemd-journal  16M nov  9 21:46 user-1000.journal
-rw-r-----+ 1 root systemd-journal  32M nov  9 21:10 user-1001.journal
-rw-r-----+ 1 root systemd-journal 8,0M nov  9 20:57 user-65534.journal
[reinout@linux ~]$ sudo journalctl --disk-usage
Journals take up 136.1M on disk.

[reinout@linux ~]$ sudo journalctl -b MESSAGE_ID=ec387f577b844b8fa948f33cad9a75e6
-- Logs begin at ma 2015-10-19 15:59:38 CEST, end at ma 2015-11-09 21:56:41 CET. --
nov 09 14:04:41 localhost systemd-journal[588]: Runtime journal is using 8.0M (max allowed 355.3M, trying to leave 532.9M free of 3.4G available â current limit 355.3M).
nov 09 14:04:41 localhost systemd-journal[588]: Permanent journal is using 128.0M (max allowed 1.2G, trying to leave 1.0G free of 2.0G available â current limit 1.0G).

I don't have a separate /var partition:

[reinout@linux ~]$ df -h
Bestandssysteem        Grootte Gebruikt Besch Geb% Aangekoppeld op
devtmpfs                  3,5G        0  3,5G   0% /dev
tmpfs                     3,5G     824K  3,5G   1% /dev/shm
tmpfs                     3,5G     1,3M  3,5G   1% /run
/dev/dm-0                  13G     8,9G  2,1G  82% /
tmpfs                     3,5G        0  3,5G   0% /sys/fs/cgroup
tmpfs                     3,5G      92K  3,5G   1% /tmp
/dev/mapper/crypt_sda6    110G      42G   67G  39% /home
tmpfs                     711M      16K  711M   1% /run/user/1001
tmpfs                     711M      16K  711M   1% /run/user/1000
tmpfs                     711M      12K  711M   1% /run/user/984
Comment 6 Marja Van Waes 2018-04-17 17:24:55 CEST
@ Reinout

I assume this got fixed, because it is very long ago that I heard about journalctl logs eating the last bit of space on /

Please reopen if I'm wrong and change the Version (near the top, at the left) to Mageia 6 or Cauldron, whichever you use.

Status: NEW => RESOLVED
Resolution: (none) => FIXED
CC: (none) => marja11

Sunny yadav 2019-04-08 19:07:53 CEST

CC: (none) => arkr17997


Note You need to log in before you can comment on or make changes to this bug.