| Summary: | /tmp as a tmpfs filesystem full does not release space when deleting files | ||
|---|---|---|---|
| Product: | Mageia | Reporter: | Pierre Fortin <pfortin> |
| Component: | RPM Packages | Assignee: | Mageia Bug Squad <bugsquad> |
| Status: | RESOLVED INVALID | QA Contact: | |
| Severity: | normal | ||
| Priority: | Normal | CC: | davidwhodgins, lewyssmith |
| Version: | Cauldron | ||
| Target Milestone: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Source RPM: | CVE: | ||
| Status comment: | |||
This problem probably comes from the use of 'tmpfs' for /tmp, rather than /tmp per se. To test that would mean re-running things with /tmp on a block device filesystem. https://docs.kernel.org/filesystems/tmpfs.html https://wiki.archlinux.org/title/tmpfs "Tmpfs is a file system which keeps all of its files in virtual memory" "The size limit of a ramfs filesystem is how much memory you have available, and so care must be taken if used so to not run out of memory" "their contents are automatically cleared upon reboot." "If you unmount a tmpfs instance, everything stored therein is lost" The last point suggests that unmounting then re-mounting the filesystem should unblock things without having to reboot. How much memory does your system have? Do you have an /etc/fstab entry for this filesystem? If so - please post it. I wonder whether the problem might be due to the almost 64G size reached, hitting a new limit? This could be tested limiting the filesystem size to something smaller, and seeing whether filling that also resisted file deletion. You said you also deleted "a few others" (files), apparently in vain. Another test could be to first create a very large file before filling the filesystem with another one; then try deleting the first file to see whether that was honoured. All this is theoretical. Pierre, we do not have the time to try the various things I suggest - and that other people might. You may have discovered a flaw in tmpfs; or (worse) any other filesystem filling up, but this must happen routinely if rarely. CC:
(none) =>
lewyssmith Operating System: Mageia 10 KDE Plasma Version: 5.27.8 KDE Frameworks Version: 5.110.0 Qt Version: 5.15.7 Kernel Version: 6.5.3-server-1.mga10 (64-bit) Graphics Platform: X11 Processors: 20 × 12th Gen Intel® Core™ i7-12700K Memory: 125.5 GiB of RAM Graphics Processor: AMD Radeon RX 6600 XT Manufacturer: Dell Inc. Product Name: XPS 8950 $ df Filesystem Size Used Avail Use% Mounted on devtmpfs 63G 0 63G 0% /dev tmpfs 63G 12M 63G 1% /dev/shm tmpfs 63G 12M 63G 1% /run /dev/nvme0n1p2 96G 51G 40G 56% / efivarfs 192K 113K 75K 61% /sys/firmware/efi/efivars tmpfs 63G 337M 63G 1% /tmp /dev/nvme0n1p1 300M 568K 299M 1% /boot/EFI /dev/nvme0n1p4 1.7T 350G 1.3T 22% /home /dev/sdb2 5.5T 4.8T 355G 94% /mnt/6A /dev/sda2 5.5T 4.3T 915G 83% /mnt/6C /dev/sdf2 5.5T 4.9T 344G 94% /mnt/6B /dev/sdd1 1.8T 1.2T 576G 67% /mnt/a1 /dev/sdc2 5.5T 2.8T 2.4T 55% /mnt/6D /dev/sde1 17T 12T 4.4T 72% /mnt/db /dev/nvme1n1p1 3.6T 1.3T 2.2T 38% /mnt/work tmpfs 13G 216K 13G 1% /run/user/1000 $ grep "^UUID" /etc/fstab UUID=1114d372-ab53-4a1b-9cf9-d2df52138f39 / ext4 noatime,acl 1 1 UUID=0415-DFFB /boot/EFI vfat iocharset=utf8,umask=000 0 0 UUID=556f86ae-60a3-4e7c-9e79-7b460b9206bd /home ext4 noatime,acl 1 2 UUID=447b948c-ee26-4893-a8c7-9f19ec63777f swap swap defaults 0 0 UUID=f508f72e-5252-4199-846c-c422a5bca1de /mnt/work ext4 noatime,acl,nofail 0 0 UUID=94cfdafe-9120-4d1c-853e-b8eec6944c6c /mnt/a1 ext4 noatime,acl,nofail 0 0 UUID=a8975c7b-2c7b-436e-9d6a-e1dd1b3deca0 /mnt/db ext4 noatime,acl,nofail 0 0 UUID=a1ade5b2-24c4-4d34-85f9-16a25fc142cf /mnt/6A ext4 noatime,acl,nofail 0 0 UUID=bd9ae805-0b8c-4d65-83ce-87c88f193d2c /mnt/6B ext4 noatime,acl,nofail 0 0 UUID=d80691e2-13f2-4298-a243-d770469acf02 /mnt/6C ext4 noatime,acl,nofail 0 0 UUID=b72e3e29-2d14-4978-bd82-e6833686c7af /mnt/6D ext4 noatime,acl,nofail 0 0 $ lsblk -o NAME,FSTYPE,LABEL,UUID,FSSIZE,MOUNTPOINT,MODEL NAME FSTYPE LABEL UUID FSSIZE MOUNTPOINT MODEL sda Expansion SW ├─sda1 vfat EFI 67E3-17ED └─sda2 ext4 6C d80691e2-13f2-4298-a243-d770469acf02 5.4T /mnt/6C sdb Expansion SW ├─sdb1 vfat EFI 67E3-17ED └─sdb2 ext4 6A a1ade5b2-24c4-4d34-85f9-16a25fc142cf 5.4T /mnt/6A sdc Expansion SW ├─sdc1 vfat EFI 67E3-17ED └─sdc2 ext4 6D b72e3e29-2d14-4978-bd82-e6833686c7af 5.4T /mnt/ftp sdd ST2000DM008-2FR102 └─sdd1 ext4 94cfdafe-9120-4d1c-853e-b8eec6944c6c 1.8T /mnt/a1 sde ST18000NM000J-2TV103 └─sde1 ext4 DB a8975c7b-2c7b-436e-9d6a-e1dd1b3deca0 16.3T /mnt/db sdf Expansion SW ├─sdf1 vfat EFI 67E3-17ED └─sdf2 ext4 6B bd9ae805-0b8c-4d65-83ce-87c88f193d2c 5.4T /mnt/6B sdg WDC WD7500BPKX-75HPJT0 ├─sdg1 vfat ESP B679-C70B ├─sdg2 ├─sdg3 ntfs OS 38D694C0D6947FB4 ├─sdg4 ntfs WINRETOOLS 727CCA7B7CCA399D ├─sdg5 ntfs Image 8A62CA8062CA710D └─sdg6 ntfs DELLSUPPORT 94FA9DBAFA9D9956 sr0 HL-DT-ST DVD+/-RW GU90N nvme0n1 Seagate FireCuda 530 ZP2000GM30013 ├─nvme0n1p1 vfat 0415-DFFB 299.4M /boot/EFI ├─nvme0n1p2 ext4 1114d372-ab53-4a1b-9cf9-d2df52138f39 95.6G / ├─nvme0n1p3 swap 447b948c-ee26-4893-a8c7-9f19ec63777f [SWAP] └─nvme0n1p4 ext4 556f86ae-60a3-4e7c-9e79-7b460b9206bd 1.7T /home nvme1n1 Seagate FireCuda 530 ZP4000GM30023 └─nvme1n1p1 ext4 work f508f72e-5252-4199-846c-c422a5bca1de 3.6T /mnt/work Thank you for all that. So the size of the main 'tmpfs' filesystem looks to be the default 1/2 size of real memory. (I ignore the second instance, ?). I do not see why you grepped for UUID in fstab; I suppose 'tmpfs' would have shown something *if* there is an entry - which is apparently not necessary except when defining specific properties; so there may not be a tmpfs entry. I would not expect lsblk to show it because - it is not a block device... We need to establish whether the problem of file deletion not freeing space is: - /tmp filling up (say on a physical block device) or - tmpfs filling up. Oh dear, who will ever have the time? CC'ing Dave, in case he is drawn to look at this despite his retirement. CC:
(none) =>
davidwhodgins No idea how I missed for all these years that /tmp was not on disk... That explains a LOT -- THANKS! (too busy using vs tinkering) fstab contains a LOT of comments; or did... had to use an old backup to find the lsblk syntax I use. Rather than use up 1/2 memory which I need for big jobs, I'd rather use part of my 4TB NVMe drive which is very fast. I created /mnt/work/tmp and would like to use that as /tmp. ln -s /mnt/work/tmp /tmp would do the job; nut I don't want to damage the system because /tmp contains a lot of stuff placed there on bootup. Is it as simple as adding: TEMP=/mnt/work/tmp to /etc/environment and reboot? My gut feels like this may be reset to /tmp(tmpfs) at boot...? Hope Dave is doing well -- I "retired" Sept/99; but never stopped... ;/ It just occurred to me that the issue may involve SWAP... with /tmp (tmpfs) filling and crowding out main memory; swapping kicks in; I've seen all of swap (32GB) used on some days. I would create a new ext4 partition on the nvme drive using diskdrake and set it to use /tmp as the mountpoint. After it updates fstab, remove or comment out the existing /tmp on tmpfs from fstab if still present, and then reboot. There are other ways it can be done, but that is the simplest. As far as using environment variables, different applications look at different variables. I currently have ... GCONF_TMPDIR=/home/dave/tmp KDETMP=/home/dave/tmp KDEVARTMP=/home/dave/tmp SCREENDIR=/home/dave/tmp TEMP=/home/dave/tmp TMPDIR=/tmp TMP=/tmp Another option ... If you don't want another partition, remove the fstab entry for /tmp on tmpfs, then "telinit 1" to close stop everything that's likely to be using /tmp now, unmount /tmp and create a symlink for /tmp pointing to a directory on the filesystem where you want it. Thanks for popping back. Ye Gods - I have just noticed that my own system is using tmpfs for /tmp (among other things)! Filesystem Size Used Avail Use% Mounted on tmpfs 3.8G 26M 3.8G 1% /tmp 'ls -l /' highlights 'tmp' drwxrwxrwt 15 root root 360 Hyd 12 21:11 tmp/ t (sticky) The last special permission has been dubbed the "sticky bit." This permission does not affect individual files. However, at the directory level, it restricts file deletion. Only the owner (and root) of a file can remove the file within that directory. A common example of this is the /tmp directory... [as above] The permission set is noted by the lowercase t, where the x would normally indicate the execute privilege Was the ownership correct for the file that did not go? In reply to comment 4, I agree with comment 5: just create a fast physical partition for tmp, and an /etc/fstab entry to mount it on /tmp. (In reply to Pierre Fortin from comment #4) > It just occurred to me that the issue may involve SWAP... with /tmp (tmpfs) > filling and crowding out main memory; swapping kicks in; I've seen all of > swap (32GB) used on some days. > Rather than use up 1/2 memory which I need for big jobs I think that the URLs I gave in comment 1 do mention swap. And they certainly show how to create an fstab line which specifically defines how much memory you want to give tmpfs - in your case, limit it. Lewis, Have you managed to get /tmp out of RAM? Using a laptop to test out moving /tmp to disk... it keeps coming up as tmpfs. The oldest backup copy of /etc/fstab on the main system, is dated Feb 25, 2022 which is when I acquired it; and none of the various backups ever had a /tmp entry... I did as Dave suggested (telinit 1, ln -s /home/tmp /tmp, reboot); but /tmp is still using tmpfs... Also have /etc/environment setup as per Dave's comment 6. Dave wrote: "If you don't want another partition, remove the fstab entry for /tmp on tmpfs, then "telinit 1" to close stop everything that's likely to be using /tmp now, unmount /tmp and create a symlink for /tmp pointing to a directory on the filesystem where you want it." I want /tmp pointing to /home/tmp where there is lots of free space on both systems (I expect to manage the growth of /tmp; but something is still forcing /tmp to tmpfs... I really do not want RAM used for /tmp; I need it for processing huge files. Tried various suggestions from search results; but nothing works for me.. Any other ideas? Found that /tmp now gets mounted by /usr/lib/systemd/system/tmp.mount So as well as deleting or commenting out the fstab entry for /tmp, it's necessary to run "systemctl mask tmp.mount", but it's probably a bad idea. As per "man file-hierarchy" /tmp is for large files, not large ones. Also, some applications may not work properly if the directories and/or files have been left from a prior boot in /tmp at boot since they expect them to be deleted on reboot. It would be better to examine how the applications putting the data in /tmp are set up, and configure them to use an on disk directory or file. Thanks Dave!! Great info! (In reply to Dave Hodgins from comment #9) > Found that /tmp now gets mounted by /usr/lib/systemd/system/tmp.mount > > So as well as deleting or commenting out the fstab entry for /tmp, it's > necessary to run "systemctl mask tmp.mount", but it's probably a bad idea. > As per "man file-hierarchy" /tmp is for large files, not large ones. "/tmp is for [small] files" :) This may have been the case by default because this system (since acquired) has never had /tmp in fstab. > Also, some applications may not work properly if the directories and/or > files have been left from a prior boot in /tmp at boot since they expect > them to be deleted on reboot. From what I see, /tmp usually requires less than 1MB. For such a tiny requirement, I get: Filesystem Size Used Avail Use% Mounted on tmpfs 63G 492M 63G 1% /tmp Then, writing python scripts for team members running Linux|MacOS|Windows, I was using os.environ['TMPDIR'] which returns /tmp ; so does os.environ['TMP']... > It would be better to examine how the applications putting the data in /tmp > are set up, and configure them to use an on disk directory or file. The way I'm now considering is to set TMPDIR=/home/tmp at bootup so that: TMP=/tmp for the system, and I use TMPDIR for my apps. "Solution" appears to be: /etc/environment: TEMPDIR=/mnt/work/tmp which is on 4TB NVMe and leaving TMP alone... Replied directly by mistake. Adding the reply here so others who monitor this
see the reply.
Keep in mind that the size of a tmpfs filesystem is the maximum that file system
can use. If it is only using a few MB, then the rest is still available for other
things that use memory.
On my 16GB m8 install ...
[root@x3 ~]# df|grep tmp
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.8G 0 7.8G 0% /dev/shm
tmpfs 7.8G 3.0M 7.8G 1% /run
tmpfs 7.8G 44K 7.8G 1% /tmp
tmpfs 1.6G 236K 1.6G 1% /run/user/500
[root@x3 ~]# free -m
total used free shared buff/cache available
Mem: 15955 4978 3302 474 7675 10173
Swap: 32761 0 32760
So all of the mounts that use tmpfs share the 7.8G maximum size, but since 99%
of that maximum size is not used, the total used memory is only around 4.9G, most
of that is used by web browsers.
Using /etc/environment works, but $HOME will not have been set yet so would
have to be hard coded. Using a file in /etc/profile.d works or ~/.bash_profile
or ~/.profile allows $HOME to be used.
I've renamed .bash_profile to .profile since at one point (Mageia 4?) there was
a problem where .bash_profile wasn't sourced by at least one of the display
managers, but .profile was always sourced (and still is).
$ grep TMP .profile
export TMPDIR=$HOME/tmp
export GCONF_TMPDIR=$TMPDIR
export KDETMP=$TMPDIR
export KDEVARTMP=$TMPDIR
export TEMP=$TMPDIR
export TMP=$TMPDIR
I don't remember which TMP variable is used by what. I just set all of them
that I could find to the same value.
My problem with /tmp in tmpfs is that at best, the growth will be 2X; at worst, maybe 3x or 4X...To wit, my original comment shows 64GB was not enough... :/ From my hopefully somewhat reliable recollection; /tmp is usually barely used (under 1MB) unless I unzip a few humongous files into it. So, I'm hopeful that picking TMPDIR=/home/tmp will suffice and leave RAM for what I need it for... Per bit, RAM is VERY expensive compared to NVMe sticks which are quite fast and have no seek delays like with platters... ARGH!!! /etc/environment contains TMPDIR=/home/tmp; but the system ignores it... I added TEMPDIR=/home/tmp and that works; but it means that I can't use python's os.environ['TMPDIR'] or os.environ['TMP'] which are cross-platform; usually, it's Windows that costs me extra code/complexity... ;P Even https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir can't get past /tmp: On Windows, the directories C:\TEMP, C:\TMP, \TEMP, and \TMP, in that order. On all other platforms, the directories /tmp, /var/tmp, and /usr/tmp, in that order. A case of the kernel and app(python) folk not aware of each other's implementation details... Thanks Dave; appreciate you jumping back in... BTW: is that you in western NC? Best regards, Pierre I'm in London, Ontario, Canada. Guessed wrong.. LOL. I grew up in North Bay. Last 20 years in Ottawa before moving to Dallas, TX in '92 then central NC in '95. Cheers! Closing the bug as invalid since it wasn't a problem fixed by a change to a Mageia package, just about how /tmp is mounted. Status:
NEW =>
RESOLVED About the releasing of the space. In an m9 vb guest ... [dave@x9v ~]$ dd if=/dev/urandom of=/tmp/myfile bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 5.36968 s, 200 MB/s [dave@x9v ~]$ df|grep tmp devtmpfs 2.0G 0 2.0G 0% /dev tmpfs 2.0G 0 2.0G 0% /dev/shm tmpfs 2.0G 1.2M 2.0G 1% /run tmpfs 2.0G 1.1G 941M 53% /tmp tmpfs 393M 108K 393M 1% /run/user/1000 [dave@x9v ~]$ rm -f /tmp/myfile [dave@x9v ~]$ df|grep tmp devtmpfs 2.0G 0 2.0G 0% /dev tmpfs 2.0G 0 2.0G 0% /dev/shm tmpfs 2.0G 1.2M 2.0G 1% /run tmpfs 2.0G 4.0K 2.0G 1% /tmp tmpfs 393M 108K 393M 1% /run/user/1000 So the space is released. Most likely in the case where the free space wasn't released, the file was still open. After a file gets deleted the space doesn't become available for reuse until all programs that have it open have closed. Definite possibility... IIRC, it was PostgreSQL outputting a very large results file. Resolution:
INVALID =>
FIXED A bug report should only be closed as fixed if Mageia changed something to fix it. In this case it's closed as invalid since it isn't about a bug in a Mageia package, or a Mageia website. Resolution:
FIXED =>
INVALID Dave, I've noticed this happened at least once before... I did NOT change the resolution. I'm now suspecting that adding a comment after a report is closed may be what is doing this... Ah... now I see the problem as I'm about to save this comment... To the left of Save Changes, the Status options are _defaulted_ to: RESOLVED FIXED so a commenter has to notice this and change it back as I am doing now... This is a bug/flaw/setup issue in bugzilla... Strange. That doesn't happen for me, but I have extra privileges on bugzilla, I've just added editbugs authority for your id, so that may make a difference. Your other id for the email starting with pf@ instead of pfortin@ already had the editbugs authority. That used to be a default, but due to spammers abusing bugzilla the settings were made more restrictive. P.S. Searching the basic problem of "How to stop /tmp being automatically assigned to tmpfs" (for it to be on a real device rather than in memory), these links are worth a look: https://unix.stackexchange.com/questions/636843/how-to-disable-override-automatic-mounting-of-tmpfs-to-tmp-by-systemd https://bbs.archlinux.org/viewtopic.php?id=148380 https://groups.google.com/g/alt.os.linux/c/zaOqq7CgBy0 https://forums.linuxmint.com/viewtopic.php?t=358346 Some go back 10y. Dave commented then: - If /tmp has an entry in /etc/fstab, then systemd will not mount it as a tmpfs filesystem. This implies giving /tmp its own partition. - As root run "systemctl mask tmp.mount". The command systemctl status tmp.mount prior to running the mask command will show it's a static, i.e. builtin to systemd service, so just disabling it won't work, it has to be masked to stop it from starting. Other similar advice was: - 'systemctl mask tmp.mount' should do the trick for you: this will disable the unit, symlinking it to /dev/null in /etc, which will prevent its being re-enabled on upgrade. It will also prevent manually starting the unit - 'systemctl mask tmp.mount' Works like a charm - Just 'systemctl disable tmp.mount'; after reboot mountpoint /tmp should say that "/tmp is not a mountpoint". You can then if so desired also remove the file again from /etc with 'sudo rm /etc/systemd/system/tmp.mount' ['disable' alone is insufficient; perhaps it works in conjunction with the 'rm'] More.. the default status is whatever was on the screen at the time one hits Reload page. That's the status that's defaulted.(In reply to Dave Hodgins from comment #20) > Strange. That doesn't happen for me, but I have extra privileges on bugzilla, From limited testing (I'd need to work with a test report), it "appears" (unconfirmed) that if I leave the Firefox tab open to the issue after I update it (comment 14), then someone else (you in this case) closes the report (15). I got the email; to see your comment, I had to reload the page to get your comment; but you had added 16 , both of which appeared on page reload. Then I confirmed and simply hit Save changes (16)... So this is likely a race condition of some sort. > I've just added editbugs authority for your id, so that may make a > difference. Not sure what more that would allow me to do; but I don't plan on doing anything new; though I will try to keep an eye on the Status before hitting Save. > Your other id for the email starting with pf@ instead of pfortin@ already > had the editbugs authority. The last bug email I have to pf@ is Dec 23/21... (Discuss list still comes to pf@) On Dec 31, 2021, I made a dumb/stupid mistake. I was cleaning out some old files, in this case some old .mozilla folders. It was too late when I realized one was a symlink; so I lost over 600 logins... so apparently, I re-registered -- there was some issue getting the pf@ registration back ( worked with Nicolas L and Thomas B.Feb 23/22; and decided to abandon pf@ since we couldn't get it to let me in. Maybe the above will eventually trigger a clue to figure out how unexpected Status changes occur... > That used to be a default, but due to spammers abusing bugzilla the settings > were made more restrictive. Thanks again for all you've done. Heard you've not been feeling well... Hope you're on the mend. Aging is not for sissies... LOL I'm about to turn a youthful 78 in Jan. Been on calming Linux since my Windows induced heart attack in 1998; seriously. If you want to re-enable tmp.mount after using "systemctl mask tmp.mount" then use "systemctl unmask tmp.mount" which deletes the symlink. |
Description of problem: I was processing a ~22M record file, generating output that exceeded available /tmp space: 22418128896 Oct 10 23:55 my.csv (22.4GB) $ df Filesystem Size Used Avail Use% Mounted on ... tmpfs 63G 63G 0 100% /tmp OK, delete the file and a few others: $ rm -f /tmp/my.csv $ ll ny.csv ls: cannot access 'my.csv': No such file or directory yet: $ df Filesystem Size Used Avail Use% Mounted on ... tmpfs 63G 63G 0 100% /tmp Version-Release number of selected component (if applicable): How reproducible: once; but likely requires reboot to clear this. Steps to Reproduce: unverified 1. write a file to /tmp Python: F = open( '/tmp/junk', 'w' ) while True: print( "anything; the longer, the faster", file = F ) 2. when the script aborts with: OSError: [Errno 28] No space left on device try to delete the junk file. 3.