Bug 19675

Summary:	32 bit server kernel very slow when using > 4 GB RAM
Product:	Mageia	Reporter:	Nicolas Pomarède <npomarede>
Component:	RPM Packages	Assignee:	Mageia Bug Squad <bugsquad>
Status:	RESOLVED WONTFIX	QA Contact:
Severity:	major
Priority:	Normal	CC:	tmb
Version:	5
Target Milestone:	---
Hardware:	All
OS:	Linux
Whiteboard:
Source RPM:	kernel-server-4.4.26-1.mga5	CVE:
Status comment:

Description Nicolas Pomarède 2016-10-28 17:36:25 CEST

This bug is a followup to https://bugs.mageia.org/show_bug.cgi?id=18031 which is close.

I'm opening a new bug to keep track of the problem and possible solution.

Basically, it seems that's the more RAM you have beyond 4GB, the slower the kernel gets (huge drop in IO performance), because "VM split config" only allocs 1 GB to kernel in current build.

With 32 GB, the system can become barely usable, especially when disk intensive task are running (cp, rsync, ...)

----------------

Initial BR :

I'm experiencing exactly the same slowness as you described when running 32 bit kernel server. I have 2 PC desktop (rather powerful ones) :

- one DELL T3400 that worked fine for years with all the mageia versions, until I recently noticed some *huge* slow down in everything. Same as you even running urpmi to update takes between 10-20 minutes just to get the list of packages that need to be updated. Running strace on the rpm process shows it's reading /var/lib/Packages at an extremely slow speed.
Sometimes, I even had to kill rpm because it took too long, then rpm database was corrupt, so I had to run rebuilddb (which took much longer than expected too). This PC runs kernel 4.4.16

- So I thought this aging PC was maybe slowly dying. And I just installed a more recent HP Z240 (32 Go RAM, 2 disks, not using HW raid, but using mdadm) using mga5. I updated to kernel 4.4.26, and same problem : disk access are extremely slow. It took 5 minutes to copy /var/lib/rpm when this dir barely contains 100-200 MB.

Before using 4.4.16 on the Dell PC, I used kernel-server-4.1.15-2.mga5-1-1.mga5. That was a long time ago, but I think this kernel worked better IO wise.

Can you check kernel 4.1.15 if it's better ? Or did you find some kernel settings/parameters that fixed the IO performance in your case ?

----------------

Possible fix :

In the meantime, I found some similar bugs with possible solution ; it seems it's another PAE problem, the more RAM you have, the slower the PC will get in 32 bit mode.
I can see this in my case, the 8 GB DELL gets slow, but the 32 GB HP is even slower (despite having a better cpu).

some people told to flush vm cache in /proc/sys/vm/drop_caches :
https://bugs.launchpad.net/ubuntu/+source/linux-meta-lts-trusty/+bug/1333294
But that's not a real fix.

It seems a real solution is here :
http://flaterco.com/kb/PAE_slowdown.html

-> kernel needs to be build with CONFIG_VMSPLIT_2G=y instead of CONFIG_VMSPLIT_3G=y . This ways kernel will have 2 GB instead of 1 GB for its internal data and the impact in performance is supposed to be minimal.

I could verify that if I boot the HP with "mem=4G" then it runs as fast as it should, so the problem is really related to the amount of memory and the "VM split" build config.

Could we have an updated mga5 32 bit kernel with CONFIG_VMSPLIT_2G=y to test this ?

Comment 1 Thomas Backlund 2016-10-28 17:59:07 CEST

No,
 
we wont build 32bit server kernel with CONFIG_VMSPLIT_2G=y
It comes with it's own sets of issues with zero real gain.


With big hw / memory, there is only one supported option ... use 64 bit.

Status: NEW => RESOLVED
CC: (none) => tmb
Resolution: (none) => WONTFIX

Comment 2 Nicolas Pomarède 2016-10-28 18:08:09 CEST

I don't know what the issues are, but the gain is not zero, the gain is to get normal speed with your PC and don't suffer from IO slowness.

As a repeatable test, it actually takes 2 min to copy /var/lib/rpm which only consist of 70 MB, while it should take just a few seconds.

Comment 3 Thomas Backlund 2016-10-28 18:11:14 CEST

Just face the facts... any 64bit capable hw, especially with 4+ GB of ram should be running 64bit...

That's the only platform that get any real optimizations...

Comment 4 Nicolas Pomarède 2016-10-28 18:26:42 CEST

I'm not interested in optimization, else indeed I would run 64 bit version.

I have some legacy binary only 32 bit applications that I want to run under mageia.
Up to know they worked correctly when I had 4 GB, after going to 8 GB, performance is dropping hugely. Adding 4 GB of RAM doesn't seem such a huge change in HW spec, I didn't expect such problems.

After looking for a fix, I see that the user here :
http://flaterco.com/kb/PAE_slowdown.html
confirmed performances were restored with no visible drawbacks.

Just to know, what are the issues you mentionned if the 2/2 split is used ?

If mageia ships a 32 bit kernel that handles more than 4 GB, then I guess it's meant to be used and to work ? Else if it's a know fact that > 4GB causes problem, maybe the 32 bit kernel server should be limited to 4 GB, so users will really know they should use 64 bit version of the kernel for more RAM ?

Or just remove 32 bit server version and user will know they have 32 bit desktop version or need to go 64 bit for server version ?

Comment 5 Thomas Backlund 2016-10-28 21:02:09 CEST

It will work on atleast up to 8GB depending on how the memory gets set up...


And I can happily run a quad-core 32bit buildsystem here with 8GB RAM without slowdown.... so the "tripping point" is not at 4GB, it's more like around 8GB
(depending on whats running on the system and how its memory map gets set up)

But we wont punish people with real 32bit-only hw on the behalf of people having 64-bit hw but want it to under-perform using 32bit installs.

For example VMSPLIT_2G on 32bit server kernels would waste 1GB of userspace RAM for people having 32bit system with 4GB RAM.

and 64bit installs can run 32bit apps just fine.

Not to mention not all 3rdparty drivers will play nice with anything but the 3/1 split.

iirc also wine gets into trouble if not using 3/1 split.