Created attachment 1260 [details]
mysql error file

Created attachment 1261 [details]
old mysql error file

Created attachment 1263 [details]
List of sql packages installed

du -s /home/dave/.local/share/akonadi/
149M    /home/dave/.local/share/akonadi/

would it be possible to get a SQL command "SHOW VARIABLES;" ?

i assume there's a socket somewhere in that directory, which can be used to connect to it.

echo "SHOW VARIABLES;" | mysql -S /path/to/socket > logfile

CC: (none) => alien

Created attachment 1265 [details]
SHOW VARIABLES

I have the same trouble.

Same problem here, nearly frozen system because mysqld uses 67.7% of memory most of the time (that was only after I finally managed to login in tty 2 and do top, before it must have been worse) and later still 63.7% a very big part of the time.

If I manage to get that logfile, I'll attach it in a later comment

CC: (none) => marja11

atm, i'd like to know how much queries it gets everytime it reboots, if someone can give me a mysqlbinlog file from that moment, it'd be nice.

all of you who have this issue? are they all desktop akonadi setups? are they new installs, or upgraded mysql setups?

if someone could perhaps give a zipped archive of their database directory, it'd be nice as well...

in any case, i'll try to reproduce, does anyone have an easy reproduce case?

I think the following will recreate the problem.

Create a new user.
Ensure /home is nearly full (less than 200M free).
Login to kde as the new user.

I booted to run level 3, renamed .local to .local.old, then started
kde, had the same problem, but also got a disk nearly full warning
for /home (39M free).

Restarted in run level 3, deleted .local, moved .local.old to a
different filesystem, and then started kde, and it was ok.

Looks like a problem when running out of disk space, and/or
crash recovery.

if anyone wish to find logs, they are at ~/.local/share/akonadi/db_data/mysql.err

i tried to recreate problem first this way:
 - install mga2a2
 - boot into KDE
 - make sure mysqld is running and akonadi is doing mysql stuff
 - update to complete cauldron
 - reboot
 - login to KDE
 - (no problem)
 - filled /home partition until around 150MB was free
 - made new user
 - logged out
 - logged into new user
  --> the logs note that mariadb can't create the ibdata1 (which is required), due to no disk space. no memory issues...

anything else i can try?

if anyone can zip me their broken akonadi folder for me to try?


is it possible that this could be related to having installed half of the new KDE and half of the old KDE? or something?

if you update completely and reboot, can you still reproduce?

did anyone try a mga1 upgrade?

my /home has 3.6 GB free space of 3.9 GB in total, nor did I login as a new user, so I don't think that is the problem.

Only mentioned on IRC: my system is a fresh Mga2alpha2 install, with updates. 

Mikala asked me to do "akonadictl stop", last night, so I did 
"akonadi stop | tee /home/marja/akonadictl-stop

The first line I see is: 
D-Bus session bus is not available!

I don't remember the rest of the lines, only that this block of lines was between "   "
minutes later, mysqld still uses 67.4 % MEM

after rebooting, I find out the akonadictl-stop file exists, but is empty.

In next comment, I'll try to paste the mysql.err of this morning. Sorry for not going to attach it, in my cauldron I'm now using iceweasel(?), and I'm not used to that DE at all

The content of /home/marja/.local/share/akonadi/db_data/mysql.err


111219  6:17:54 [Note] Plugin 'PBXT' is disabled.
111219  6:17:54 InnoDB: The InnoDB memory heap is disabled
111219  6:17:54 InnoDB: Mutexes and rw_locks use GCC atomic builtins
111219  6:17:54 InnoDB: Compressed tables use zlib 1.2.5
111219  6:17:54 InnoDB: Using Linux native AIO
111219  6:17:55 InnoDB: Initializing buffer pool, size = 80.0M
111219  6:17:55 InnoDB: Completed initialization of buffer pool
111219  6:17:55 InnoDB: highest supported file format is Barracuda.
111219  6:17:55  InnoDB: Waiting for the background threads to start
111219  6:17:56 Percona XtraDB (http://www.percona.com) 1.1.8-20.1 started; log sequence number 1718691
111219  6:17:56 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.18-MariaDB'  socket: '/home/marja/.local/share/akonadi/socket-localhost/mysql.socket'  port: 0  Mageia - MariaDB Community Edition (GPL)
111219  6:38:08 [Note] /usr/sbin/mysqld: Normal shutdown

Created attachment 1272 [details]
profile

My system fully updated. Yes, it's akonadi base. This reproduces with both old mysql base and new created by mariadb base. /home have 150 GiB free space.

Attached profile created with following steps:
1. create new user
2. login
3. see that mysqld ate my ram
4. logout

I've sent the contents of my akonadi directory via email.

This install where this is happening was originally a Mandriva 2009.1
system, upgraded via urpmi to 2010.2, then by mistake, converted to
Mageia with the Mageia 1, alpha 2, and has been kept as a cauldron
install since.

It still has many packages from Mandriva.

Created attachment 1273 [details]
Marja's akonadi folder

In the same folder as mysql.err, I have a akonadi_control.error:
D-Bus session bus went down - quitting 

and a akonadiserver.error
Control process died, committing suicide! 

Attaching my akonadi folder (although I won't be surprised if Bugzilla thinks the file is too big, in that case I'll mail the folder to you, AL13N)

it's likely too big for personal email, either email to maarten.vanraes@gmail.com or perhaps find some place where i can dl it?

assigning to maintainer

I don't have time now to "upgrade" to Mageia 1, sorry

Assignee: bugsquad => alien

(In reply to comment #17)
> I don't have time now to "upgrade" to Mageia 1, sorry

I don't understand that... nor do i know what you think i'm asking of you?

(In reply to comment #18)
> (In reply to comment #17)
> > I don't have time now to "upgrade" to Mageia 1, sorry
> 
> I don't understand that... nor do i know what you think i'm asking of you?

I didn't think you were asking that of me, but in comment 10 you asked whether anyone had tried that, no one confirmed having tried (or I overlook it). I'm a bug squad member, so I feel it my duty to help gather needed information about a bug wherever possible and within my power.

(In reply to comment #19)
> (In reply to comment #18)
> > (In reply to comment #17)
> > > I don't have time now to "upgrade" to Mageia 1, sorry
> > 
> > I don't understand that... nor do i know what you think i'm asking of you?
> 
> I didn't think you were asking that of me, but in comment 10 you asked whether
> anyone had tried that, no one confirmed having tried (or I overlook it). I'm a
> bug squad member, so I feel it my duty to help gather needed information about
> a bug wherever possible and within my power.

oic. looking back i see what i had asked.

it's a bit of a last possibility, if this is unreproducable when upgrading straight from mga1, it might have been a cauldron glitch.

in any case, i've installed a alpha2 via DVD, upgraded to cauldron and i cannot seem to reproduce it no matter what i try.

upstream noted that i might have used safemalloc (but i didn't), so that's not it either

(well, .2.mga2 had the safemalloc bug; .3.mga2 doesn't), that's why it's important if everyone has .3.mga2 ...

next step for me is to try the akonadi folders and see if i can reproduce that way...

The problem returned on my first start of cauldron today.  I noticed that
the number of mysqld threads had jumped to over 60.

I edited ~/.local/share/akonadi/mysql.conf and reduced the number of
connections from 256 to 4  (I have a single core system).

The res shown by htop is down from over 1500M to 809M. The mysqld
VIRT usage is still now 1039M (was over 2000M).

Tolerable, but still very excessive in my opinion.  I will not be able
to have mysqld running and VirtualBox, at the same time.

We need a way to limit the max memory mysqld is allowed to use, and
have a reasonable default for the akonadi mysqld server.

ok, can anyone who can reproduce this, set their .local/share/akonadi/mysql.conf to http://paste.pocoo.org/show/523239/ ? and make sure mysqld is not running, then log in and let it start up akonadi, then when it goes OOM(or at least very high in memory) stop it again?

there should be a mysql.full file, and i'd like to have that (may contain private info, or so i've heard), so perhaps you guys should remove your akonadi folders from here... you can email it to me.

thanks alot. i've been so far unable to reproduce it, so it's not easy.

(In reply to comment #21)
> The problem returned on my first start of cauldron today.  I noticed that
> the number of mysqld threads had jumped to over 60.
> 
> I edited ~/.local/share/akonadi/mysql.conf and reduced the number of
> connections from 256 to 4  (I have a single core system).
> 
> The res shown by htop is down from over 1500M to 809M. The mysqld
> VIRT usage is still now 1039M (was over 2000M).
> 
> Tolerable, but still very excessive in my opinion.  I will not be able
> to have mysqld running and VirtualBox, at the same time.
> 
> We need a way to limit the max memory mysqld is allowed to use, and
> have a reasonable default for the akonadi mysqld server.

hmm, now that you've mentioned this, i noticed that akonadi seems to do each query in a separate thread/connection, i've noticed at login time, sometimes 10 threads or more...

each thread is configured to have caches, buffers, etc... so in my system it takes for each thread at least 50MB.

question is, should akonadi do that much different connections? shouldn't it reuse connections?

btw: what kind of setup do you have? do you have 200 email accounts? or?

how many and what kind of email accounts are you having? imap/pop/offline_imap, etc...? do you have other types of connections, ical to google, and other stuff?

(In reply to comment #21)
[...]
> 
> We need a way to limit the max memory mysqld is allowed to use, and
> have a reasonable default for the akonadi mysqld server.
This mysql.conf configuration is the upstream's one & not some stuff specific to mageia.
You can eventually try the mysql-global-mobile.conf (available in /etc/akonadi/ ).
Anyway i'm not able to reproduce here (even if i'm using akonadi a lot :p )

but you have alot of memory.

how many mysqld threads do you see when logging in (keep a ksysguard open, logout, log back in again), and of what size? ie: ~60MB ? also, perhaps we can disable loading schema when logging in? for an automated process, this is not going to help alot...

we should ask upstream if it's wise to disable loading schema instead of doing it on our own without knowing the reason behind this.
my akonadi database is around 7GB,mysqld using around 153MB currently.
I did not check so far  but i would expect that mysql could eventually have several threads on starting (simply because it could eventually check the integrity of the database i would say).

(In reply to comment #21)
> The problem returned on my first start of cauldron today.  I noticed that
> the number of mysqld threads had jumped to over 60.
> 
> I edited ~/.local/share/akonadi/mysql.conf and reduced the number of
> connections from 256 to 4  (I have a single core system).
> 
> The res shown by htop is down from over 1500M to 809M. The mysqld
> VIRT usage is still now 1039M (was over 2000M).
> 
> Tolerable, but still very excessive in my opinion.  I will not be able
> to have mysqld running and VirtualBox, at the same time.
> 
> We need a way to limit the max memory mysqld is allowed to use, and
> have a reasonable default for the akonadi mysqld server.

can you also find a way to get the "SHOW PROCESSLIST" output when it's at it's highest amount of threads? (you can increase the connections back)

(In reply to comment #27)
> we should ask upstream if it's wise to disable loading schema instead of doing
> it on our own without knowing the reason behind this.
> my akonadi database is around 7GB,mysqld using around 153MB currently.
> I did not check so far  but i would expect that mysql could eventually have
> several threads on starting (simply because it could eventually check the
> integrity of the database i would say).

yes, of course, even though that might lessen the amount of queries done at startup (perhaps even per connection/thread), it is likely not the reason of the problem, so it was an idea, sometime to check later on...

(In reply to comment #28)
> (In reply to comment #21)
> > The problem returned on my first start of cauldron today.  I noticed that
> > the number of mysqld threads had jumped to over 60.
> > 
> > I edited ~/.local/share/akonadi/mysql.conf and reduced the number of
> > connections from 256 to 4  (I have a single core system).
> > 
> > The res shown by htop is down from over 1500M to 809M. The mysqld
> > VIRT usage is still now 1039M (was over 2000M).
> > 
> > Tolerable, but still very excessive in my opinion.  I will not be able
> > to have mysqld running and VirtualBox, at the same time.
> > 
> > We need a way to limit the max memory mysqld is allowed to use, and
> > have a reasonable default for the akonadi mysqld server.
> 
> can you also find a way to get the "SHOW PROCESSLIST" output when it's at it's
> highest amount of threads? (you can increase the connections back)

and also a "lsof | grep mysql" at sort of the same time, if possible

(In reply to comment #30)
> (In reply to comment #28)
> > (In reply to comment #21)
> > > The problem returned on my first start of cauldron today.  I noticed that
> > > the number of mysqld threads had jumped to over 60.
> > > 
> > > I edited ~/.local/share/akonadi/mysql.conf and reduced the number of
> > > connections from 256 to 4  (I have a single core system).
> > > 
> > > The res shown by htop is down from over 1500M to 809M. The mysqld
> > > VIRT usage is still now 1039M (was over 2000M).
> > > 
> > > Tolerable, but still very excessive in my opinion.  I will not be able
> > > to have mysqld running and VirtualBox, at the same time.
> > > 
> > > We need a way to limit the max memory mysqld is allowed to use, and
> > > have a reasonable default for the akonadi mysqld server.
> > 
> > can you also find a way to get the "SHOW PROCESSLIST" output when it's at it's
> > highest amount of threads? (you can increase the connections back)
> 
> and also a "lsof | grep mysql" at sort of the same time, if possible

sorry, a "lsof | grep local/share/akonadi/db_misc/mysql.socket

The lsof|grep on the socket|wc -l  is showing 4383, at the point where
the mem usage has reached a little over 1 GB.

I cannot get the SHOW PROCESSLIST; output.  If the number of connections
is limited, it just generates an error.  If the limit is 256, after waiting
for over an hour, it's still waiting for the akonadi requests that got in
ahead of the query.

htop is showing most of the mysqld threads are in a device wait state, as
are pretty much all processes that are trying to run.

It's pretty clear to me, that the main problem is akonadi generating far to
many sql requests.

Note: nepomuk and search are disabled on this account.  Thare are no kmail
accounts, no notes, not kde pim usage whatsoever.  The akonadi database is
empty!

I previously had akonadi disabled on this particular install.  It got
re-enabled by the kde updates.

If having akonadi enabled means kde cannot run in less than 8 GB, than I
will uninstall it, and put it in the skip list.

I really doubt that kde need 8 gb because :
- we should have far more people affected than now
- akonadi 1.6.2 is use since mysql era and did not shown this kind of problem so far
- you're not even using akonadi functionality when i'am 
So either it's related to a :
- miss upgraded/corrupted rpm
- to mariadb innodb implementation since if I'm not wrong is not the native one ( that does not save the fact that not everyone is able to reproduce it )
I'll try to install on virtualbox with 1 gb of memory to see if I can reproduce (from a  mageia alpha 2 install)

For the record nepomuk does not use mysql at all

Created attachment 1276 [details]
mysql.full + SHOW PROCESSLIST

The akonadi database is empty. Mem usage of mysqld ~1 GiB

i've tried quite some things, still unable to reproduce... i'm thinking maybe it's i586 related, i've been trying x86_64 all the time...

i'm gonna try with install from livecd i586 next

ok, upstream MariaDB provided me with a preliminary patch! i've submitted, can people test if it's resolved? mariadb-5.5.18-0.bzr3169.20111216.4.mga2 should be on the mirrors...

what was the issue?

there was a variable that had the wrong offset; which as a result, overwrote some other variable, but different in 64bit than i586. for 64bit it overwrote a harmless variable; but for i586 it had the following effect:

a certain variable was being allocated 128MB instead of 26 bytes, and this variable was being allocated for each connection, and also a few global variables.

akonadi has a lot of connections at startup (perhaps we can reduce this somehow), thus for 6 connections you already had 1GB allocated, and 14 connections had 2GB+ allocated.

the strange part is that mysql actually also had this issue, but due to the number of global variables it overwrites atm a harmless part. if an extra global variable is being made, mysql might overwrite another part of variables.

Now it uses 55 MiB for empty akonadi and 65 MiB for my main account. Thanks :-)

(In reply to comment #38)
> Now it uses 55 MiB for empty akonadi and 65 MiB for my main account. Thanks :-)

it's a big difference, isn't it? :-)

should we close the bug? or what now?

Thanks a lot, AL13N :)

Most of the time, I don't see mysqld in "top" anymore, and the short moments I do see it, it uses only 6.4% MEM

Closing as fixed

Status: NEW => RESOLVED
Resolution: (none) => FIXED