Bug 20143 - mysqld is not Active after start
Summary: mysqld is not Active after start
Status: RESOLVED FIXED
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: Cauldron
Hardware: x86_64 Linux
Priority: Normal major
Target Milestone: ---
Assignee: AL13N
QA Contact:
URL:
Whiteboard:
Keywords: 6sta2
: 20195 (view as bug list)
Depends on:
Blocks: 20139
  Show dependency treegraph
 
Reported: 2017-01-19 12:49 CET by Bit Twister
Modified: 2017-03-15 01:56 CET (History)
9 users (show)

See Also:
Source RPM: mariadb-10.1.21-2.mga6.src.rpm
CVE:
Status comment:


Attachments
Patch mysqld-wait-ready to wait 30 seconds maximum for socket and pidfile to avoid problems with mysqld_safe exiting before mysqld drop the socket and pidfile (1.62 KB, patch)
2017-01-31 16:44 CET, Raphael Gertz
Details | Diff
Patched version (2.12 KB, text/plain)
2017-02-03 15:59 CET, Raphael Gertz
Details
mysqld.service patch to replace mysqld-wait-ready with systemd notify mechanism (524 bytes, patch)
2017-02-06 12:47 CET, Arne Spiegelhauer
Details | Diff
mysqld_safe patch to test systemd notify mechanism (797 bytes, patch)
2017-02-06 13:02 CET, Arne Spiegelhauer
Details | Diff
Konsole I/O from patching (3.79 KB, text/plain)
2017-02-06 16:12 CET, Arne Spiegelhauer
Details
Patch systemd service to move to notify mechanism (685 bytes, patch)
2017-02-10 15:33 CET, Raphael Gertz
Details | Diff
mysqld_safe patch handling syslog case (1.71 KB, patch)
2017-02-12 15:48 CET, Arne Spiegelhauer
Details | Diff
Patch creates a working mysqld service (758 bytes, patch)
2017-02-13 01:57 CET, Dave Hodgins
Details | Diff
revised patch (594 bytes, patch)
2017-02-13 02:31 CET, Dave Hodgins
Details | Diff
Revised again as I forgot to run diff -u before uploading it before (557 bytes, patch)
2017-02-13 02:34 CET, Dave Hodgins
Details | Diff

Description Bit Twister 2017-01-19 12:49:18 CET
Description of problem:   6_s2:

 mysqld is not Active after start

systemctl status mysqld  snippet

   Active: deactivating (stop-sigterm) (Result: exit-code)
  Process: 21856 ExecStartPost=/usr/sbin/mysqld-wait-ready $MAINPID (code=exited, status=1/FAILURE)


Version-Release number of selected component (if applicable):


How reproducible: Always


Steps to Reproduce:
1. urpmi --download-all  --downloader wget --auto --auto-update
2. systemctl stop mysqld
3. systemctl start mysqld
4. systemctl status mysqld
Comment 1 Samuel Verschelde 2017-01-19 14:10:21 CET
Bit Twister, thanks for reporting those bugs, but could you avoid using your own formalism in bug report summaries?

There is a keyword to identifiy 6sta2 bugs, that is 6sta2.

I think the summary should be "mysqld fails to start"

Keywords: (none) => 6sta2
Summary: 6_s2: mysqld is not Active after start => mysqld is not Active after start

Comment 2 Marja Van Waes 2017-01-20 16:23:11 CET

Assigning to the registered maintainer of mariadb

CC: (none) => marja11
Assignee: bugsquad => alien

Comment 3 Bit Twister 2017-01-20 17:43:56 CET
(In reply to Samuel Verschelde from comment #1)
> Bit Twister, thanks for reporting those bugs, 

Your welcome.

> but could you avoid using your own formalism in bug report summaries?

Guessing you will be dinging the other users doing the same thing. :)

> I think the summary should be "mysqld fails to start"

Well, I did agonize over that for a minute. 

But I did not want someone to do a ps aux | grep mysqld and close the report because they saw it running. It did start but is running in the safe mode and apps like MythTv that use the database will have problems.

Status comment: (none) => 6_s1

Comment 4 Samuel Verschelde 2017-01-20 19:03:12 CET
(In reply to Bit Twister from comment #3)
> (In reply to Samuel Verschelde from comment #1)
> > but could you avoid using your own formalism in bug report summaries?
> 
> Guessing you will be dinging the other users doing the same thing. :)

Of course, actually most of the time we already just alter the summary directly without a comment :)

> > I think the summary should be "mysqld fails to start"
> 
> Well, I did agonize over that for a minute. 
> 
> But I did not want someone to do a ps aux | grep mysqld and close the report
> because they saw it running. It did start but is running in the safe mode
> and apps like MythTv that use the database will have problems.

My comment was because "is not active" can be understood as "does not even try to start", hence the "failed". But I was wrong since it does start in safe mode.
Comment 5 Bit Twister 2017-01-23 17:47:25 CET
Any chance of rolling out the previous working release?

I also think the problem should be a STA2 release blocker.
Comment 6 Doug Laidlaw 2017-01-23 19:52:57 CET
I have a "test" version of sta 2, and it was running so well that I decided to install it as my default system as well.  Then in both installs, I discovered that MySQL (Mariadb) was refusing to start.  On logging out, the non-existent service was being given over 4 minutes to shut down.  I disabled mariadb in MCC -> Services, and now shutdowns are normal. See

https://forums.mageia.org/en/viewtopic.php?f=15&t=11575

I can't imagine that I am not the only one, because if this is a bug, it is a release-blocker.  On a forum somewhere, I saw that the same issue (/etc/krb5.keytab doesn't exist) was causing problems with Samba.  Yet officially, the file is no longer needed.

BitTwister tells me I have a duplicate of this bug.

CC: (none) => laidlaws

David Walser 2017-01-24 02:38:41 CET

Blocks: (none) => 20139

Comment 7 David Walser 2017-01-29 18:01:35 CET
*** Bug 20195 has been marked as a duplicate of this bug. ***

CC: (none) => vincent78v

Comment 8 Doug Laidlaw 2017-01-29 18:49:11 CET
Bug 20195 sounds identical to mine.
Bit Twister 2017-01-30 02:53:35 CET

Status comment: 6_s1 => 6_s2
Source RPM: mariadb-10.1.21-1.mga6.src.rpm => mariadb-10.1.21-2.mga6.src.rpm

Comment 9 magnux77 2017-01-31 07:43:14 CET
I was developing a web server on M6 Sta1 on a PC; It worked fine but i can  not use it anymore. So installed M6 sta2 in VBox and i have the same problem.

CC: (none) => magnux77

Comment 10 Bit Twister 2017-01-31 07:58:03 CET
(In reply to magnux77 from comment #9)
> I was developing a web server on M6 Sta1 on a PC; It worked fine but i can 
> not use it anymore. So installed M6 sta2 in VBox and i have the same problem.

That is correct. The last two releases of mariadb rpm have a problem.

In my opinion, the mariadb rpm should be reverted back to the working mariadb rpm until the latest release works and can pass a go nogo test in cauldron testing.

This should be done before the next sta2 iso is generated.
Comment 11 Doug Laidlaw 2017-01-31 08:22:04 CET
Can a user wind back to the working release?  If that works, that is what needs to happen.  Does it raise other problems?
Comment 12 Bit Twister 2017-01-31 09:19:39 CET
(In reply to Bit Twister from comment #11)
> In my opinion, the mariadb rpm should be reverted back to the working mariadb
>  rpm until the latest release works and can pass a go nogo test in cauldron
> testing.

(In reply to Doug Laidlaw from comment #11)

> Can a user wind back to the working release?

No, This is cauldron and pre mariadb-10.1.21 rpm no longer exists.

> If that works, that is what needs to happen.
>  Does it raise other problems?

No idea for current mariadb-10.1.21 installs.

Good news would be apps needing mysqld active like php admin, mythtv, ...
can be tested and shutdown would not encounter the 300 second delay.
Comment 13 Arne Spiegelhauer 2017-01-31 14:31:54 CET
I recently found that the ftp.belnet.be mirror retains older package versions.
It is possible to downgrade to version 10.1.20 mariadb RPMs from this site.

This is, however, possibly only symptoms treatment as the actual problem seems to be in the interaction between mysqld and systemd.

/usr/lib/systemd/system/mysqld.service contains a.o. the following lines:

[Service]
Type=forking
User=mysql
Group=mysql

ExecStartPre=/usr/sbin/mysqld-prepare-db-dir
ExecStart=/usr/bin/mysqld_safe --nowatch
ExecStartPost=/usr/sbin/mysqld-wait-ready $MAINPID

When the failure occurs, $MAINPID is undefined/empty which causes the mysqld-wait-ready script to falsely conclude that mysqld terminated during startup.
Apparently with mariadb 10.1.21, systemd has not been able to guess the PID for mysqld before invoking mysqld-wait-ready.

Looking at mysqld.service or mariadb.service files of other distributions (e.g. Fedora) shows that they have replaced Type=forking with Type=notify (which requires mysqld to signal systemd, when it is ready). mysqld-wait-ready is thus no longer needed.


For a workaround, I have tried adding a sleep between

    eval "$cmd &"
and
    exit 0

in /usr/bin/mysqld_safe


It works, but I had to make it sleep for more than 10 seconds to get reliable mysqld start. With a smaller sleep period, first start might fail at boot, but succeed when systemd retried the start after the 5 minutes timeout. This was on a system with a relative slow hard disk and booting into Plasma with a lot of windows to restore.

CC: (none) => gm2.asp

Comment 14 Doug Laidlaw 2017-01-31 14:45:05 CET
"I recently found that the ftp.belnet.be mirror retains older package versions.
It is possible to downgrade to version 10.1.20 mariadb RPMs from this site."

But they would be for 5.1?

"This is, however, possibly only symptoms treatment as the actual problem seems to be in the interaction between mysqld and systemd."

In that case, could it affect mysql as well?
Comment 15 Doug Laidlaw 2017-01-31 14:47:03 CET
P.S.: I commented elsewhere that PCLinuxOS was O.K. But PCL has not adopted systemd.
Comment 16 Bit Twister 2017-01-31 15:26:49 CET
(In reply to Arne Spiegelhauer from comment #13)
 
> For a workaround, I have tried adding a sleep between
> 
>     eval "$cmd &"
> and
>     exit 0
> 
> in /usr/bin/mysqld_safe
> 
> 
> It works, but I had to make it sleep for more than 10 seconds to get
> reliable mysqld start. 

ok, adding sleep 10 above exit 0 at line 198 in /usr/bin/mysqld_safe gets me an active mysqld.service.
Comment 17 Raphael Gertz 2017-01-31 16:44:58 CET
Created attachment 8915 [details]
Patch mysqld-wait-ready to wait 30 seconds maximum for socket and pidfile to avoid problems with mysqld_safe exiting before mysqld drop the socket and pidfile

CC: (none) => mageia

Comment 18 Raphael Gertz 2017-01-31 16:46:58 CET
You can modify my patch for hardcoded pidfile if required or grow timeout to a better balanced timing until mysqld pid and socket file are waited for.
Comment 19 Raphael Gertz 2017-01-31 16:50:45 CET
For information, here on ssd and competition desktop, I only need 5 seconds for mysqld to start.

If someone has a slow server with huge database can tell the time this command takes with my patch it's welcome to balance timeout.

time systemctl start mysqld.service
Comment 20 Raphael Gertz 2017-01-31 16:52:43 CET
Seems realted to this one :
https://bugzilla.redhat.com/show_bug.cgi?id=1082018
Comment 21 Bit Twister 2017-01-31 17:42:49 CET
(In reply to Raphael Gertz from comment #17)
> Created attachment 8915 [details]
> Patch mysqld-wait-ready to wait 30 seconds maximum for socket and pidfile to
> avoid problems with mysqld_safe exiting before mysqld drop the socket and
> pidfile

The sad part about this problem, is it looks like we are putting a bandage on the problem instead of looking into why a long delay is required.
Bit Twister 2017-01-31 18:25:51 CET

Status comment: 6_s2 => (none)

Comment 22 Raphael Gertz 2017-01-31 18:28:19 CET
My patch isn't a bandage.

Currently systemd fork mysqld_safe --nowrap script, who fork a process of real mysqld and return 0.
Then systemd immediatly run the script mysqld-wait-ready without a pid as argument because parent mysqld_safe script returns before the child has landed the pid file and socket.

My patch makes sure we wait 30 seconds before returning something went wrong.

This delay is cut down to around one second after pid and socket are available.

Either we shoot the mysqld_safe script and make mysqld start directly in the ExecStart of the mysqld.service and we will need to change the Type of mysqld.service to something else than fork.

Or we make the wait script realy wait for the process to start.

It would be better to kick out mysqld_safe, but it parse some configuration, so I am not sure we can.
Comment 23 magnux77 2017-02-01 18:25:32 CET
(In reply to Raphael Gertz from comment #20)
> Seems realted to this one :
> https://bugzilla.redhat.com/show_bug.cgi?id=1082018

The problem looks the same but it's not really. mariadb is the the service lauched by systemd and the solution is finally to uncomment pid-file in my.cnf and it is yet uncommented.

I applied Arne's workaround and mysqld is alive.
Comment 24 Arne Spiegelhauer 2017-02-01 19:58:12 CET
No doubt Raphael's patch is by far the fastest way to a functioning mysqld service.

Kicking out mysqld_safe should also be a possibility as the RedHat family has done that. They are using the notify service type I mentioned in comment 13.
This type can also be used with mysqld_safe if eval "$cmd &" is replaced with exec $cmd. This, however, requires some "SELinux fixing" according to the following comment:

   # We'd prefer to exec $cmd here, but SELinux needs to be fixed first

Not having SELinux installed, I tried to do the replacement. After also removing I/O redirections that mysqld complained about from $cmd and making the following modifications to mysqld.service:
-changing type to notify
-removing post execution of mysqld-wait-ready

mysqld came up fine on reboot and could be stopped and started without problems.

It appears that the required ready notification is already sent by the currently packaged mysqld process.
Comment 25 Doug Laidlaw 2017-02-02 16:22:06 CET
I applied Raphael's patch to a clean install of Cauldron.  I stopped mysqld before rebooting, and the system rebooted promptly.  When it came up again, mysqld was running.
Comment 26 Doug Laidlaw 2017-02-02 17:34:12 CET
To be a bit more specific, no, I did not test first whether mysqld was running or not.  I assumed that it wasn't.  The reason for shutting it down anyway was so that systemd knew it was definitely off (see my Comment 6.)  Before, mysqld wasn't running, but systemd was still waiting for it to shut down!
Comment 27 Raphael Gertz 2017-02-02 17:48:54 CET
Currently mysqld is running. The hidden problem is that each time systemd wait for the timout of 300 seconds after mysqld-wait-ready failed before killing harshly mysqld and starting it again.

Keeping mysqld-wait-ready is not a bad idea if we start directly mysqld, as it handle the job of a watchdog that check that mysqld is in a really good shape.
Comment 28 Doug Laidlaw 2017-02-03 01:38:51 CET
(In reply to Raphael Gertz from comment #27)
> Currently mysqld is running. The hidden problem is that each time systemd
> wait for the timout of 300 seconds after mysqld-wait-ready failed before
> killing harshly mysqld and starting it again.
> 
> Keeping mysqld-wait-ready is not a bad idea if we start directly mysqld, as
> it handle the job of a watchdog that check that mysqld is in a really good
> shape.

Do you mean that mysqld is restarted every 300 secs?

For me (and probably for a lot of others) all that matters is that mysqld  starts on bootup.  I have two local apps that need it.  I can wait 5 minutes for it to start.
Barry Jackson 2017-02-03 01:46:25 CET

CC: (none) => zen25000

Comment 29 Barry Jackson 2017-02-03 01:54:56 CET
Well Raphael's patch is working fine for me.

I have been waiting 5 mins for shutdown/reboot for a week or so and had not realized that mysqld was not starting, only that it would not stop on shutdown.

With the patch applied it now starts on boot and shutdown is back to a few seconds.
Comment 30 Bit Twister 2017-02-03 15:54:02 CET
(In reply to Raphael Gertz from comment #17)
> Created attachment 8915 [details]
> Patch mysqld-wait-ready to wait 30 seconds maximum for socket and pidfile to
> avoid problems with mysqld_safe exiting before mysqld drop the socket and
> pidfile

How does one apply the patch?
I clicked details, did a Ctrl+a, saved result to mysqld-wait-ready.patch.
Removed diff line from patch.
cp /usr/sbin/mysqld-wait-ready /usr/sbin/mysqld-wait-ready.orig

And am at a loss as what to do next since everything tried since fails.
Comment 31 Raphael Gertz 2017-02-03 15:59:58 CET
Created attachment 8925 [details]
Patched version
Comment 32 Raphael Gertz 2017-02-03 16:02:55 CET
Just replace /usr/sbin/mysqld-wait-ready with the attachment #8925 [details].

chmod 0755 /usr/sbin/mysqld-wait-ready

chown root.root /usr/sbin/mysqld-wait-ready

Then restart or kill all mysqld process and and start mysqld service again through systemctl.

Do we really have selinux or something that may block from using my patch and require a more proper solution which is ?
Comment 33 Barry Jackson 2017-02-04 00:46:20 CET
(In reply to Bit Twister from comment #30)
> 
> How does one apply the patch?
> I clicked details, did a Ctrl+a, saved result to mysqld-wait-ready.patch.

OK so far.

> Removed diff line from patch.
> cp /usr/sbin/mysqld-wait-ready /usr/sbin/mysqld-wait-ready.orig
> 
> And am at a loss as what to do next since everything tried since fails.

Yes that was all wrong ;)

You just needed to:
su
patch -p0 /usr/sbin/mysqld-wait-ready < mysqld-wait-ready.patch
Comment 34 Barry Jackson 2017-02-04 00:51:03 CET
Oops, sorry, should be just:

patch -p0 < mysqld-wait-ready.patch
Comment 35 Bit Twister 2017-02-04 01:51:35 CET
(In reply to Barry Jackson from comment #34)
> Oops, sorry, should be just:

That's ok, all the examples I searched/looked at and man page did not show I had to enter target filename or have the diff line.

> patch -p0 < mysqld-wait-ready.patch

All the messages I saw made me abort that attempt when I had tried it.
Especially when prompted for "File to patch:" I would hit Ctrl+c

Resulting run looks like:

[root@wb ~]# patch -p0 < mysqld-wait-ready.patch
Ignoring potentially dangerous file name /usr/sbin/mysqld-wait-ready
can't find file to patch at input line 4
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff -urNp /usr/sbin/mysqld-wait-ready.orig /usr/sbin/mysqld-wait-ready
|--- /usr/sbin/mysqld-wait-ready.orig   2017-01-29 20:19:14.000000000 +0100
|+++ /usr/sbin/mysqld-wait-ready        2017-01-31 16:42:40.330396856 +0100
--------------------------
File to patch: /usr/sbin/mysqld-wait-ready
patching file /usr/sbin/mysqld-wait-ready


A diff shows the patch installed.

Thanks to you and Raphael for your time.
Comment 36 Arne Spiegelhauer 2017-02-04 11:45:55 CET
(In reply to Raphael Gertz from comment #32)
....
> 
> Do we really have selinux or something that may block from using my patch
> and require a more proper solution which is ?

Was this question triggered by comment 24?

That comment was just an attempt to list some alternative solutions:

1) minimal effort fix of current implementation (your patch).

2) getting rid of both mysqld_safe and mysqld-wait-ready, replacing it with systemd provided functionality. In my opinion, this is the more clean solution from a purely architectural point of view. It also seems to align with implementation by some other distributions (e.g. Fedora).

3) an in-between solution, keeping mysqld_safe for the configuration parsing, but replacing mysqld-wait-ready with the systemd notify mechanism already supported by mysqld.


It is solution 3) that has a potential problem with selinux (according to the comment in the mysqld_safe script). With the current mysqld_safe script, using eval to start mysqld, systemd sees the ready notification as coming from a wrong PID. Changing eval to exec solves that, but apparently causes a problem with selinux.


I guess a fourth solution approach could be to change eval to exec in mysqld_safe, change Type accordingly in mysqld.service, and fix the SELinux issue (whatever that may entail).
Comment 37 David Walser 2017-02-05 19:16:02 CET
We don't use or support SELinux, so that problem is irrelevant to us.

Arne, could you post a patch to mysqld_safe.sh and mysqld.service implementing your solution?
Comment 38 Arne Spiegelhauer 2017-02-06 12:47:11 CET
Created attachment 8936 [details]
mysqld.service patch to replace mysqld-wait-ready with systemd notify mechanism
Comment 39 Arne Spiegelhauer 2017-02-06 13:02:02 CET
Created attachment 8937 [details]
mysqld_safe patch to test systemd notify mechanism

Note that this patch is more of a quick hack to test use of the notify mechanism than a polished solution patch.
Items for closer investigation:
1) other adjustments than reopening standard out and standard err needed before exec'ing mysqld?
2) I ignored the syslog case. What needs to be done?
3) What should be done about the line 980, I commented out?
4) ...?
Comment 40 Doug Laidlaw 2017-02-06 14:04:03 CET
Is it necessary?  I patched with the mysqld.service.patch O.K., but somehow my aplying mysqld_safe.patch ended up looking like BitTwister's output in Comment 35.

I will need to restore the original mysqld_safe and try again.
Comment 41 Barry Jackson 2017-02-06 14:23:16 CET
The path in the patch is incorrect, so just enter the correct one when asked (without the ".orig")
Comment 42 Doug Laidlaw 2017-02-06 14:28:21 CET
Thanks Barry.  Trying with just the mysqld.service patch, I still can't get mysqld to start.  When I try to add mysqld_safe.patch, I get "Permission Denied."  I don't get "File to patch."

Since the patch was intended for developers, I will just sit tight and await the outcome.
Comment 43 Arne Spiegelhauer 2017-02-06 16:12:54 CET
Created attachment 8938 [details]
Konsole I/O from patching

Sorry about the incorrect path in patches.
This attachment contains Konsole I/O from downloading and applying patches on another Cauldron installation, following Barry's instruction from comment 41
Comment 44 Arne Spiegelhauer 2017-02-06 16:30:47 CET
(In reply to Doug Laidlaw from comment #40)
> Is it necessary?  I patched with the mysqld.service.patch O.K., but somehow
> my aplying mysqld_safe.patch ended up looking like BitTwister's output in
> Comment 35.
> 
> I will need to restore the original mysqld_safe and try again.

Yes, the mysqld_safe patch is necessary to have the ready notification come from the PID of the process started by systemd with the ExecStart=... line.

As can be seen from my uploaded Konsole dump, the patching should work following Barry's instruction. This assuming that you don't have an old mysqld_safe.orig file.

Note that if the system is not rebooted after patching mysqld_safe, the command:

systemctl daemon-reload

must be executed to make systemd reread the changed file.
Comment 45 Raphael Gertz 2017-02-10 15:33:35 CET
Created attachment 8944 [details]
Patch systemd service to move to notify mechanism

It seems that mariadb now support notify mechanism :
https://github.com/MariaDB/server/pull/83

Sadly it don't support watchdog, but with this patch it start/restart/stop correctly.

Tried to kill and kill -9 main process and it comes back correctly.

The mysqld will read the /etc/my.cnf and /etc/my.cnf.d/*.cnf by itself, so mysqld_safe service is basicaly useless now.

Can we switch to this patch ?
Comment 46 Arne Spiegelhauer 2017-02-11 14:43:04 CET
(In reply to Raphael Gertz from comment #45)
> Created attachment 8944 [details]
> Patch systemd service to move to notify mechanism
> 
> It seems that mariadb now support notify mechanism :
> https://github.com/MariaDB/server/pull/83
> 
> Sadly it don't support watchdog, but with this patch it start/restart/stop
> correctly.
> 
> Tried to kill and kill -9 main process and it comes back correctly.
> 
> The mysqld will read the /etc/my.cnf and /etc/my.cnf.d/*.cnf by itself, so
> mysqld_safe service is basicaly useless now.

As far as I gather, getting rid of mysqld_safe also involves replacing its configuration settings with systemd equivalents (as far as possible).

See:
https://mariadb.com/kb/en/mariadb/systemd/
https://github.com/MariaDB/server/blob/10.1/scripts/mariadb-service-convert

I expect it would involve some documentation update as well
Comment 47 David Walser 2017-02-11 22:17:11 CET
Arne, I've committed your changes in Cauldron and pushed it to the build system.

Let's see how this goes (feedback good or bad is welcome).

As for rapsys's proposed changes, as we're trying to stabilize Mageia 6 and need to fix this for Mageia 5 as well, I don't want to make more dramatic changes than are needed at this time.  Once Cauldron re-opens for Mageia 7 we can try it.
Comment 48 Raphael Gertz 2017-02-11 23:51:38 CET
Arne, David :
I understand that we need to fix it for MGA6 and 5.

But, this fix is just a patch on a wood leg, my second fix should be the way to go.

I read all the things about mysqld_safe and I don't see any real case where it's required to migrate these features automaticaly.

For me the people that have these kind of needs :
  --no-defaults              Don't read the system defaults file
  --core-file-size=LIMIT     Limit core files to the specified size
  --defaults-file=FILE       Use the specified defaults file
  --defaults-extra-file=FILE Also use defaults from the specified file
  --ledir=DIRECTORY          Look for mysqld in the specified directory
  --open-files-limit=LIMIT   Limit the number of open files
  --crash-script=FILE        Script to call when mysqld crashes
  --timezone=TZ              Set the system timezone
  --malloc-lib=LIB           Preload shared library LIB if available
  --mysqld=FILE              Use the specified file as mysqld
  --mysqld-version=VERSION   Use "mysqld-VERSION" as mysqld
  --dry-run                  Simulate the start to detect errors but don't start
  --nice=NICE                Set the scheduling priority of mysqld
  --no-auto-restart          Exit after starting mysqld
  --nowatch                  Exit after starting mysqld
  --plugin-dir=DIR           Plugins are under DIR or DIR/VERSION, if
                             VERSION is given
  --skip-kill-mysqld         Don't try to kill stray mysqld processes
  --syslog                   Log messages to syslog with 'logger'
  --skip-syslog              Log messages to error log (default)
  --syslog-tag=TAG           Pass -t "mysqld-TAG" to 'logger'
  --flush-caches             Flush and purge buffers/caches before
                             starting the server
  --numa-interleave          Run mysqld with its memory interleaved
                             on all NUMA nodes

Have the skills to fix their mysqld.service with required changes.

Could we use the fix of arne on MGA5 and mine on MGA6 ?

My second fix remove the requirement on watchdog scripts, avoid requirement to kill by hand mysqld service when something goes wrong like upgrade Cauldron.

It would just require to add a note to the errata pointing on migrating mysqld.service.
Comment 49 Raphael Gertz 2017-02-12 00:00:29 CET
This is for oracle mysqld, but this should be appliable to mariadb as well..

See :
https://dev.mysql.com/doc/refman/5.7/en/mysqld-safe.html
https://dev.mysql.com/doc/refman/5.7/en/using-systemd.html

« Note
As of MySQL 5.7.6, for MySQL installation using an RPM distribution, server startup and shutdown is managed by systemd on several Linux platforms. On these platforms, mysqld_safe is no longer installed because it is unnecessary. For more information, see Section 2.5.10, âManaging MySQL Server with systemdâ. »

« Note
Controlling mysqld logging from mysqld_safe is deprecated as of MySQL 5.7.5. Use the server's native syslog support instead. For more information, see Section 6.4.2, âThe Error Logâ. »
Comment 50 Doug Laidlaw 2017-02-12 04:30:16 CET
New build in Cauldron works O.K. for me.
Comment 51 Bit Twister 2017-02-12 05:04:39 CET
Works on two hosts and 3 virtualbox guests.

I'll close this bug resolved.

Status: NEW => RESOLVED
Resolution: (none) => FIXED

Comment 52 Arne Spiegelhauer 2017-02-12 15:48:42 CET
Created attachment 8952 [details]
mysqld_safe patch handling syslog case

(In reply to David Walser from comment #47)
> Arne, I've committed your changes in Cauldron and pushed it to the build
> system.
> 
I am afraid that was a bit premature. I hadn't got around to looking at the syslog case yet. Not only does syslog not work, the /var/lib/mysql/aria_log_control file could be corrupted if mysqld writes to stdout.
I have uploaded a new version that should fix the syslog case.

> Let's see how this goes (feedback good or bad is welcome).
> 
> As for rapsys's proposed changes, as we're trying to stabilize Mageia 6 and
> need to fix this for Mageia 5 as well, I don't want to make more dramatic
> changes than are needed at this time.  Once Cauldron re-opens for Mageia 7
> we can try it.

It can't get any simpler than Raphael's latest proposal though, but I have no idea whether it would be OK to leave it up to the, presumably few, users who need unusual configurations to do the systemd configuration manually.
Arne Spiegelhauer 2017-02-12 15:55:38 CET

Status: RESOLVED => REOPENED
Resolution: FIXED => (none)

Comment 53 Dave Hodgins 2017-02-12 18:26:23 CET
After the 5 minute delay during the install of the update (there has to be a better way than holding urpmi/rpmdrake for 5 minutes),

# rpm -qa|grep mariadb-1
mariadb-10.0.29-1.2.mga5

# systemctl -a status mysqld.service 
â mysqld.service - MySQL database server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled)
   Active: activating (start) since Sun 2017-02-12 12:09:41 EST; 30s ago
 Main PID: 17281 (mysqld)
   CGroup: /system.slice/mysqld.service
           ââ17281 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin...

Feb 12 12:09:41 x3.hodgins.homeip.net systemd[1]: mysqld.service holdoff time over, scheduling restart.
Feb 12 12:09:41 x3.hodgins.homeip.net systemd[1]: Stopping MySQL database server...
Feb 12 12:09:41 x3.hodgins.homeip.net systemd[1]: Starting MySQL database server...
Feb 12 12:09:42 x3.hodgins.homeip.net mysqld_safe[17281]: 170212 12:09:42 mysqld_safe Logging to '/var/log/mysqld/mysqld.log'.
Feb 12 12:09:42 x3.hodgins.homeip.net mysqld_safe[17281]: 170212 12:09:42 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql

The /var/log/mysqld/mysqld.log looks ok, with mysqld ready for connections and
the /var/lib/mysql/mysql.sock setup.

It's also trying to restart every 5 minutes ...
journalctl -b|grep mysqld.service|tail -n 6
Feb 12 12:14:45 x3.hodgins.homeip.net systemd[1]: mysqld.service failed.
Feb 12 12:14:45 x3.hodgins.homeip.net systemd[1]: mysqld.service holdoff time over, scheduling restart.
Feb 12 12:19:45 x3.hodgins.homeip.net systemd[1]: mysqld.service start operation timed out. Terminating.
Feb 12 12:19:47 x3.hodgins.homeip.net systemd[1]: Unit mysqld.service entered failed state.
Feb 12 12:19:47 x3.hodgins.homeip.net systemd[1]: mysqld.service failed.
Feb 12 12:19:48 x3.hodgins.homeip.net systemd[1]: mysqld.service holdoff time over, scheduling restart.

CC: (none) => davidwhodgins

Comment 54 Bit Twister 2017-02-12 20:09:28 CET
Can not speak for the urpm problem. Can not remember the release (mga 3/.4) but a mysql updated wiped out my database. Modified my install_updates script to stop mysqld prior to any mysql rpm install.


None of my installs are restarting.

I wanted clean logs so I have delayed mysqld start until network is up and decreased the delay.

$ cat /etc/systemd/system/mysqld.service.d/xx__local.conf
#
#  Created by /local/bin/mysqld_service_changes Sun 22 Jan 09:50 2017 
#

[Unit]
After=network-online.target
 
[Service]
# Give a reasonable amount of time for the server to start up/shut down
TimeoutSec=150


Other changes are I have given mysql root a password and disabled auth_gssapi plugin.

$ dif /var/local/vorig/etc/my.cnf.d/auth_gssapi.cnf_vinstall /etc/my.cnf.d/auth_gssapi.cnf
2c2,3
< plugin-load-add=auth_gssapi.so
---
> # changed by /local/bin/auth_gssapi_changes Wed 01 Feb 03:53 2017
> # plugin-load-add=auth_gssapi.so
Comment 55 Doug Laidlaw 2017-02-12 23:42:39 CET
I use mysql databases only occasionally, and I let mysqld run all the time.  What is there works for me, and I haven't got the knowledge to fiddle with systemd.  I will just continue "reading the mail."
Comment 56 Dave Hodgins 2017-02-13 01:57:56 CET
Created attachment 8953 [details]
Patch creates a working mysqld service

Using https://gist.githubusercontent.com/thomasfr/e4e4bb64352ee574334a/raw/e8388f6c99c21fd96a343c597eec22aaec86f560/mysqld.service as a base,
I modified my mysqld.service file to create one that works, including restarting
if mysqld is killed.

The only thing I'm not sure about is the option --plugin-dir=/usr/lib64/mysql/plugin, since that's architecture dependent.
Comment 57 Dave Hodgins 2017-02-13 02:31:07 CET
Created attachment 8954 [details]
revised patch

Revised patch so settings in /etc/my.cnf are used.

Attachment 8953 is obsolete: 0 => 1

Comment 58 Dave Hodgins 2017-02-13 02:34:25 CET
Created attachment 8955 [details]
Revised again as I forgot to run diff -u before uploading it before

Attachment 8954 is obsolete: 0 => 1

Comment 59 David Walser 2017-02-18 15:47:58 CET
OK, new Cauldron build building now with rapsys's mysqld.service file which no longer uses mysqld_safe.  Mageia 5 build building now with mysqld.service changes reverted and using rapsys's patch to mysqld-wait-ready.

CC: (none) => luigiwalser

Comment 60 David Walser 2017-03-15 01:56:49 CET
Previous updates for this have been pushed.

Status: REOPENED => RESOLVED
Resolution: (none) => FIXED


Note You need to log in before you can comment on or make changes to this bug.