Bug 32442 - haproxy: default config does not work!
Summary: haproxy: default config does not work!
Status: RESOLVED FIXED
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: Cauldron
Hardware: All Linux
Priority: Normal major
Target Milestone: ---
Assignee: Marc Krämer
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-10-25 13:16 CEST by Marc Krämer
Modified: 2023-11-25 16:39 CET (History)
2 users (show)

See Also:
Source RPM: haproxy
CVE:
Status comment:


Attachments

Description Marc Krämer 2023-10-25 13:16:42 CEST
if you install haproxy it fails to start with the default config.

The output of systemd does not help much:
Process: 242751 ExecStartPre=/usr/sbin/haproxy -f ${CONFIG} -c -q (code=exited, status=1/FAILURE)

running sudo -u haproxy /usr/sbin/haproxy -f /etc/haproxy.d -c
[NOTICE]   (242842) : haproxy version is 2.8.1-a90123a
[NOTICE]   (242842) : path to executable is /usr/sbin/haproxy
[ALERT]    (242842) : config : parsing [/etc/haproxy.d/0_default.cfg:222] : 'bind :::8443' in section 'frontend' :  '/etc/pki/tls/private/haproxy.pem' is present but cannot be read or parsed.


shows the real error.
ls -la /etc/pki/tls/private/haproxy.pem
-rw------- 1 root root 1704 Oct 25 12:20 /etc/pki/tls/private/haproxy.pem


a) it would be good, if systemd returns the error messages
b) config should be corrected, or the rights for haproxy.pem must be granted for user haproxy as stated in service file
Marc Krämer 2023-10-25 13:37:34 CEST

Depends on: (none) => 32319

Comment 1 Raphael Gertz 2023-10-25 19:01:17 CEST
Default pem rights is fixed since haproxy-2.8.3-7.mga9.

It don't makes sense imao to remove the quiet parameter from the configuration check.

Else it will spam /var/log/haproxy/access.log with:
"Configuration file is valid"

Tell me if it's resolved for u.

Best regards

CC: (none) => mageia

Comment 2 Marc Krämer 2023-10-25 19:07:19 CEST
no - it should only report the error in case the config is not valid.
at the moment the service does not start and there is no actual hint why.
Comment 3 Marc Krämer 2023-10-25 19:09:03 CEST
If you test ExecStartPre for the return code and run without -q, we would get what is wanted - maybe a helper script is needed to solve this.
Comment 4 Marja Van Waes 2023-10-25 21:58:14 CEST
Assigning to the haproxy maintainer

CC: (none) => marja11
Assignee: bugsquad => mageia

Comment 5 Raphael Gertz 2023-10-26 17:28:57 CEST
(In reply to Marc Krämer from comment #3)
> If you test ExecStartPre for the return code and run without -q, we would
> get what is wanted - maybe a helper script is needed to solve this.

I tried various combinations, no idea how to do it.

A helped script like:
#! /bin/sh

# Set notify socket
export NOTIFY_SOCKET=/run/systemd/notify

# Send check through systemd notify
/usr/bin/systemd-notify -- /usr/sbin/haproxy -f ${CONFIG} -c 

Only result with:
oct. 26 17:22:18 host.example.com systemd[1]: haproxy.service: Got notification message from PID 1617914, but reception only permitted for main PID which is currently not known

I understand the idea, but have no idea how to make it works.

The problem is that we use the systemd service to redirect the stdout and stderr to /var/log/haproxy/{access,error}.log

See this section of haproxy.service:
# Log to file
StandardOutput=append:/var/log/haproxy/access.log
StandardError=append:/var/log/haproxy/error.log
Comment 6 Raphael Gertz 2023-10-26 17:58:03 CEST
In my opinion it's not a bug anymore, default configuration will start without hasle.

When user change his configuration and has an error, it's his job to check his configuration.

The -q parameter seems pretty trivial...

This package will be used by very few peoples that should be able to remove a quiet parameter...
Comment 7 Raphael Gertz 2023-10-26 18:00:48 CEST
We may use a script like this one one to get the informations in error log file.

haproxy.service:
ExecStartPre=/usr/sbin/haproxy-check

haproxy-check:
#! /bin/sh

# Get pid
pid=$$;

# Prefix stderr
printf '[NOTICE] %*s(%d) : ' $((9-${#pid})) ' ' $pid >&2

# Send check through stderr
/usr/sbin/haproxy -f ${CONFIG} -c >&2

For me it's nonsense but if it works for you I will add it.
Comment 8 Marc Krämer 2023-10-26 18:25:38 CEST
Yepp. that helps! I guess just removing -q would be enough, since the logs go to error log.
If you see e.g. the startup message of apache, it really does not matter what is logged here.

You really expect to get startup errors also due to config problems, inside the error log, or in systemd.


I am new to haproxy... So I am making many mistakes.
Comment 9 Marc Krämer 2023-10-26 19:14:29 CEST
may I ask, if you use haproxy yourself?
Comment 10 Marc Krämer 2023-10-27 00:25:01 CEST
One more suggestion:
I found /var/log/haproxy/error.log not very helpful, as it does not contain any timestamps.

I suggest setting 
StandardError=journal
by default.

With this all error log is in the journal WITH timestamp, is autorotated and compressed.
Comment 11 Raphael Gertz 2023-10-27 03:48:27 CEST
(In reply to Marc Krämer from comment #8)
> Yepp. that helps! I guess just removing -q would be enough, since the logs
> go to error log.

No, it's not enough, as the haproxy config check is displayed on stdout and not stderr, thus finishing in /var/log/haproxy/access.log

(In reply to Marc Krämer from comment #8)
> I am new to haproxy... So I am making many mistakes.

My goal was to provide a usefull base template with most use cases, so anyone may start from there.

Company process will likely be: someone configure haproxy until they reach what is needed and nobody will touch it anymore without debugging it in preproduction first.

(In reply to Marc Krämer from comment #9)
> may I ask, if you use haproxy yourself?

I have it in production on my dedicated server.

It's my main reason to try to have a polished package.

With it it's possible to hide a apache+prosody+videobridge behind http(s) port(s) or load balance between server.

I tried to use stdout and stderr sent directly to journald, but it was unusable on my dedicated server and made it suffocate...

That's why I moved to:
StandardOutput=append:/var/log/haproxy/access.log
StandardError=append:/var/log/haproxy/error.log

It had the benefit to looks like apache configuration with it's two logs.

(In reply to Marc Krämer from comment #10)
> One more suggestion:
> I found /var/log/haproxy/error.log not very helpful, as it does not contain
> any timestamps.
> 
> I suggest setting 
> StandardError=journal
> by default.
> 
> With this all error log is in the journal WITH timestamp, is autorotated and
> compressed.

I would prefer to remain with current config with two separate logs.
Comment 12 Raphael Gertz 2023-10-27 03:49:35 CEST
While searching for log stuff, seems that my choices were shit and result in log loss :'(

https://github.com/haproxy/haproxy/issues/1861
Comment 13 Marc Krämer 2023-10-27 08:57:54 CEST
I put mine in production today. Got a setup of 3 apache servers loadbalanced with haproxy.

Thanks a lot for your work so far - I am glad the package is inside mageia!

I guess I will remove the access logs in general (they will grow too big). And I don't want to loose the statistics on logrotate (found they can be preserved by export import). I guess for error output stderr will do, but still the output does not help without timestamps. 

Did you make a HA setup with two haproxy's and some heartbeat mechanism?

I've added a script for a status output on command line:
hastatus.sh
echo "show stat no-maint" | socat stdio unix-connect:/run/haproxy/haproxy.sock|cut -d "," -f 1,2,5-10,18,36,50,34,36,37,38,56 | column -s, -t

Maybe you find this usefull and add it to the package.
Comment 14 Marc Krämer 2023-10-27 09:25:04 CEST
> Company process will likely be: someone configure haproxy until they reach what is needed and nobody will touch it anymore without debugging it in preproduction first.
Maybe, but sometime you have to do things directly. I don't think I will have a setup to test. For sure I will have a backup system with (heartbeat, peacemaker or keepalived). This is at least needed for updates, restart or the router/switch fail case.
heartbeat is not present in mga, peacemaker looks too complicated maybe keepalived will do what I need (or do you have recommendations?)
Comment 15 Marc Krämer 2023-10-31 13:00:33 CET
@Raphael: can we add a config option to disable output logging.

I don't want to modify the service files for every command line option....

a) I don't want logrotate to restart haproxy
b) log files getting big in just a few days (100M in 4 days) and I don't see any benefit for me - just in case of debugging...

I don't want to interfere your changes, just give you some more input.
Comment 16 Raphael Gertz 2023-10-31 18:55:44 CET
Ok, I seem to have narrowed the source of log problem.

On first start haproxy correctly write log and everything is fine.

Bug for unknown reason if stdout is append to some file, it will start to write at the file beggining.

$ echo -e 'line1\nline2\nline3' > /tmp/log.out
$ haproxy -f /etc/haproxy/haproxy.conf >> tmp/log.out

Log.out content was trashed.

I looked at trace, but don't seems to find any call that may have seeked to position 0.
Comment 17 Raphael Gertz 2023-10-31 19:24:40 CET
Ok, I developped a fix to prevent haproxy to drop the O_APPEND flag which was corrupting log.
Comment 18 Raphael Gertz 2023-10-31 19:29:32 CET
You may disable the log in global, I think.

Replace with no log or comment this line:
    log stdout format short daemon
Comment 19 Raphael Gertz 2023-11-01 05:46:39 CET
(In reply to Marc Krämer from comment #13)
> I don't want to loose the statistics on logrotate (found they can be
> preserved by export import).

How does the export/import work, maybe we can add it to postrotate script ?

Logrotate restart is monthly, it's instant in my configuration, I think it's safe to consider as default.

> I've added a script for a status output on command line:
> hastatus.sh
> echo "show stat no-maint" | socat stdio
> unix-connect:/run/haproxy/haproxy.sock|cut -d "," -f
> 1,2,5-10,18,36,50,34,36,37,38,56 | column -s, -t
> 
> Maybe you find this usefull and add it to the package.

May you test this one (more clean):
https://rapsys.eu/mageia/haproxy/haproxy-status.pl

And tell me if default index may be improved ?
(It requires perl-Sub-delete and perl-Monkey-Patch available in cauldron to workaround perl trouble with ordering hash)
Raphael Gertz 2023-11-01 06:14:05 CET

Depends on: 32319 => (none)

Raphael Gertz 2023-11-01 06:14:44 CET

Version: 9 => Cauldron

Raphael Gertz 2023-11-01 06:39:09 CET

Assignee: mageia => mageia
Status: NEW => NEEDINFO

Comment 20 Raphael Gertz 2023-11-01 17:19:09 CET
(In reply to Marc Krämer from comment #15)
> a) I don't want logrotate to restart haproxy

You may change the logrotate file, it will be kept upon update with a .rpmnew.

I changed it myself to retain 13 months of log instead of the 5 copied frop apache default config.

> b) log files getting big in just a few days (100M in 4 days) and I don't see
> any benefit for me - just in case of debugging...

You may disable logs in the haproxy config, logs will be empty.
Comment 21 Marc Krämer 2023-11-01 18:42:54 CET
(In reply to Raphael Gertz from comment #19)
> (In reply to Marc Krämer from comment #13)
> > I don't want to loose the statistics on logrotate (found they can be
> > preserved by export import).
> 
> How does the export/import work, maybe we can add it to postrotate script ?
https://github.com/haproxy/haproxy/blob/e826bc3dfa167b2b4596fcf22d9c9a4575c52040/doc/seamless_reload.txt

> Logrotate restart is monthly, it's instant in my configuration, I think it's
> safe to consider as default.
> 
> > I've added a script for a status output on command line:
> > hastatus.sh
> > echo "show stat no-maint" | socat stdio
> > unix-connect:/run/haproxy/haproxy.sock|cut -d "," -f
> > 1,2,5-10,18,36,50,34,36,37,38,56 | column -s, -t
> > 
> > Maybe you find this usefull and add it to the package.
> 
> May you test this one (more clean):
> https://rapsys.eu/mageia/haproxy/haproxy-status.pl
>
> And tell me if default index may be improved ?
> (It requires perl-Sub-delete and perl-Monkey-Patch available in cauldron to
> workaround perl trouble with ordering hash)

do those packages need a newer perl version? this is a production machine, so I don't want to add too much extra stuff just testing.


Btw. what heartbeat process to you use to monitor and switch over another haproxy instance?
Comment 22 Marc Krämer 2023-11-01 18:46:15 CET
(In reply to Raphael Gertz from comment #20)
> (In reply to Marc Krämer from comment #15)
> > a) I don't want logrotate to restart haproxy
> 
> You may change the logrotate file, it will be kept upon update with a
> .rpmnew.
> 
> I changed it myself to retain 13 months of log instead of the 5 copied frop
> apache default config.
> 
> > b) log files getting big in just a few days (100M in 4 days) and I don't see
> > any benefit for me - just in case of debugging...
> 
> You may disable logs in the haproxy config, logs will be empty.

Adding -q to the command line was easier.

Maybe you can add OPTIONS to /etc/sysconfig/haproxy like e.g. /etc/sysconfig/memcached which are added to the startup script. And appended to the comandline.
Comment 23 Raphael Gertz 2023-11-25 16:39:28 CET
Set this bug as resolved, better discuss config proposal for cauldron under bug #32440

Resolution: (none) => FIXED
Status: NEEDINFO => RESOLVED


Note You need to log in before you can comment on or make changes to this bug.