why is jail.local created with e.g. shorewall as a default, when this is already defined in jail.conf??
This new file breaks existing setups!

the problem arises, since this file was not present in earlier releases.
I've seen, jail.conf should not have been modified; but it was...

Hi. Sorry to hear you have faced issues.
You nailed down the cause: user/admin settings should be done in .local files.
I tested it again while updating a server and I confirm the jail.local file doesn't get replaced by the update. Instead, you get a jail.local.rpmnew.
So, if you put your settings in jail.local, it will be preserved during the update.

Actually, any .conf might be replaced by an update. We don't enforce it though, and you might want to check the presence .rpmnew files in the folder filter.d

shorewall is the banaction default because it is the Mageia default firewall; for someone testing it for the first time, it allows a quick setup out of the box, after activating filters. 

During QA of the new package, a user who had "action = iptables-ipset-proto6-allports" haven't had any issue with this update.

this update really makes problems:
after some time the fail2ban server process uses 100% cpu

/etc/fail2ban/paths* other than mageia SHOULD be removed, they are not needed or wanted. 


jail.conf MUST contain
before = paths-mageia.conf

paths-mageia.conf SHOULD BE adapted from paths-fedora.conf

btw. why did we have such a critical update as regular update and not as backport or with the next release? Usally we stick to the versions we have when the distro is released!

Linked bug: https://github.com/fail2ban/fail2ban/issues/3370

URL: (none) => https://github.com/fail2ban/fail2ban/issues/3370

(In reply to Marc Krämer from comment #5)
> btw. why did we have such a critical update as regular update and not as
> backport or with the next release? Usally we stick to the versions we have
> when the distro is released!

Because we're human. We try very hard to avoid making mistakes, but well, being perfect is something we don't achieve.


(In reply to Marc Krämer from comment #6)
> Linked bug: https://github.com/fail2ban/fail2ban/issues/3370

Thanks for the link.

So this is the fix, right?
https://github.com/fail2ban/fail2ban/commit/ca2b94c5229bd474f612b57b67d796252a4aab7a

Squidf (Christian) isn't fail2ban's official maintainer and you, Marc, are able to test the fix.

So, could you please either fix this bug yourself or test the fix and tell squidf whether it works? Thanks :-)

CC: (none) => marja11
Source RPM: fail2ban-1.0.1-1.mga8.noarch => fail2ban-1.0.1-1.mga8, fail2ban-1.0.1-1.mga9
Whiteboard: (none) => MGA8TOO
Version: 8 => Cauldron

(In reply to Marja Van Waes from comment #7)
> (In reply to Marc Krämer from comment #5)
> > btw. why did we have such a critical update as regular update and not as
> > backport or with the next release? Usally we stick to the versions we have
> > when the distro is released!
> 
> Because we're human. We try very hard to avoid making mistakes, but well,
> being perfect is something we don't achieve.

I know that. I just wanted to raise this isse for the testers too. We have a backports policy, and this package is more critical than it seems to be. It protects servers from attacks.

 
> (In reply to Marc Krämer from comment #6)
> > Linked bug: https://github.com/fail2ban/fail2ban/issues/3370
> 
> Thanks for the link.
> 
> So this is the fix, right?
> https://github.com/fail2ban/fail2ban/commit/
> ca2b94c5229bd474f612b57b67d796252a4aab7a
> 
they are planning a new release soon. As it is known now, I would wait for the next release. It should have a bunch more fixes, as new majors always have.

Hi. Sorry for late reply, I have been too busy recently.

@Marc: where have you got this paths-mageia.conf from? It was not in the previous package either. It could be a good idea to make it as a default for the MGA package. Could you please share it?
I have added paths-overrides.local to do adjustment to the original configuration. It contains only Apache log paths correction for the time being. This file, as any .local, is preserved from one version to another. Without any bug reported previously, I didn't want to change anything more.

As a general rule, I think we should follow the guidance provided by Fail2ban which is to do our configurations in .local files, to benefit from the source updates and still preserving the user configurations.

@Marja: there is no official maintainer for this package. It is why I went ahead and updated it, because the current version was quite old and starting by a 0. 
To me, 1. means the version is now stable. Maybe I was wrong.
We have many packages which are updated because the branch is not maintained anymore. For me, this update was entering into the category of Software versions that are no longer supported upstream with updates. I thought it would be good to update such a package contributing to server security, as Marc mentioned.


The bug highlighted by Marc seems to be valid only when the dovecot jail is activated. @Marc, is it your case? Because, if not, there is something else.

What I can propose as next step:
1/Marc attaches to this report his paths-mageia.conf and I will include its content into paths-overrides.local (the idea is not to change source files as much as possible and to keep user configurations in .local)
2/I will monitor the arrival of the new Fail2ban update and I will push it into Testing. Marc will then be able to test it and to give is Okau.

What do you think?

(In reply to christian barranco from comment #9)
> Hi. Sorry for late reply, I have been too busy recently.
same for me, that is why this update hit me so much.
 
> @Marc: where have you got this paths-mageia.conf from? It was not in the
> previous package either. It could be a good idea to make it as a default for
> the MGA package. Could you please share it.
it is not shipped. But in general it is just the fedora.conf. But it looks odd to me, to ship configs not named with different names than the system used. So I'd remove the debian,.. conf's as they are never used on mga systems, rename the fedora-conf and make some adjustments, or create a new mga conf.
As we ship this, this is a normal procedure.

> I have added paths-overrides.local to do adjustment to the original
> configuration. It contains only Apache log paths correction for the time
> being. This file, as any .local, is preserved from one version to another.
> Without any bug reported previously, I didn't want to change anything more.
> 
> As a general rule, I think we should follow the guidance provided by
> Fail2ban which is to do our configurations in .local files, to benefit from
> the source updates and still preserving the user configurations.
Yes and no. I'd change the defaults for any mga system directly by patching the original configs. .local files should be empty, or just have an example and be up to the user. So updates don't interfer with running configs.


> @Marja: there is no official maintainer for this package. It is why I went
> ahead and updated it, because the current version was quite old and starting
> by a 0. 
> To me, 1. means the version is now stable. Maybe I was wrong.
I'm glad you did. But the new version was just a few days old. In general we don't update packages to newer versions in the stable release. We have them as backports. You may understand if you run several servers and do your regular updates, you don't expect to have a few run at 100%.

> We have many packages which are updated because the branch is not maintained
> anymore. For me, this update was entering into the category of Software
> versions that are no longer supported upstream with updates. I thought it
> would be good to update such a package contributing to server security, as
> Marc mentioned.
> 
> 
> The bug highlighted by Marc seems to be valid only when the dovecot jail is
> activated. @Marc, is it your case? Because, if not, there is something else.
It was dovecot. But it took me some time to understand what hit it. In earlier versions we didn't have .local files shipped, and to be honest this system is quite irregular, so some of my changes were made in the original config like it is done in any other system. I must have overread the intro with .local files.

> What I can propose as next step:
> 1/Marc attaches to this report his paths-mageia.conf and I will include its
> content into paths-overrides.local (the idea is not to change source files
> as much as possible and to keep user configurations in .local)
as stated - .local should be up to the user - we should patch the original files. This is the best for the user, as you never get .rpmnew files.

> 2/I will monitor the arrival of the new Fail2ban update and I will push it
> into Testing. Marc will then be able to test it and to give is Okau.
Sure, I can test these updates. I just run a few jails (apache*,dovecot,ssh) and as seen by the one of dovecot, you have to hit the regex to run into that case.

> What do you think?

I'm good with it.

(In reply to christian barranco from comment #9)

 @Marja: there is no official maintainer for this package. It is why I went
> ahead and updated it, because the current version was quite old and starting
> by a 0. 
> To me, 1. means the version is now stable. Maybe I was wrong.

This is actually a very bad assumption.

There is a lot  of projects that are at 0.something, but still declared stable/production use... 

Different projects treats numbers differently.

Some follows the "main version bump means/allows breakage", so going from 0->1 means breakage... (even if declared stable)

Others even break stuff in minor or patchlevel version bumps as they dont care/check for stuff like that...

Others like Linux kernel goes by "version is just a number, nothing more"

and so on...

So care needs always need to be taken when doing version bumps to think of possible fallouts

Oh, and just because there is a new version, it does not mean we need to push it... especially not in a stable mageia release...

remember "if it is not broken, dont fix it..."

Hi. I come back to what to do with .conf modification.
By preserving the .conf files modified by the user, there is a high risk to break fail2ban update because .conf files are essentials for fail2ban to work as intended by the update.
It is why it is written, for instance, in jail.conf:
# YOU SHOULD NOT MODIFY THIS FILE.
# It will probably be overwritten or improved in a distribution update.

It is why .local files should be used and to preserve the user configuration.
For filter adjustment, it is also recommended to create a new name and to activate it in a .local file.

That being said, if a user does use .local files to configure fail2ban, there is no issue for him/her to preserve the modified .conf, as there will be none.

It might be more academic than anything else maybe, but why our fail2ban package is made in such a way updated files will not have the priority over user modifications, as there is %config(noreplace) for the complete fail2ban folder?

(In reply to christian barranco from comment #13)
> Hi. I come back to what to do with .conf modification.
> By preserving the .conf files modified by the user, there is a high risk to
> break fail2ban update because .conf files are essentials for fail2ban to
> work as intended by the update.
> It is why it is written, for instance, in jail.conf:
> # YOU SHOULD NOT MODIFY THIS FILE.
> # It will probably be overwritten or improved in a distribution update.
Yeah I'm fine with that - it was my fault, but this is quite uncommon. If files are not intended to be modified they're usally placed in e.g. /usr/lib/ as it is for systemd. Files placed in etc are sth. similar than the overrides by systemd.
Maybe I'll place an FR to change this behaviour at fail2ban

> It is why .local files should be used and to preserve the user configuration.
> For filter adjustment, it is also recommended to create a new name and to
> activate it in a .local file.
Thats perfectly fine.

> That being said, if a user does use .local files to configure fail2ban,
> there is no issue for him/her to preserve the modified .conf, as there will
> be none.
> 
> It might be more academic than anything else maybe, but why our fail2ban
> package is made in such a way updated files will not have the priority over
> user modifications, as there is %config(noreplace) for the complete fail2ban
> folder?
just ignore it.

I've checked with fail2ban maintainer. He doesn't see any reason to place the conf files in usr/lib as he wants those configs to be changeable. So .local ist just convienent not so diff .rpmnew with the old config.

Thanks Marc for your investigation and your insights. 
I will not change anything. I was just trying to understand how it is intended to work. I would have separated more the configuration files from the rest of the “code” but it is what it is. I’ll open a new bug report for verification of the update when available and will link this one to it.

Ready for QA.

 @Marc: could you test it?  Thanks.


ADVISORY NOTICE PROPOSAL
========================

Update of major version of fail2ban with primary target to fix a dovecot-filter regression


Description
Update of major version of fail2ban with primary target to fix a dovecot-filter regression #3370.

Fixes
* backend systemd: code review and several fixes:
- wait only if it is necessary, e. g. in operational mode and if no more entries retrieved (end of journal);
- ensure we give enough time after possible rotation, vacuuming or adding/removing journal files, and move cursor back and forth to avoid entering dead space
* filter.d/named-refused.conf:
- support BIND named log categories, gh-3388
- allow info: as possible error prefix too ("query (cache) denied" may occur as info)
* filter.d/dovecot.conf:
- fixes regression introduced in gh-3210: resolve extremely long search by repeated apply of non-greedy RE-part with following branches (it may be extremely slow up to infinite search depending on message), gh-3370
- fixes regression and matches new format in aggressive mode too (amend to gh-3210)


References
https://bugs.mageia.org/show_bug.cgi?id=30952
https://github.com/fail2ban/fail2ban/releases/tag/1.0.2
https://github.com/fail2ban/fail2ban/blob/1.0.2/ChangeLog

SRPMS
8/core
fail2ban-1.0.2-1.mga8.src.rpm



PROVIDED PACKAGES:
=================
NOARCH
fail2ban-1.0.2-1.mga8.noarch.rpm

Source RPM: fail2ban-1.0.1-1.mga8, fail2ban-1.0.1-1.mga9 => fail2ban-1.0.1-1.mga8
Assignee: chb0 => qa-bugs
Version: Cauldron => 8
Whiteboard: MGA8TOO => (none)

Installed and tested without issues.

System: Mageia 8, x86_64, AMD CPU.

Tested on a workstation and server, protecting Apache and sshd.
fail2ban is configured with "action = iptables-ipset-proto6-allports" which is different from the default.



# uname -a
Linux jupiter 5.19.16-desktop-1.mga8 #1 SMP PREEMPT_DYNAMIC Sat Oct 15 18:19:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
#
#
# rpm -q fail2ban 
fail2ban-1.0.2-1.mga8
#
#
# fail2ban-client status
Status
|- Number of jail:      2
`- Jail list:   apache-auth, sshd
# fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     64
|  `- Journal matches:  _SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
   |- Currently banned: 1270
   |- Total banned:     1374
   `- Banned IP list:   <SNIP>
#
#
# iptables --numeric --list | grep f2b
REJECT     tcp  --  0.0.0.0/0            0.0.0.0/0            match-set f2b-apache-auth src reject-with icmp-port-unreachable
REJECT     tcp  --  0.0.0.0/0            0.0.0.0/0            match-set f2b-sshd src reject-with icmp-port-unreachable
#
#
# ipset list | grep f2b
Name: f2b-sshd
Name: f2b-apache-auth
#
#
# systemctl status fail2ban.service 
● fail2ban.service - fail2ban attack scanner
     Loaded: loaded (/usr/lib/systemd/system/fail2ban.service; disabled; vendor preset: disabled)
     Active: active (running) since Mon 2022-11-14 13:23:52 WET; 2h 7min ago
TriggeredBy: ● fail2ban.timer
    Process: 3183 ExecStart=/usr/bin/fail2ban-client -x start (code=exited, status=0/SUCCESS)
   Main PID: 3187 (fail2ban-server)
      Tasks: 7 (limit: 37626)
     Memory: 199.5M
        CPU: 9.531s
     CGroup: /system.slice/fail2ban.service
             └─3187 /usr/bin/python3 /usr/bin/fail2ban-server --async -b -s /var/run/fail2ban/fail2ban.sock -p /run/fail2ban/fail2ban.pid -x --loglevel INFO --logtarget SYSLOG --syslogsocket auto
<SNIP>

CC: (none) => mageia

sorry for the delay - it was busy. Just installed the package.

@PC_LX: thanks for the action - i was using the old iptables action; switched now.

You can obviously remove those files:
paths-arch.conf  paths-debian.conf  paths-fedora.conf  paths-freebsd.conf  paths-opensuse.conf  paths-osx.conf

mageia is neither of suse, macos, ... so there is no need to preserve configs which will never be used.
If the should stay, move them as documentation - but I'd say just remove them.


#uname -a
Linux borachio.qmsystems.de 5.18.15-server-1.mga8 #1 SMP PREEMPT_DYNAMIC Fri Jul 29 21:03:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
# fail2ban-client status
Status
|- Number of jail:	8
`- Jail list:	apache-badbots, apache-botsearch, apache-noscript, apache-overflows, dovecot, postfix-sasl, sendmail-auth, sshd
# fail2ban-client status dovecot
Status for the jail: dovecot
|- Filter
|  |- Currently failed:	4
|  |- Total failed:	4
|  `- Journal matches:	_SYSTEMD_UNIT=dovecot.service
`- Actions
   |- Currently banned:	2
   |- Total banned:	2
   `- Banned IP list:	XXX


Looks good so far.

giving ok for mga8

Whiteboard: (none) => MGA8-64-OK

Validating the update. Advisory committed to svn.

Keywords: (none) => advisory, validated_update
CC: (none) => davidwhodgins, sysadmin-bugs

An update for this issue has been pushed to the Mageia Updates repository.

https://advisories.mageia.org/MGAA-2022-0145.html

Status: NEW => RESOLVED
Resolution: (none) => FIXED