Thank you for reporting this, Marc; and for the upstream bug.

Can you please say:
- what is the version of duplicity
(FWIW the current Cauldron version is duplicity-0.8.13-1.mga8.src.rpm)
- what the 3 German lines mean (even though the essential bit is in English).

I think we should await a response to your upstream bug report before assigning this; it will have to go to pkg-bugs because Duplicity has no consistent maintainer.

CC: (none) => lewyssmith
URL: (none) => https://launchpad.net/bugs/1895353

I have a sneaking feeling that the maximum allowable file descriptors is a system-wide value which can be fiddled. I will try looking into that, in case it is relevant.

OK, there is a standard limit if 1024 file descriptors per process - which can be increased by various means.
 $ ulimit -n
 1024
I doubt that this is the root of the bug, it would give an error message like "Too many files opened".

Upstream launchpad triager wonders:
Kenneth Loafman (kenneth-loafman) wrote on 2020-09-12: 	#1

What duplicity version? What OS version? What Python version?

Infos needed there to help this.

sorry for the late reply, I was on holiday and this error occured just a few hours before I had to leave :(

This is duplicity-0.8.12.1612-1.mga7.x86_64, python-2.7.17-1.1.mga7.x86_64 (python3-3.7.6-1.mga7.x86_64)

I tried to raise the limit via ulimit, but this was not successfull.

I've searched for this error, and if I don't misunderstand it in total, select has a hard limit set to "FD_SETSIZE", which is equal to 1024, so this is a hard limit. Most articles suggest using "poll" instead of "select".

I'm not sure why duplicity spawns this many processes instead of closing them and contiuing with the next, this might be an error in general.

Marc
Re your upstream bug:
(In reply to Aurelien Oudelet from comment #4)
> Kenneth Loafman (kenneth-loafman) wrote on 2020-09-12: 	#1
> 
> What duplicity version? What OS version? What Python version?
Can you please reply there (if you have not already done so).

Some answers upstream bug report.
Also, suggested to do:

--Quote---
Try this:
# ulimit -n 65536
# duplicity ...your options...

If that works, set it globally via one of the solutions on the net. Google "linux too many file handles". I am not familiar with your distro, so find the solution closest.

Also, so I can rule out one problem I've encountered from downstream distros, please check the duplicity directory for the file 'gpginterface.py', then check line 696 for threaded_waitpid(). I had to modify the original to harvest the fileid's of completed processes. Some distro's will make the mods back to the original in error, something I've never understood.
---End quote---

Also it is suggested to upgrade to more recent version of Duplicity.

But for now, we already have duplicity-0.8.12.1612-1.mga7 in M7 core/updates.
We already have latest 0.8.15, released July 27, 2020 in M8 Cauldron.

Here is changelog between these two versions:

--Begin-Changelog---
New in v0.8.15 (2020/07/27)
---------------------------
* Fix bug #1887689 with patch from Matthew Barry
  - Cleanup with Paramiko backend does not remove files due to missing
    filename byte decoding
* Fix bug #1211481 with merge from Raffaele Di Campli
  - Ignores the uid/gid from the archive and keeps the current user's one.
  - Recommended for restoring data to mounted filesystem which do not
    support Unix ownership or when root privileges are not available.
* Fix issue #10 - ppa:duplicity-release-git fails to install on Focal Fossa
  - Set correct version requirements in debian/control.
* Merged in joshAppdev:pydriveshared - Backend for Shared Drives on Google
  - pydrive://developer.gserviceaccount.com/target-folder/?driveID=<SHARED DRIVE ID>
* Merged in martin-sucha:pydrive-notfound - Fix missing FileNotUploadedError in pydrive
  - Since dadbe2d2, FileNotUploadedError is not imported anymore, resulting in an
    exception in case some of the files failed to upload. Adding the import back.
* Merged in hupfdule:s3-boto3-region-and-endpoint - Allow setting s3 region and endpoint
  - This commit introduces the new commandline options --s3-region-name, --s3-endpoint-url
    to specify these parameters. This allows using s3 compatible providers like Scaleway
    or OVH.
  - It is probably useful for Amazon accounts, too, to have more fine grained influence
    on the region to use.

New in v0.8.14 (2020/07/04)
---------------------------
* Fixes for rclonebackend from Francesco Magno (original author)
  - copy command has been replaced with copyto, that is a specialized
    version for single file operation. Performance-wise, we don't have
    to include a single file in the local side directory, and we don't
    have to list all the files in the remote to check what to syncronize.
    Additionally, we don't have to mess up with renaming because the
    copy command didn't support changing filename during transfer
    (because was oriented to transfer whole directories).
  - delete command has been replaced with deletefile. Same here, we
    have a specialized command for single file operation. Much more efficient.
  - ls command has been replaced with lsf, that is a specialized version
    that returns only filenames. Since duplicity needs only those, less
    bytes to transfer, and less parsing to do.
  - lastly, I have reintroduced a custom subprocess function because the one
    inherithed from base class is checked, and throws an exception in case of
    non zero return code. The ls command family returns a non zero value if
    the directory does not exist in the remote, so starting a new backup
    in a non existent directory is impossible at the moment because ls fails
    repeatedly until duplicity gives up. This is a bug in the current implementation.
    There is the same problem (but less severe) in _get method, using the default
    self.subprocess_popen a non zero return code will throw an exception before we
    can cleanup the partially downloaded file, if any.
* Fixed bug #1875937 - validate_encryption_settings() fails w/S3 glacier
  - Skip validation with a warning if S3 glacier or deep storage specified
* Patched in a megav2backend.py to update to MEGAcmd tools.
  - Author: Jose L. Domingo Lopez <github@24x7linux.com>
  - Man pages, docs, etc. were included.
* More fixes for bug #1877885 - Catch quota overflow on Mega upload.
* Merged in jmakovicka:master - Support PyDrive2 library in the pydrive backend
  - Unlike PyDrive, the PyDrive2 fork is actively maintained.
* Merged in mikix:mikix/rename-fix - Fix --rename encoding
  - The --rename argument wasn't working because its arguments were coming in and
    being saved as unicode. But then compared against bytestring index path parts.
  - This MR fixes that by saving the rename pieces as bytestrings up front.
* Fixes for issue #7, par2backend produces badly encoded filenames.
* Set deprecation version to 0.9.0 for short filenames.

New in v0.8.13 (2020/05/05)
---------------------------
* Fixed bug #1868414 - timeout parameter not passed to
  BlobService for Azure backend
* Merged in lp:~kenneth-loafman/duplicity/duplicity-pylint
  - Enable additional pylint warnings. Make 1st pass at correction.
      unused-argument,
      unused-wildcard-import,
      redefined-builtin,
      bad-indentation,
      mixed-indentation,
     unreachable
  - Renamed globals to config to fix conflict with __builtin__.globals()
  - Resolved conflict between duplicity.config and testing.manual.config
  - Normalized emacs mode line to have encoding:utf8 on all *.py files
* Fixed bug #1869921 - B2 backup resume fails for TypeError
* Fixed bug #1872332 - NameError in ssh_paramiko_backend.py
* Fixed bug #1875529 - Support hiding instead of deletin on B2
* Fixed bug #1876778 - byte/str issues in megabackend.py
* Fixed bug #1876446 - WebDAV backend creates only tiny or 0 Byte files

--End Changelog---


Leaving this open for now in Bugsquad until further testing from Marc.

Source RPM: duplicity => duplicity-0.8.12.1612-1.mga7.src.rpm
Keywords: NEEDINFO => NEEDTEST

I can now reproduce the problem. Ulimit does NOT solve the issue!
Can we have a newer version in updates_testing? So I can check if that solves the issue?

Current release is 0.8.16, it does not look like this bug is fixed there, but it still would be good to make the tests against their latest release.

A few hours ago: 0.8.17

CC: (none) => fri

pushed the update myself - still the same. Waiting for reply upstream. I'll write here if we have news.

Good:
So we now have 0.8.17 in testing, and you have tested it in real use.

The problem you see with it is the same as the old version.

So to me it at least seem we can OK and ship 0.8.17 as it is a good update?

0.8.17 works as expected.

It looks like my problem is based on the used backend "pexpect" which uses "select" and fails.
Using "paramiko" causes a filelimit error, but this can be handled by setting 
ulimit -Sn 4096

Bug is closed upstream as they say paramiko is the default.
So we can close this too, but we should push the update to 0.8.17.

Please write advisory here that I can push it to SVN.
SRC package is mandatory.
List of updated packages too.

This release fixes many backend errors (S3, ssh-paramiko) and problems using python3.

References:
http://duplicity.nongnu.org/vers8/CHANGELOG.md

Updated packages in core/updates_testing:
========================
duplicity-0.8.17-2.mga7

SRPM:
duplicity-0.8.17-2.mga7.src.rpm

Thanks Mark.
Validating update. Packages and Advisory in Comment 16.
Advisory pushed to SVN.

Keywords: (none) => advisory, validated_update
CC: (none) => sysadmin-bugs

An update for this issue has been pushed to the Mageia Updates repository.

https://advisories.mageia.org/MGAA-2020-0230.html

Status: NEW => RESOLVED
Resolution: (none) => FIXED