Bug 24092 - Update rawtherapee backport to version 5.5
Summary: Update rawtherapee backport to version 5.5
Status: RESOLVED FIXED
Alias: None
Product: Mageia
Classification: Unclassified
Component: Backports (show other bugs)
Version: 6
Hardware: All Linux
Priority: Normal normal
Target Milestone: ---
Assignee: QA Team
QA Contact:
URL:
Whiteboard: MGA6-64-OK
Keywords: validated_backport
Depends on:
Blocks:
 
Reported: 2018-12-29 19:31 CET by Mike Rambo
Modified: 2019-01-15 22:50 CET (History)
4 users (show)

See Also:
Source RPM: rawtherapee
CVE:
Status comment:


Attachments

Description Mike Rambo 2018-12-29 19:31:37 CET
Description of problem:
Update backport to version 5.5
Comment 1 Mike Rambo 2018-12-29 19:41:52 CET
Updated package uploaded for Mageia 6.

Advisory:
========================

Updated rawtherapee package fixes bugs and enables new functionality:

* Filter to remove striping artifacts caused by Phase Detection Auto Focus (PDAF) as seen in Sony cameras, and to remove banding artifacts caused by Nikon's too-aggressive in-camera PDAF correction. These are available for any camera which has a PDAF entry in camconst.json.

* Ability to specify custom working color spaces through the workingspaces.json file.

* Unbounded processing - allows you to decide whether out-of-gamut colors should get clipped.

* Improved support for Canon mRaw format variants.

* New Shadows/Highlights tool (replaced previous one).

* Contrast threshold mask which divides an image into areas of high and low detail, allowing the effect of certain tools to be focused where it matters most and to mitigate the effect on areas where it would be undesirable, for example having the Sharpening tool affect only the in-focus subject without affecting the out-of-focus background. 

* Dual-demosaic algorithms, making use of the new contrast threshold mask, allowing one to use a combination of demosaicing algorithms where one is best for details and the other best for plain areas.

* New color toning methods:
- Grid, allowing you to separately tone the shadows and highlights using two points on a simple color grid.
- Regions, allowing you to tone based on any number of masks. Supports functions from the American Society of Cinematographers Color Decision List (ASC CDL).

* Resizable main histogram with scaling modes:
  Linear
  Log
  Log-Log

* Support for Blackmagic and Canon Magic Lantern lj92 encoded files.

* Allows you to specify how many border rows/columns to discard during 
demosaicing - those who shoot raw video at a standard resolution such as 
1920x1080 will appreciate being able to preserve the dimensions.

* New Soft Light tool which enhances contrast and saturation by emulating the effect of blending an image with a copy of itself in "soft light" blending mode in GIMP.

...and much more...

Updated packages in core/backports_testing:
========================

Assignee: bugsquad => mrambo

Mike Rambo 2018-12-29 19:42:33 CET

Summary: Update backport to version 5.5 => Update rawtherapee backport to version 5.5

Comment 2 Mike Rambo 2019-01-03 21:06:49 CET
(finish the package list)

Updated packages in core/backports_testing:
========================
rawtherapee-5.5-1.mga6

from rawtherapee-5.5-1.mga6.src.rpm


Testing procedure https://bugs.mageia.org/show_bug.cgi?id=12693#c7

Assignee: mrambo => qa-bugs

Comment 3 Herman Viaene 2019-01-11 12:23:19 CET
MGA6-32 MATE on IBM Thinkpad R50e
No installation issues
Repeated tests as per bug 22125 Comment 3
$ rawtherapee

(rawtherapee:17834): Gtk-WARNING **: Allocating size to gtkmm__GtkPaned 0xdd0b2c8 without calling gtk_widget_get_preferred_width/height(). How does the code know the size to allocate?
Premature end of JPEG file
JPEG datastream contains no image
What I did was opening an the same ORF file, changing shadow lighting parameters and save as jpeg file.
Despite the last sentence of the CLI feedback, the jpeg file was created and perfectly viewable in ristretto.
Tried also the CLI:
$ rawtherapee-cli -o p.tif -t -c P7212390.ORF 
RawTherapee, version 5.5, command line.
Output is 16-bit integer.
Processing: P7212390.ORF
TIFFWriteDirectoryTagIfdIfd8Array: Attempt to write value larger than 0xFFFFFFFF in Classic TIFF file..
TIFFWriteDirectoryTagIfdIfd8Array: Attempt to write value larger than 0xFFFFFFFF in Classic TIFF file..
Error saving to: p.tif
No tif file found, which is consistent. But this operation, on the same ORF file worked perfectly with the older version.
I really don't know what to do with this update. I think withholding it is not a viable option, but I'm not really happy.

CC: (none) => herman.viaene

Comment 4 Len Lawrence 2019-01-11 12:56:46 CET
Working in parallel with Herman.

rawtherapee does not seem to require any external libraries.
This package really needs to be tested by somebody versed in image processing.  QA tests shall cover basic functions only.

The application can be run immersively from a gui or with the cli.
The gui allows browsing files in whichever directory is specified at launch.
e.g. for the current directory
$ rawtherapee .
Selected a Canon raw image and brightened it using one of the sliders at the right-hand side.  Saved the file using the save icon at the bottom of the frame, which offers several formats for the output.  Chose TIFF 16-bit, specified a name; saved immediately.  The output file appeared in the ~/Pictures directory alongside a "sidecar" file containing the image profile parameters.  There is a lot of information in that file.
$ ll canon*
-rw-r--r-- 1 lcl lcl 108278212 Jan 11 11:24 canon_1.tif
-rw-r--r-- 1 lcl lcl     11446 Jan 11 11:24 canon_1.tif.out.pp3
$ cat canon_1.tif.out.pp3 | wc -l
623

The image could be displayed with ImageMagick.

Tried the cli tool for image conversions.
Convert RAW file to JPEG format.
$ rawtherapee-cli -js3 -o kodak_1.jpg -c 'KODAK C603 C643 Format 420 CCDI0001.RAW'
RawTherapee, version 5.5, command line.
Output is 8-bit integer.
Processing: KODAK C603 C643 Format 420 CCDI0001.RAW
Cannot use camera white balance.
$ rawtherapee-cli -n -o nikon_1.png -c RAW_NIKON_E5700_SRGB.NEF
Processing: RAW_NIKON_E5700_SRGB.NEF
$ rawtherapee-cli -b8 -t -o olympus_1.tif -c RAW_OLYMPUS_E420.ORF 
Output is 8-bit integer.
Processing: RAW_OLYMPUS_E420.ORF

All output images look fine with IM display.

At this basic level rawtherapee works very well.
Good for 64-bits.

CC: (none) => tarazed25

Len Lawrence 2019-01-11 12:58:43 CET

Whiteboard: (none) => MGA6-64-OK

Comment 5 Len Lawrence 2019-01-11 13:07:51 CET
Tried a 16-bit TIFF output to match Herman's command and it worked.
$ rawtherapee-cli -t -o olympus_2.tif -c RAW_OLYMPUS_SP350.ORF
Output is 16-bit integer.
Processing: RAW_OLYMPUS_SP350.ORF
$ display olympus_2.tif 
display: Incompatible type for "FileSource"; tag ignored. `TIFFFetchNormalTag' @ warning/tiff.c/TIFFWarnings/949.
display: Incompatible type for "SceneType"; tag ignored. `TIFFFetchNormalTag' @ warning/tiff.c/TIFFWarnings/949.
display: Wrong data type 3 for "GainControl"; tag ignored. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/949.

Despite those warnings the displayed image looked fine.
Comment 6 Lewis Smith 2019-01-11 21:22:51 CET
From comment 3:
> But this operation, on the same ORF file worked perfectly with the older
> version.
= a reversion. I agree with Herman's doubts, so withold the advisory in case...

I only have 1 ginormous raw image to play with, so offer this devious suggestion (because bug attachments should not be huge, and I guess both Herman's & Len's examples are - unless you can post here URLs to get them):
IF Herman can send to Len or myself his test raw image files (or post here the URLs to them), and the RawTherapee operations/commands used on them which did not work, we can try them at our end. Note that the unspecified image, first test, gave the same
 Premature end of JPEG file
 JPEG datastream contains no image
as Herman's previous test - even though with a good output.

Conversely, if Len could send Herman *his* test images (or post here the URLs to get them), and the corresponding operations/commands that worked for him, so Herman can see whether he gets the same success? All this is quickly done.

RawTherapee is mature and basic software, it should not revert.

CC: (none) => lewyssmith

Comment 7 Len Lawrence 2019-01-11 23:12:44 CET
@Lewis, re comment 6.
I had already thought about attachments but even after using xz one of them exceeded 2 MB.  The others are probably similar.

I don't exactly know how to make them available over the internet.  I have not used my dropbox account for a while - forget how to open it.  Have not had the time to investigate that.  Later maybe.
Comment 8 Len Lawrence 2019-01-11 23:44:54 CET
Re comment 7.
Managed to open Dropbox and have deposited 5 raw images there.  Compression does very little good so we are stuck with anything up to 23MB.  That is not a problem for me but it would be for people with lower bandwidth connections.

More reading to do.
Comment 9 Herman Viaene 2019-01-12 10:28:13 CET
Files in https://drive.google.com/open?id=1hfwOEeP612L_181qmI5QDRTnFEBa3VlR
Tell me when you've downloaded them, so I can delete the folder.
Comment 10 Len Lawrence 2019-01-12 11:25:31 CET
Re comment 9; Thanks Herman, done.  Keep them until Lewis confirms.
Mine are in DropBox - I have to send links to you by email for each image.  Need some spare time for that.  Later.
Comment 11 Len Lawrence 2019-01-12 18:07:43 CET
OK Herman.  The four .ORF images you provided present no problems here.  Converted all four to other image formats, PNG, JPEG, 8 and 16-bit TIFF.  All displayed properly although display issued a warning for the 16-bit tif.
display: Incompatible type for "FileSource"; tag ignored. `TIFFFetchNormalTag' @ warning/tiff.c/TIFFWarnings/949.
display: Wrong data type 3 for "GainControl"; tag ignored. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/949.

Despite that there was nothing wrong with the displayed image.

So, this does look like a regression for 32-bit.  Maybe the fix does not take account of differences in architectures.  The "value larger than 0xFFFFFFFF" message means larger than -1 which is a classic sign of such an error; pushing a positive number beyond the 32-bit boundary switches it negative.
Comment 12 Len Lawrence 2019-01-12 18:35:28 CET
Correction, pushing a 32-bit integer past the 31-bit boundary switches it negative.
Comment 13 Herman Viaene 2019-01-13 10:43:22 CET
@ Len: yes, that looks like a good-oldfahioned overflow.
Anyway, repeated the CLI commands with your files and got:
$ rawtherapee-cli -t -o olympus_2.tif -c RAW_OLYMPUS_SP350.ORF
RawTherapee, version 5.5, command line.
Output is 16-bit integer.
Processing: RAW_OLYMPUS_SP350.ORF
TIFFWriteDirectoryTagIfdIfd8Array: Attempt to write value larger than 0xFFFFFFFF in Classic TIFF file..
TIFFWriteDirectoryTagIfdIfd8Array: Attempt to write value larger than 0xFFFFFFFF in Classic TIFF file..
Error saving to: olympus_2.tif
and of course no tif file to be seen.
$ rawtherapee-cli -js3 -o kodak_1.jpg -c KODAK_C603_C643_FORMAT422_CCDI0001.RAW 
RawTherapee, version 5.5, command line.
Output is 8-bit integer.
Processing: KODAK_C603_C643_FORMAT422_CCDI0001.RAW
Cannot use camera white balance.
$ display kodak_1.jpg 
jpg picture looks OK
$ rawtherapee-cli -n -o nikon_1.png -c RAW_NIKON_E5700_SRGB.NEF
RawTherapee, version 5.5, command line.
Output is 8-bit integer.
Processing: RAW_NIKON_E5700_SRGB.NEF
$ display nikon_1.png 
Looks OK as well.

So the tif conversion works OK with our two ORF files in 64-bit, but fail in 32-bit. Nice........
Comment 14 Lewis Smith 2019-01-13 12:20:35 CET
(In reply to Herman Viaene from comment #9)
> Files in https://drive.google.com/open?id=1hfwOEeP612L_181qmI5QDRTnFEBa3VlR
> Tell me when you've downloaded them, so I can delete the folder.
Got them all (Len likewise), thanks a lot. Delete away.
P.S. Thanks for persuing the problem you found.

M6/x64: Cross-checking some of Herman's tests with his images.
BEFORE update: rawtherapee-5.3-1.mga6
 $ rawtherapee
(rawtherapee:25132): Gtk-CRITICAL **: gtk_box_gadget_distribute: assertion 'size >= 0' failed in GtkScrollbar       [*n, but it started OK in the end]
[but baffling to do basic operations like opening & saving]
I selected a couple, fiddled randomely, and saved as JPEG (only 8-bit offered), with default (high) quality level - why the .jpg files are so big, perhaps.
 Premature end of JPEG file
 JPEG datastream contains no image
 Premature end of JPEG file
 JPEG datastream contains no image
Standard errors! The output is however good.
-rw-rw-r-- 1 lewis lewis  1922388 Ion  13 10:30 P7212389.jpg
-rw-rw-r-- 1 lewis lewis 14127616 Ion  12 01:03 P7212389.ORF
-rw-rw-r-- 1 lewis lewis  2678491 Ion  13 10:36 P7212391.jpg
-rw-rw-r-- 1 lewis lewis 14127616 Ion  12 01:03 P7212391.ORF

 $ rawtherapee-cli -o x.tif -t -c P7212389.ORF
RawTherapee, version 5.3, command line
Processing: P7212389.ORF
-rw-rw-r-- 1 lewis lewis 49934642 Ion  13 10:55 x.tif
 $ rawtherapee-cli -o y.tif -t -c P7212391.ORF
RawTherapee, version 5.3, command line
Processing: P7212391.ORF
-rw-rw-r-- 1 lewis lewis 49934642 Ion  13 10:56 y.tif
 Both outputs good, but with console errors:
$ display x.tif
display: Incompatible type for "FileSource"; tag ignored. `TIFFFetchNormalTag' @ warning/tiff.c/TIFFWarnings/949.
display: Wrong data type 3 for "GainControl"; tag ignored. `TIFFReadCustomDirectory' @ warning/tiff.c/TIFFWarnings/949.
--------------------------------------------------------------------
AFTER update: rawtherapee-5.5-1.mga6        [caught out by Backport]
 $ rawtherapee
yielded a pop-up:
"The default profile for raw photos could not be found or is not set
Please check your profiles directory, it may be missing or damaged
${G]/Auro-Matched Curve - ISO Low will be used instead"
OK loads the program; but with a new error msg as per Herman c3:
 (rawtherapee:24548): Gtk-WARNING **: Allocating size to gtkmm__GtkPaned 
 0x84b8940 without calling gtk_widget_get_preferred_width/height(). How does
 the code know the size to allocate?

Fiddling P7212389.ORF and P7212391.ORF and saving as JPEG worked (with same errors) as before,  also the command line TIF conversions.
Second Len's OK. BUT...
=======================
@ Mike
Herman has definitely found a 32-bit problem. What to do?
* This is posted as a Backport rather than Update. Does a jump from 5.3 to 5.5 warrant that? Can we change it to an Update?
* Backports do not have advisories...
* If the advisory in c1 is to go out somehow, we could add to it warning of possible problems on 32-bit systems.
* Should we raise a bug?

Keywords: (none) => feedback

Comment 15 Mike Rambo 2019-01-14 17:42:44 CET
I had just pushed the new version in cauldron because it is the latest and we always put the latest in cauldron if we can. Now I'm wondering if cauldron 32 bit has the same problem. But anyway...

My google-fu didn't turn up any known problem with rawtherapee 5.5 32 bit. I pushed a backport for mga6 because it adds a lot of nice things that I thought it would be nice to provide for those on mga6. At this point I don't really know what to do with this. We could just drop the backport IMO. Mageia 6 already has 5.3 backported and that is also a nice package. I guess we could just leave it at that. I don't know what the process looks like for that or whether I need to do something to make it happen.

I didn't know backports didn't need any advisory or note. I just did it out of habit.
Comment 16 Lewis Smith 2019-01-14 20:27:56 CET
(In reply to Mike Rambo from comment #15)
> Now I'm wondering if
> cauldron 32 bit has the same problem.
Something our M7 testing can (now should) specifically look for.

> I pushed a backport for mga6 because it adds a lot of nice things that I
> thought it would be nice to provide for those on mga6.
Decent of you.

> At this point I don't
> really know what to do with this. We could just drop the backport IMO.
> ...
> I don't know what the process looks like for
> that or whether I need to do something to make it happen.
Best to ask on your side (packagers/devs) about these things.
If it is decided to pull the backport, it would need removing from backports-testing; and this bug can be closed with an appropiate reason.

> I didn't know backports didn't need any advisory or note.
No worry. It *is* curious; I think they should.

Herman & Len might have opinions about what to do.
Comment 17 Mike Rambo 2019-01-15 15:28:27 CET
Opened a bug upsteam. Appears Debian has run into the same thing.

https://github.com/Beep6581/RawTherapee/issues/5141
Comment 18 Lewis Smith 2019-01-15 20:20:33 CET
(In reply to Mike Rambo from comment #17)
> Opened a bug upsteam. Appears Debian has run into the same thing.
Thanks Mike. So we are up with the big boys.

I have had second thoughts about this backport: I think we *should* let it out (simply by validating it), even with this known bug - because of all the work done both to prepare and test it. It would be a shame to throw that away. The chances of a user falling foul of the fault are remote; and they can always, and easily, --downgrade the package if necessary.

What do you others think? If you agree - please validate it!
Already removed the feedback marker.

Keywords: feedback => (none)

Comment 19 Mike Rambo 2019-01-15 21:25:24 CET
fwiw - If/when upstream produces a fix for this I will package an update if it is in time for mga6. And since I expect cauldron suffers from the same problem, I hope they do.

I agree that it should be safe to let it out. I know at least one other person has used the current backport aside from me, but I don't expect there are many, and even less on 32 bit. I don't think very many would be affected by this.

But I'm a packager - I'll let someone in QA make that call.
Comment 20 Len Lawrence 2019-01-15 21:32:42 CET
Yes, agreeing with Mike and Lewis.  Validating.

Keywords: (none) => validated_backport

Comment 21 Thomas Backlund 2019-01-15 22:50:27 CET
Packages moved.

Resolution: (none) => FIXED
Status: NEW => RESOLVED
CC: (none) => tmb


Note You need to log in before you can comment on or make changes to this bug.