Bug 11507 - image registration task ccmap fails on iraf's provided test case
Summary: image registration task ccmap fails on iraf's provided test case
Status: RESOLVED FIXED
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: Cauldron
Hardware: x86_64 Linux
Priority: Normal major
Target Milestone: Mageia 4
Assignee: Joseph Wang
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-10-21 23:35 CEST by Chris Denice
Modified: 2013-11-02 19:58 CET (History)
0 users

See Also:
Source RPM: iraf-2.16-6.mga4.src.rpm
CVE:
Status comment:


Attachments

Description Chris Denice 2013-10-21 23:35:37 CEST
Hi Joseph,
I am going on into testing iraf and I found that the task ccmap, quite crucial to register images by inputing a list of star coordinates, fails to produce a fit and/or produce a completely crazy result.

The easiest way to check that it goes wrong is to take the iraf tutorial example provided in the help (help ccmap, section EXAMPLE) that I reproduce here:

0) if you do not start from a fresh install, type "unlearn ccmap"

1) create a file, named "coords", containing this list
13:29:47.297  47:13:37.52  327.50  410.38
13:29:37.406  47:09:09.18  465.50   62.10
13:29:38.700  47:13:36.23  442.01  409.65
13:29:55.424  47:10:05.15  224.35  131.20
13:30:01.816  47:12:58.79  134.37  356.33

2) imcopy dev$pix pix
(that's m51, std test image for iraf)

3) hedit pix epoch 1987.26
(can be forgotten)

4) ccmap coords coords.db image=pix results=STDOUT xcol=3 ycol=4 lngcol=1 latcol=2 inter-

should produce the result:

-----------------------------------------------------------------------

# Coords File: coords  Image: pix
    #     Database: coords.db  Record: pix
    # Refsystem: j2000  Coordinates: equatorial FK5
    #     Equinox: J2000.000 Epoch: J2000.00000000 MJD: 51544.50000
    # Insystem: j2000  Coordinates: equatorial FK5
    #     Equinox: J2000.000 Epoch: J2000.00000000 MJD: 51544.50000
    # Coordinate mapping status
    #     XI fit ok.  ETA fit ok.
    #     Ra/Dec or Long/Lat fit rms: 0.229  0.241   (arcsec  arcsec)
    # Coordinate mapping parameters
    #     Sky projection geometry: tan
    #     Reference point: 13:29:48.129  47:11:53.37  (hours  degrees)
    #     Reference point: 318.735  273.900  (pixels  pixels)
    #     X and Y scale: 0.764  0.767  (arcsec/pixel  arcsec/pixel)
    #     X and Y axis rotation: 179.110  358.958  (degrees  degrees)
    # Wcs mapping status
    #     Ra/Dec or Long/Lat wcs rms: 0.229  0.241   (arcsec  arcsec)
    # 
    #                     Input Coordinate Listing
    # X      Y       Ra          Dec        Ra(fit)      Dec(fit)    Dra    Ddec
    # 
    327.5  410.4  13:29:47.30  47:13:37.5  13:29:47.28  47:13:37.9  0.128 -0.370
    465.5   62.1  13:29:37.41  47:09:09.2  13:29:37.42  47:09:09.2 -0.191 -0.062
    442.0  409.6  13:29:38.70  47:13:36.2  13:29:38.70  47:13:35.9  0.040  0.282
    224.3  131.2  13:29:55.42  47:10:05.2  13:29:55.40  47:10:05.1  0.289  0.059
    134.4  356.3  13:30:01.82  47:12:58.8  13:30:01.84  47:12:58.7 -0.267  0.091
    
------------------------------------------------------------------------------

As you can check, with our IRAF version, the fit fails with singular (while it should not) and the fitted coordinates are crazy, so registration will fail.

our output ---->



# Refsystem: j2000  Coordinates: equatorial FK5
#     Equinox: J2000.000 Epoch: J2000.00000000 MJD: 51544.50000
# Insystem: j2000  Coordinates: equatorial FK5
#     Equinox: J2000.000 Epoch: J2000.00000000 MJD: 51544.50000

# Coords File: coords  Image: pix
#     Database: coords.db  Solution: pix
# Coordinate mapping status
#     Warning singular XI fit.  Warning singular ETA fit.
#     Ra/Dec or Long/Lat fit rms: 108.  127.   (arcsec  arcsec)
# Coordinate mapping parameters
#     Sky projection geometry: tan
#     Reference point: 13:29:48.130  47:11:53.44  (hours  degrees)
#     Reference point: INDEF  INDEF  (pixels  pixels)
#     X and Y scale: 0.000  0.000  (arcsec/pixel  arcsec/pixel)
#     X and Y axis rotation: 0.000  0.000  (degrees  degrees)
# Wcs mapping status
#     Ra/Dec or Long/Lat wcs rms: 108.  127.   (arcsec  arcsec)

# Input Coordinate Listing
#     Column 1: X (pixels)
#     Column 2: Y (pixels)
#     Column 3: Ra / Longitude (hours)
#     Column 4: Dec / Latitude (degrees)
#     Column 5: Fitted Ra / Longitude (hours)
#     Column 6: Fitted Dec / Latitude (degrees)
#     Column 7: Residual Ra / Longitude (arcseconds)
#     Column 8: Residual Dec / Latitude (arcseconds)

   327.500    410.380  13:29:47.297  47:13:37.52  13:29:48.119  47:11:53.25 -8.377 104.27
   465.500     62.100  13:29:37.406  47:09:09.18  13:29:48.119  47:11:53.25  -109.3 -164.1
   442.010    409.650  13:29:38.700  47:13:36.23  13:29:48.119  47:11:53.25  -95.95 102.98
   224.350    131.200  13:29:55.424  47:10:05.15  13:29:48.119  47:11:53.25  74.490 -108.1
   134.370    356.330  13:30:01.816  47:12:58.79  13:29:48.119  47:11:53.25  139.55 65.543


Let me know if you need additional info.
cheers,
chris
Comment 1 Joseph Wang 2013-10-22 15:40:54 CEST
Thanks.  I'll take a look at this.

Also how far have you been able to test iraf?
Comment 2 Chris Denice 2013-10-22 15:57:37 CEST
:)

it is pretty good Joseph, that's already professionally usable, congrats!

I have been able to remove dark frames, zeroing images and applying flat fielding corrections. The only step missing is having a good WCS header, hence the need of ccmap, to do the final alignement and image stacking. If that work, I think you can sell it to astronomers without doubt (that ccmap may be some upstream bug or some 32bits/64 conversion, I looked a bit but I have no clue).

I have also used ds9 and ximtool for downloading catalogs and images from remote servers, so far so good!

thanks.
Comment 3 Joseph Wang 2013-10-24 20:16:26 CEST
I've been able to reproduce and I'll do a deep dive to figure out what is going on.
Comment 4 Joseph Wang 2013-10-28 15:26:56 CET
OK.  It's not a 32/64 bit issue since it happens on 32-bit.  I'll dump out a bunch of things to see where the trouble is.
Comment 5 Joseph Wang 2013-10-29 20:45:22 CET
This is bizzarre

on l186 in gsacptsd.x

                        do i = 0, npts-1 {
                           sum = sum + Memd[bw+i] *
                                XBASIS(bbxptr+i) * YBASIS(bbyptr+i)
                           call printf("")
                        }
                        MATRIX(mindex) = sum

it works if I put in a the call printf, but it stops working if I take it out.

bizarre
Comment 6 Chris Denice 2013-10-29 20:54:05 CET
You're indeed diving deep :)

Might it be some wild pointer, or simply by adding printf() you are preventing some optimizations to take place that break the code?
Comment 7 Joseph Wang 2013-10-29 22:29:55 CET
It's the optimizer.  What happening is that iraf simulates dynamic memory by creating an array of length one to locate the bottom of the heap, and then performing operations that are out of the array bounds.

Unfortunately, the optimizer sees that the loop is acting on out of bounds memory, and says "if it's out of bounds, the behavior is undefined so I can get rid of the loop altogether."

The solution is to add the option -fno-aggressive-loop-optimizations in the f77.sh wrapper, and to heavily comment why that option is essential.

I'll check in a fix.
Comment 8 Chris Denice 2013-10-29 22:37:17 CET
well done! that is *good* hunting!!

But if confirmed, the iraf guys need a serious bug report in their face, that is quite an ugly way of progamming and they should fix it upstream. I am amazed that no one complained before :)
Comment 9 Joseph Wang 2013-10-29 22:41:18 CET
My guess is that this may be an issue that is specific to gcc 4.8.

The dynamic memory handling of IRAF is extremely ugly since they go through a lot of nasty tricks to get dynamic memory working with fortran 77.  The trouble is that those tricks are not necessary with fortran 90, but IRAF is stuck with them.
Comment 10 Joseph Wang 2013-10-29 23:41:23 CET
fixed in cauldron

Status: NEW => RESOLVED
Resolution: (none) => FIXED

Comment 11 Chris Denice 2013-10-30 21:28:14 CET
Hi Joseph,
ok the fits is good for me too now, but there is a problem in updating the header of the image. In the ccmap help, this is the tutorial example 5)

ccmap coords coords.db image=pix results=STDOUT xcol=3 ycol=4  \
    lngcol=1 latcol=2 refpoint=user lngref=13:27:46.9 latref=47:27:16    \
    refsystem=b1950.0 inter- update+

that should update the header of the image to be WCS (refpoint=user lngref=13:27:46.9 latref=47:27:16 refsystem=b1950.0 can be forgotten, the important option is update+).

Now checking the header with

imheader pix l+

should have the following fields:

WCSDIM  =                    2
CTYPE1  = 'RA---TAN'
CTYPE2  = 'DEC--TAN'
CRVAL1  =     202.471969550729
CRVAL2  =     47.1967667056819
CRPIX1  =     250.255619786203
CRPIX2  =     266.308757328719
CD1_1   =  -2.1224568721716E-4
CD1_2   =  -3.8136850875221E-6
CD2_1   =  -3.2384199624421E-6
CD2_2   =  2.12935798198448E-4



On our test case, we get:

WCSDIM  =                    2
CTYPE1  = 'RA---TAN'
CTYPE2  = 'DEC--TAN'
CRVAL1  =  2.100808536427E-312
CRVAL2  =     47.1981766432095
CRPIX2  =     273.980754205693
CDELT1  =                   0.
CDELT2  =  2.12934727219336E-4
CD2_2   =  2.12934727219336E-4

some fields are missing, like CRPIX1, and others are uninitialized, like CRVAL1. It sounds exactly the same kind of bug you fix, but elsewhere in the code. The fit seems good though, so that's coming from some routine making the translation.

I also tried with another task, ccsetwcs using coords.db and it produces also missing and uninitialized fields in the header.

Could you have a look again?

Thanks!
cheers,
chris.

Status: RESOLVED => REOPENED
Resolution: FIXED => (none)

Comment 12 Joseph Wang 2013-10-30 22:53:25 CET
OK.  Will look into this.
Comment 13 Joseph Wang 2013-10-31 13:09:13 CET
OK.  I haven't put in a fix but I know what the problem is.

If you have several double assigns in a row, gcc will attempt to optimize it with a streaming double instruction.

The trouble is that for this to work, the doubles have to be aligned on a 16-byte boundary, which is not guaranteed by salloc.

If may take me a bit of time to work out the pointer numerology.
Comment 14 Joseph Wang 2013-10-31 23:46:51 CET
Fix attempted in revision 2.16-8.mga4

For SSE2 to work doubles have to be aligned on 128-bit boundaries so I changed the IRAF memory management so that happens.

Status: REOPENED => RESOLVED
Resolution: (none) => FIXED

Comment 15 Chris Denice 2013-11-02 01:14:51 CET
Thanks Joseph, tested it on other images as well and works fine, well done!

I can pursue with other tests now, like alignement and stacking.

Cheers,
chris.
Comment 16 Joseph Wang 2013-11-02 19:58:46 CET
FYI, I just uploaded a new iraf in which I set everything to optimization level O2.

Setting things to O3 caused the problems to reappear.

Note You need to log in before you can comment on or make changes to this bug.