[info-gnuastro] Gnuastro-0.8.36 released

[Mohammad Akhlaghi]

------------------------------ NOTICE ------------------------------
This release contains an important bug fix that directly affects
NoiseChisel as described below. If you use NoiseChisel, please update.
--------------------------------------------------------------------

Hi everyone,

I am happy to announce the first Gnuastro alpha-release after version
0.8. As described in the NEWS file below many convenience features
have been added to the programs. But the main motivation behind this
release was a bad bug that was just found and fixed in Gnuastro's
library and directly affects programs like NoiseChisel.

As a summary, the image of the links below, can display the effect of
this bug (and show how important it is to update your version of
Gnuastro). Note that both links point to the same image (on different
servers in case one is not available).

https://akhlaghi.org/gnuastro/bug-55740.jpg
https://savannah.gnu.org/support/download.php?file_id=46317

To get a little more technical: this bug occurred in Gnuastro's
nearest-neighbor interpolation library (in particular the function
`gal_interpolate_close_neighbors'). This function is used during
several steps of NoiseChisel. In this function, the interpolated value
for each input element (tile in NoiseChisel) is defined to be the
median of the `N' nearest non-blank neighbors to that element (value
to NoiseChisel's `--interpnumngb' option).

However, because the sorting flag wasn't reset for every element (a
single line!), the values weren't being sorted (as it should be when
calculating a median). Therefore, the middle value of the unsorted
array was reported as the median! This created the diamond-like shapes
you see around the outlier tiles before this bug fix. The reason they
are diamond shaped (as opposed to a circle) is that (for performance
reasons) we define distance in searching for neighbors in the "taxicab
metric" (or "Manhattan distance").

This interpolation is used by NoiseChisel and Statistics to associate
a value to all the tiles and cover the input image area. In the case
of NoiseChisel this includes the thresholding step, the initial Sky
estimation step an the final Sky estimation step. Therefore it
fundamentally affects the results if there are local outliers in the
tiles.

This bug is present in Gnuastro since version 0.2-177 (Git commit hash
`b03f7f96aa3fdd5', on March 22nd, 2017). So if you use NoiseChisel in
your work, you will probably notice an improvement in the results on
certain datasets once you update your installation.

Here is the compressed source, to uncompress Lzip tarballs, see [2]:
  https://alpha.gnu.org/gnu/gnuastro/gnuastro-0.8.36-9e13.tar.lz    (3.3MB)

Here are the GPG detached signatures, to check the validity of the
tarballs using the GPG detached signature see [3].
  https://alpha.gnu.org/gnu/gnuastro/gnuastro-0.8.36-9e13.tar.lz.sig (833B)

Here are the MD5 and SHA1 checksums (other ways to check if the
tarball you download is what we distributed):
7a1f39a53befd7f0d8bde219c40cb4d4  gnuastro-0.8.36-9e13.tar.lz
a8d444abc016fe9bf3ab68c8b2d00f8c5b5ca60e  gnuastro-0.8.36-9e13.tar.lz

If any of Gnuastro's programs or libraries are useful in your work,
please cite _and_ acknowledge them. For citation and acknowledgment
guidelines, run the relevant programs with a `--cite' option (it can
be different for different programs). Citations _and_ acknowledgments
are vital for the continued work on Gnuastro, so please don't forget
to support us by doing so.

I am very grateful to (in alphabetical order) to Roberto Baena Gallé,
Leindert Boogaard, Raúl Infante Sainz, David Valls-Gabaud and Ignacio
Trujillo for the great suggestions and bug reports.

This tarball was bootstrapped (created) with the tools below. Note
that you don't need these to build Gnuastro from the tarball, these
are the tools that were used to make the tarball itself. They are only
mentioned here to be able to reproduce/recreate this tarball later.
  Texinfo 6.5
  Autoconf 2.69
  Automake 1.16.1
  Help2man 1.47.8
  ImageMagick 7.0.8-28
  Gnulib v0.1-2445-g35e462817
  Autoconf archives v2019.01.06-35-gfb869b7

The dependencies to build Gnuastro from this tarball are described
here:
  https://www.gnu.org/s/gnuastro/manual/html_node/Dependencies.html

Best wishes,
Mohammad

--
Postdoctoral research fellow,
Instituto de Astrofísica de Canarias (IAC),
Calle Vía Láctea, s/n, E38205,
San Cristóbal de La Laguna, Tenerife, Spain.





[1] NEWS for Gnuastro 0.8.36

** New features

  All programs:
   --checkconfig: print the names and values given to options as they are
     parsed on the command-line or in various configuration files (the
     configuration file name is also printed). This can be very useful in
     debugging (finding which configuration file is responsible for a given
     option's value).

  Arithmetic:
   - The new `tofile-' and `tofilefree-' operators will save the top
     operand into a file. They can be very handy in debugging/understanding
     an Arithmetic command (especially as it gets complicated), or to
     produce multiple files/extensions with a single call to Arithmetic.
   - Four new operators have been added to allow coadding multiple datasets
     into one using sigma-clipping: `sigclip-number', `sigclip-mean',
     `sigclip-median', and `sigclip-std'. These are very useful when
     several inputs have strong outliers that affect the median, or the
     mean is required.
   --wcsfile and --wcshdu: these two options can be used to specify a
     different file for reading the WCS of the output. This is useful when
     the default (the WCS of the first dataset that is read) is not the
     required one.

  Fits:
   - Add "title" to group FITS keywords with `--write=/,"title name". This
     "title" is composed of two keyword records: a blank one, followed by
     another starting with `/' and ending in any string given to this
     option. Classifying the keywords into contextually similar groups
     greatly helps in visual inspection and is encouraged.
   - Calculate and write `CHECKSUM' and `DATASUM' integrity keywords into
     the specified header using `--write=checksum' (for both) or
     `--write=datasum' (only for `DATASUM').
   --verify: confirm if the `DATASUM' and `CHECKSUM' keyword values agree
     with the specified HDU's content and/or data.
   --copykeys: Copy several keyword records (in a given range) from one
     FITS HDU/extension into another (possibly in another file).
   --outhdu: The name/number of the output HDU (for `--copykeys').

  Match:
   - All the columns from one of the input catalogs can now be merged with
     any of the columns of the second using the special `_all' name of
     `--outcols'. For example the output of `--outcols=a_all,b5' will
     contain all the columns from the first input and the 5th column of the
     second input. This greatly simplifies the merging of different table
     columns into one.

  NoiseChisel:
   --snthresh: Manually set the signal-to-noise ratio of true
     pseudo-detections. With this option, NoiseChisel will not attempt to
     find pseudo-detections over the noisy regions of the dataset, but will
     directly go onto applying the manually input value.
   - Several new options have been added to remove NoiseChisel's dependence
     on values that were hard-coded in its source and thus not modifiable
     at run-time by the user. To allow full configurability, these steps
     can also be configured by the user.
     --pseudoconcomp: allows setting the connectivity (4 or 8, in a 2D
       image) to define separate pseudo-detections. If its stronger,
       pseudo-detections that are touching on the corner will be identified
       as one.
     --dopening: The number of openings to do after applying `--dthresh'.
     --dopeningngb: The connectivity (4 or 8) to use for `--dopening'.

  Library:
    GAL_BLANK_LONG: new macro for the `long' type blank value.
    GAL_BLANK_ULONG: new macro for the `unsigned long' type blank value.
    gal_blank_number: Return the number of blank elements in a dataset.
    gal_statistics_outlier_cumulative: Uses flatness of the cumulative
       distribution to find outliers.

** Removed features

** Changed features

  Arithmetic:
   - `num' operator is renamed to `number'.
   - `numvalue' operator is renamed to `numbervalue'.
   - `--dontdelete' will append the output to any existing file. Note that
     this change is only in Arithmetic, other programs will still just
     complain and abort.

  ConvertType:
   --forcemin & --forcemax: until now, `--flminbyte' and `--flmaxbyte' were
     used to force the range of conversion to color channels (even if the
     range is beyond the limits of the dataset). With the introduction of
     color maps in Gnuastro 0.8, it is also necessary to force a range on
     non-byte datasets. It is thus necessary to use a more generic name.

  MakeCatalog:
   --std: Until now, this option would measure the mean standard deviation
     under the label. But this is not a statistically meaningful measure
     for the Sky standard deviation and could be incorrectly used. From now
     on, this option measures the square root of the mean variance, or root
     mean square (the correct definition of the Sky standard deviation).

  NoiseChisel:
   --ignoreblankintiles: Until now `--ignoreblankinsky', would specify if
     blank values should also be written into the tiled Sky and Sky
     standard deviation outputs. But NoiseChisel can optionally produce
     many more tiled outputs (for example with `--checkqthresh'). So the
     option was renamed to `--ignoreblankintiles' to highlight that the
     status of blank elements can be set in all tiled outputs.

  Statistics:
   --ignoreblankintiles: similar to same option in NoiseChisel.

** Bugs fixed
  bug #55313: Fits program writing --write values in reverse order
  bug #55333: Fits program crash when --write or --update have no value.
  bug #55439: Arithmetic segmentation fault when reusing dataset name.
  bug #55478: Memory mapping crashes when .gnuastro is not writable.
  bug #55491: NoiseChisel crash when no tiles good for quantile 
thresholding.
  bug #55544: Arithmetic's output WCS with where operator is not as 
expected.
  bug #55740: Diamond shapes in nearest-ngb interpolation affecting 
NoiseChisel.





[2] Lzip has better compression ratio and archival features compared
to the common `.gz' or `.xz' formats. Therefore Gnuastro's alpha/test
releases are only in this format, but for historical reasons we also
include `.gz' tarballs in the official releases. If you don't have
Lzip (you can check with `lzip --version' command), download and
install it from its webpage:

  https://www.nongnu.org/lzip/lzip.html

If Lzip is present and you use GNU Tar, then the single command below
should uncompress and un-pack the tarball:

  $ tar xf gnuastro-0.8.36-9e13.tar.lz

If the command above doesn't work, you have to un-compress and un-pack
it with two separate commands (or use a pipe to feed the output of the
first into the second with this command:
`lzip -cd gnuastro-0.8.36-9e13.tar.lz | tar -xf -'):

  $ lzip -d gnuastro-0.8.36-9e13.tar.lz
  $ tar xf gnuastro-0.8.36-9e13.tar





[3] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify gnuastro-0.8.36-9e13.tar.gz.sig

If that command fails because you don't have the required public key,
then run this command to import it:

  gpg --keyserver keys.gnupg.net --recv-keys 71E899012D174B66

and rerun the 'gpg --verify' command.