Error Correlation Study of IASI and CrIS Bias Using SNOs

Wed, Aug 9, 2017 C.L.Hepplewhite

Scope and Purpose

This is a supplement in support of studies to evaluate and compare the channel to channel correlation of the bias between observation and calculations for the CrIS and IASI instruments.

The question relates to the application of the hyperspectral data from CrIS and IASI to numerical weather prediciton (NWP), in which one has to correctly allow for inter-channel observation errors that may be correlated.

An estimate of the observational error is available to us with the simulataneous nadir observations (SNOs) in which the analysis fields are co-located to the observations and the forward model (RTA) called SARTA - is used to estimate the top of atmosphere (TOA) radiance as measured by the sensor.

The best data to use for this study would be tropical clear sky ocean observations. Unforntunately CrIS and IASI have very different orbits so there are no close observation pairs in this region. A pseudo-SNO tropical pairing can be constructed with large separations, and it is possible that with a suitably large data set a meaningful comparison can be made.

For the main study true SNO pairs are obtained at the high latitudes from the clear subset RTP data for 2016, using ECMWF model fields and a common forward model (RTA).

The correlated observational error, (by error we mean the difference between the observed and calcuated radiance), will be influenced by the native noise of the measurements. The CrIS has lower noise than IASI, based on in-flight calibration data as shown in figure: (need a reference).

It is therefore of interest to know what the IASI error correlation looks like if the IASI data are deconvolved to the CrIS instrument line shape (ILS) and with the corresponding noise characteristic. Results are shown next section.

RTP Data - Overview

The source data that are utlized in this study are the clear subset RTP files provded by S. Sbuczkowski, and which have either the ERA or the ECMWF model fields and the SARTA TOA calculated radiance. Data from

Four studies are performed:
- pseudo-SNOs in the tropical oceans. May and June 2012, with ERA model fields.
- presudo-SNOs in the high-latitudes in the year 2012, with ERA model fields.
- True SNOs in the high latitudes in the year 2012, with ERA model fields and common SARTA.
- True SNOs in the high latitudes in the year 2016 with ECMWF model fields and common SARTA.

Original root directories of the RTP data are for CrIS and IASI respectively:
**/asl/rtp/rtp_cris_ccast_lowres/clear/ **
and
/asl/rtp/rtp_iasi1/clear/

The SNO data derived from the clear RTP subsets are located in:
/asl/s1/chepplew/data/sno/iasi_cris/LR/
and
/asl/s1/chepplew/data/sno/iasi2_cris/LR/

Data Production: Method and Code

The RTP generation and RTA calculated fields are described elsewhere (really - where??)

The IASI:CrIS Clear SNO from RTP subsets are formed using:

/home/chepplew/projects/sno/makeSNO/make_IASI_CRIS_SNO_frmRTP.m
and batch script:
batch_IASI_CRIS_SNO_frmRTP.m

The IASI CrIS tropical pseudo-SNO pairs are formed using:

/home/chepplew/projects/sno/makeSNO/make_IASI_lrCRIS_tsno_frm_rtp.m
and the batch scripts:
batch_IASI_lrCRIS_tsno_frm_rtp.m
batchJobs/batch_IASI_lrCRIS_tsno_frm_rtp.sh
and a list of dates for the batch processor:
batchJobs/jobDates.drv

All other IASI CrIS high-latitude sets are formed using:

/home/chepplew/projects/sno/makeSNO/make_IASI_lrCRIS_tsno_frm_rtp.m
and associated batch and driver files.

Data Analysis: Method and Code

The IASI:CrIS clear SNO correlation study uses matlab code:

/one/chepplew/gitLib/cris-corr/corr_clear_sno.m

The IASI:CrIS Tropical psuedo-SNO correlation study uses:

/home/chepplew/projects/corrcoef/ana_clear_tosno.m
which is based very closely on:
~strow/Work/Cris/Noaa/Ccorr/corr_processing.m
and associated scripts.

In all proceeding analysis and results a simple screening is applied for outliers based on the value of the bias error being greater than 6-sigma of the sample mean. This removes less than about 1% of samples. In addition, in each case the ILS radiances are hamming apodized.

The comparison of interest is how the CrIS correlation looks compared to that of IASI having been convolved to the CrIS ILS, but with CrIS having the equivalent of the IASI noise, which is larger than the CrIS noise. In other words we want the final CrIS noise to be the same as the IASI-to-CrIS noise. Therefore we want to modify the CrIS bias error, by an amount $ xn $, given by: $ (xn)^2 = (icn)^2 - (ncr)^2 $ where, $ icn $ is the IASI-to-CrIS noise, and $ ncr $ is the unmodified native CrIS noise.

The graphics shown here are also available in various formats (fig, pdf, png) on the maya computer at:
/home/chepplew/projects/corrcoef/figs

Results

IASI:CrIS clear RTP SNOs

IASI-1 and CrIS Overview plots:

SNO sets1 and 2: CrIS:IASI-1 and CrIS:IASI-2

It is important to remember that the actual observations determined to be cloud free from the two IASI instruments are different, and therefore the SNO sets are different. We refer to them as set 1 for CrIS:IAS-1 and set 2 for CrIS:IASI-2.

In this figure note that IASI-2 bias (Obs-calc) is slightly larger than that for IASI-1, and in the window region the Obs are colder than the calculations, perhaps due to effect of residual un-modelled clouds i nthe clear scenes. These observations have NOT had their noise adjusted (what I call native signals).

Bias Correlation plots

The pseudo-Tropical SNO data with ERA model:

As mentioned above these are NOT SNOs as the separation criteria are less than 110 km and 3 hrs 20 minutes. The matched pairs therefore include observations from adjacent orbits. They are restricted to +/- 40-deg latitude oceans, and for clear scenes there is expected to be small changes in brightness temperature. The sample provides 39,296 pairs. These are distributed as shown in this figure:

The temporal separation as a function of SNO sample through the two months is shown in this figure: You can see that for 110 km maximum separation there are no SNOs with time delay less than about 1 hr 20 minutes.

#+CAPTION: Figure 3. #+ATTR_HTML: :width 500px [[hpub:clear_TOSNO_timeDelay.png]]

The population also falls to about 50% when the geographical separation is reduced from 110 km to 100 km.

Mean Sample Spectral Brightness Temperature

These data include sunlit and eclipsed data, so the SW band is not included. The graphs should be self-explanatory.

#+CAPTION: Figure 4. #+ATTR_HTML: :width 500px [[hpub:CrIS_TOSNO_mean_bias_spectrum.png]]

#+CAPTION: Figure 5. #+ATTR_HTML: :width 500px [[hpub:IASI_TOSNO_mean_bias_spectrum.png]]

#+CAPTION: Figure 6. #+ATTR_HTML: :width 500px [[hpub:IASI2CrIS_TOSNO_mean_bias_spectrum.png]]

Correlation Maps: Full Set

These are the correlation structures of the bias error, as defined above, for the full data sample (only 6-sigma outliers were removed). In each case the native sensor noise is unchanged. The order of the three figures is the CrIS, the IASI and the IASI deconvolved to CrIS. In the third case the sample noise characteristics propogate through the deconvolution and no further modifcation is made. The fourth plot repeats the first two but displays them side-by-side for easier comparison.

#+CAPTION: Figure 7. #+ATTR_HTML: :width 500px [[hpub:CrIS_corr_LW.png]]

#+CAPTION: Figure 8. #+ATTR_HTML: :width 500px [[hpub:IASI_LW_corr.png]]

#+CAPTION: Figure 9. #+ATTR_HTML: :width 500px [[hpub:I2C_corr_LW.png]]

#+CAPTION: Figure 10. #+ATTR_HTML: :width 500px [[hpub:CrIS_IASI_LW_corr_sbs.png]]

Sub-setting by time delay

In order to check whether the time delay between the IASI and CrIS observations in the SNO set affects the results, the same corrlation maps are repeated on three subsets. Each subset is determined by the size of the time delay, being:
(i) <= 2.75 hrs (which has sample size 8410);
(ii) > 2.75 <= 3.1 hrs (with a sample size of 13984);
(iii) > 3.1 <= 3.3 hrs (with a sample size of 16902).

The correlation maps are presented in order. There appears to be no discernable difference. So the conclusion is - within the limits of the SNO samples - that the time delay between about 2 hrs and 3.3 hrs does not impact the inter-channel bias correlation.

#+CAPTION: Figure 13. #+ATTR_HTML: :width 500px [[hpub:i2c_crisWnoise_td1_LW_corr_maps_sbys.png]]

#+CAPTION: Figure 14. #+ATTR_HTML: :width 500px [[hpub:i2c_crisWnoise_td2_LW_corr_maps_sbys.png]]

#+CAPTION: Figure 15. #+ATTR_HTML: :width 500px [[hpub:i2c_crisWnoise_td3_LW_corr_maps_sbys.png]]

High latitude pseudo SNOs with ERA model

TBD

High latitude true SNOs with ECMWF model.

Description of the data.

These true SNOs are obtained from the clear subset RTP files for 2016 with ECMWF model fields. The separation criteria are 20 minutes and 20 km, and include the center 4 FORs. Original RTP data are at: =/asl/rtp/rtp_cris_ccast_lowres/clear/2016/cris_lr_ecmwf_d2016.rtp=
and
=asl/rtp/rtp_iasi1/clear/2016/iasi1_ecmwf_d2016.rtp_{1,2}= \

There are 1065 SNO pairs for the year, occuring mostly around May and June. They occur in a narow high-latitude band, as shown in the figure, only 24 of them occur in the southern hemisphere. The full data set are used, consiting 1065 SNO pairs in total. Only those samples greater than 6-sigma outliers are removed as determined by the obs-calc bias, resulting in 17 SNO pairs being rejected. All spectral observations and calculations are hamming apodized.

#+CAPTION: Distibution map #+ATTR_HTML: :width 500px [[hpub:2016Y_IC_SNO_distro_map_900wn_BT.png]]

#+CAPTION: Window BT bias vs sample separation #+ATTR_HTML: :width 500px [[hpub:2016Y_IC_SNO_900wn_Bias_vs_separation.png]]

#+CAPTION: Window BT bias histogram #+ATTR_HTML: :width 500px [[hpub:2016Y_IC_SNO_900wn_Bias_histogram.png]]

*** Results for non-subset sampling The following plots show the statistics of the full sample bias, with no subsetting, and no change to the native sensor noise levels.

#+CAPTION: Mean Obs-Calc Bias and Standard Deviation of full sample #+ATTR_HTML: :width 500px [[lpub:2016Y_IC_SNO_Obs_Calc_mean_bias_Std_spect.png]]

[[hpub:2016Y_IC_SNO_CrIS_I2C_pNoise_mean_std_spec_LW.png]]

#+CAPTION: Bias Correlation for IASI, native noise #+ATTR_HTML: :width 500px hpub:2016Y_IC_SNO_IASI_native_corr_LW_sub-null.png

#+CAPTION: Bias Correlation for CrIS, native noise #+ATTR_HTML: :width 500px hpub:2016Y_IC_SNO_CrIS_native_corr_LW_sub-null.png

#+CAPTION: Bias Correlation for IASI-to-CrIS, native IASI noise. #+ATTR_HTML: :width 500px hpub:2016Y_IC_SNO_I2C_native_corr_LW_sub-null.png

Conclusions

TBD <>