Radiance Anomaly Creation and Retrievals
Thu, Apr 16, 20201 Introduction
The goal is to start global retrievals of profile anomalies of some combination of clear and cloudy time-averaged scenes. There are several recent technical limitation we need to overcome. In addition, the details of how to create these anomalies are still somewhat uncertain, although the actual retrieval algorithm should be in pretty good shape.
2 Limitations
- We should have gained about 500 TB of disk space in March, but COVID-19 stopped the vendor from installation.
- UMBC's OIT hopes to install it themselves by June, but that will require approval of VP for Research
- Generation of a large CHIRP data set is underway now, and once that is done we can hopefully regain space under lustre.
- I want to use the official AIRS L1c (v672) we downloaded from the DAAC. We have to finish validation of that data set, which should only take a few more days. We can then possibly free up ~100 TB for the anomaly retrievals.
- We have been using ERA-Interim for a-priori clouds and for retrieval Jacobians. Unfortunately, ERA-Interim ceased production on Aug. 31, 2019. That is actually a nice date, since it coincides with AIRS operations for 17 years. I suggest we go forward with ERA-Interim for now since we would have to wait until Aug. anyway to get another year. In the meantime we are downloaded ERA5.
- We have not yet integrated in the UW land surface emissivity product. We should do that, the question is should we proceed assuming constant emissivity for now? The UW product is probably a year or two behind.
So, some combination of the above two bullets might make it possible to do a full-up AIRS L1c data transpose in the near future.
3 Approach
The basic retrieval step is a retrieval of time/space averaged radiances in a grid cell. Parameters to specify include:
- Cell size (3x5 deg lat/lon?)
- Time average: 16 days seems to work well
- Do full radiance averaging?
3.1 Full Radiance Averaging
Sergio and I found that doing zonal retrievals with full radiance averaging (all data in a 16-day window and a latitude bin) worked pretty well, but in tropical regions with persistent clouds our SST (we have not done land) we found that our SST retrieval trends were a bit off, not horrible, but clearly in error. Since zonal averaging really gives you quite strange radiance averages since you are including regions with few clouds with regions with persistent clouds.
We think that once we move to a grid cell this persistent cloud issue will be less problematic since when we mix grid clear and cloudy scenes. But, with a grid cell, we will be mixing them over time, not over space, which may help.
3.1.1 Raidance Averaging by Scene Type
Another approach is to take each 16 day (or maybe longer) time step and instead of averaging all the radiances, separate them into N bins (N really small to start) where we sort them by the B(T) of a window channel. Say we choose N=3 to start. The first bin might have highest 10 percentile (hottest scenes), the second bin the 11-90 percentile scenes, and the third that coldes 10% of scenes. Note that we will have average cloud amount (and type) estimates from ERA for each of these bins. So, in a region of persistent clouds, we still would expect to have significant clouds in bin one.
The retrievals would then use bin one first, using that to get the best estimates of surface temperature and atmospheric profiles for clearer scenes. The retrieved surface T, and profiles from this bin would then be used as the a-priori profiles and surface T for the cloudier bin, etc. Maybe too complicated to start with?
I did some tests where I took the hottest 10% of a 1% subset (1/100 of all AIRS data), and trended a surface channel for that effective 0.1% of all data, and got retrievals (after minor correction for variable water vapor), that agreed extremely well with ERA SST trends. (I only did ocean.)
3.2 Creation of Gridded Radiances
What I wanted to do is reformat (transpose) the whole AIRS L1c record, meaning create files that contain all AIRS data for a single gridpoint over time. (Maybe something like all AIRS data in grid cell for 16-days, and all channels, in one file.) That way we can work on each grid cell separately without re-opening the full AIRS L1c record. This will require at least ~200+TB. The advantage is that we can make finer grids later from the larger grid cells used for this transpose.
We might want to wait until we get the new disk space for this…
3.3 FIRST SHOT
Maybe the first thing to do is forget the AIRS L1c transpose, go thru the full record once, do full radiance averaging for a 3x5 lat/lon grid over 16 days, and start there. The sad part of this is that this will still require a very high cpu and I/O load since we really need to match to ERA for cloud a-priori and Jacobians.
But, this will minimize disk space usage and may actually be good enough!