Getting DAAC Data with OpenDap

Introduction

Opendap can be used to read any AIRS (and other instruments) file at the GSFC DAAC. We routinely use opendap to pull in CalFlag (detector problems) from AIRS L1b files for AIRS clear scene processing. We have to do this because JPL did not put CalFlag into the AIRXBCAL clear subset files, at least for Version 5, which is the only version presently available at the DAAC for AIRS Level 1 files. We may also be getting MERRA data via opendap as well.

Code History (Clear Scene RTP Generation)

Paul Schou has written routines to pull in CalFlag (and other variables) via opendap. This script, /asl/rtp_prod/airs/utils/get_meta_data pulls in a binary version of CalFlag which Breno saves at /asl/data/rtprod_airs/raw_meta_data. Paul used to convert these data to .mat files and saved them at /asl/data/airs/META_DATA. We probably have processing scripts that use what is at both locations. Breno called this script from /asl/rtp_rpod/airs/rtp/rtp_core_l1bcm.m. What a mess.

CalFlag is then accessed during rtp creation using /asl/matlib/opendap/getdata_opendap_file.m. This is a complicated code since it is reading binary data.

Proposed Changes to Clear Scene RTP Generation

The GSFC DAAC now supports netcdf output from opendap. This avoids complicated binary reads, and I want to switch to this approach. Until JPL adds CalFlag to AIRXBCAL, we will have to retrieve it from the L1b radiances files via opendap. (Note, we could get CalFlag from the AIRS L1b files that reside on our file-system at UMBC, but I don’t want to assume we always have all the AIRS L1b files.)

Overview of Opendap at the GSFC DAAC

To understand opendap at the DAAC open the URL opendap sample page at the DAAC. This URL is their http interface to a specific AIRS L1b granule. You can select various variables and subsets of those variables, and they generate the appropriate opendap URL at the top of the page. This shows you the generate way to generate an opendap query.

There are two useful methods for getting data via opendap, (1) wget and (2) ncread within Matlab. The following command shows you have to get CalFlag for this L1b file via wget. There is no subsetting, all a-tracks are download for all channels. (Note, calflag is only generated once per x-track/scanline.)

wget http://airsl1.gesdisc.eosdis.nasa.gov/opendap/hyrax/Aqua_AIRS_Level1/AIRIBRAD.005/2012/022/AIRS.2012.01.22.023.L1B.AIRS_Rad.v5.0.21.0.G13007155725.hdf.nc?CalFlag[0:1:134][0:1:2377] -O ourfile.nc

This command does the same thing, but within Matlab:

fn = 'http://airsl1.gesdisc.eosdis.nasa.gov/opendap/hyrax/Aqua_AIRS_Level1/AIRIBRAD.005/2012/022/AIRS.2012.01.22.023.L1B.AIRS_Rad.v5.0.21.0.G13007155725.hdf'
calflag = ncread(fn,'CalFlag');

For example, you can read all radiances in this file into Matlab using:

rads = ncread(fn,'radiances',[1,1,1],[Inf,Inf,Inf]);

This takes about 12 seconds, versus 1 second for a local L1b file (you can’t use ncread with a local file, you must use standard hdf Matlab commands). A final example reads in the detector noises for this file:

nen     = ncread(fn,'NeN');

Implementation

I think the best approach is to use a modified version of /asl/rtp_prod/airs/utils/get_meta_data to download CalFlag, etc. before RTP processing. I spoke to Paul, and he thinks the only change needed is to substitute file.dods? with file.nc? on line 64 of this m-file. Note above in the wget script the .nc? after the .hdf extension in the URL.

I guess curl is OK, the DAAC recommended wget, but it probably doesn’t matter.

Action Items

• What variables need to be downloaded besides CalFlag. Sergio probably wants dust_flag? Anything else?

• Steven should start coding and testing this. Basically this means changing /asl/rtp_prod/airs/utils/get_meta_data and modifying /asl/rtp_prod/airs/readers/readl1bcm_v5_rtp.m to read in calflag, etc. using netcdf (ncread).