l1indexing Package

Mon, Feb 1, 2016 Steven Buczkowski

Location

/asl/packages/l1indexing/

Overview

The general indexing process is the same across AIRS,CRiS, and IASI.

• Compile granule list
  • Simple list of paths to individual baseline data files
• Create index
  • For each granule, save subset of time, lat, long, etc sufficient to match arbitrary observation time, lat, long, etc into a granule and return name and location of the granule file for subsequent processing.
  • In general, index creation does not need to be run much more than once
• Find granules given obs (or list of obs)

AIRS

Creating index

Finding granules

CRiS

Compiling granule list

• run the script _cris_makeglist.sh. Output textfile glist${year}.txt with fully qualified file paths to each granule.

e.g. to rebuild all cris granule lists

$ for year in {2012..2015}; do
>     cris_make_glist.sh $year
> done

The resulting glist files are to be stored in /asl/data/cris/index.

Building indices

CRiS indices are built through the matlab function cris_make_index.m which takes the year to be run as it’s only parameter

function cris_make_index(year)

Finding granules

Granules can be matched to a list of observations through the matlab function cris_find_gfile.m

unction glist = cris_find_gfile(dset, t1, t2, opts)

See cris/demo1.m for an example search

IASI

Compiling granule list

• run the script _iasi_makeglist.sh. Output textfile glist${year}.txt with fully qualified file paths to each granule.

e.g. to rebuild all iasi granule lists

$ for year in {2007..2015}; do
>     iasi_make_glist.sh $year
> done

The resulting glist files are to be stored in /asl/data/IASI/index.

IASI glists will be large as there are, of order, 200000+ granules/year.

Creating index

Creating IASI indices is broken out and parallelized within a year to make runtime manageable: a series of sub-indices is created which are subsequently concatenated into a single index file. This process is scripted in sbatch to use a fixed 100 cpu array and the index concatenation is predicated on this setup. In this configuration, runtime is under 2 hours per array. Since it is usually fairly easy to grab 100 cpus, this means a year runs in under 2 hours (normal queue allows up to 256 cpus per user so, full queue utilization should run about 2.5 years at a shot and indexing the entire mission, to date, should run less than half a day of actual computing time).

$ year=2013
$ sbatch --array=0-99 run_iasi_make_index.sh $year

This script currently outputs sub-index files to a directory off my home directory and this should be changed if someone else needs to run it for any reason. A year of sub-index files only needs megabytes of space and can be concatenated into a single index with matlab function _iasi_catindex.m.

function [ngran, status] = iasi_cat_index(year)

iasi_cat_index produces a single matlab .mat file: index${year}.mat

Concatenated index files are to be stored in /asl/data/IASI/index

NOTE A small percentage of IASI granules are not processing through the IASI reader at the present time and they are excluded form the index. The current list of excluded granules is in the file _/asl/data/IASI/index/IASI_misinggranules

Finding granules

Granules are located by running the matlab script _iasi_findgfile.m

function glist = iasi_find_gfile(dset, t1, t2, opts)

See test routine demo1.m for example use.