Quick Synopsis of SDS Processing
The AVIRIS Science Data System, or SDS, is responsible for low-level processing (up to Level 1B for data collected 1993 to 2012 and up to Level 2 for data collected 2013 to present), data archiving, and data distribution, along with assisting the hardware team in judging the performance of the AVIRIS instrument and modeling instrument anomalies. The nominal processing chain is as follows:
Receipt of AVIRIS ship disk
- Performance evaluation
- Flight disk and run confirmation
- Download and decommutation of runs
- Performance Evaluation Program (PEP) execution
- Generation of single-band quicklook products and upload to AVIRIS website
- L1B and L2 Data Processing Pipeline
- Generate resampled calibrated data in units of spectral radiance
- Generate orthocorrected, scaled radiance data using full, three-dimensional ray tracing and a 30m spatial resolution digital elevation model (Level 1B)
- Generate RGB quicklook products
- Generate atmospherically-corrected surface reflectance data products (Level 2)
- Package L1B and L2 data products as gzipped tarballs
- Generate KML files and supporting Data Products Portal files
- Import L1B and L2 data products into AVIRIS Data Products Portal for Public accessibility
Detailed Description of SDS Processing
Upon receipt of an AVIRIS ship disk, the SDS processes the science runs to determine whether or not the instrument performed properly over the course of the flight. This performance evaluation (PEP) stage is mainly a subset of normal processing, but is performed on the science runs on disk before normal processing can commence.
AVIRIS processing is done on a per run basis.
Normal processing begins with downloading and decommutating the data, known as the download process. All data is stored in 16-bit integers, The image data is reversed (within each scan line), since the data coming off AVIRIS, if displayed directly, is actually reversed from how the data would look from the aircraft. Each line of data is expanded and reversed, with any bad data marked as such, and then written to SDS archive media (RAID array and tape library backup system).
The AVIRIS archiving process also compiles information about the image, navigation, and engineering data and stores it in the SDS database. This stored data is extracted at will with the Performance Evaulation Programs, or PEPs, which also plot the data to model instrument behavior graphically.
The AVIRIS quicklook images are also created during the archive process. These are initially stored grayscale JPEGs, and show band a single band of each flight run. Quicklooks can be viewed here.
L1B and L2 data products are made available via a private FTP site (or shipped via hard disk) for AVIRIS investigators to download upon request.
L1B data products are orthocorrected and radiometrically corrected which means converted into units of radiance, as opposed to the unitless AVIRIS digital numbers and each pixel in the imagery is individually ray traced using the best-estimate of sensor location and attitude until it intersects the digital elevation model. Radiance is measured in units of microwatts per square centimeter per nanometer per steradian, or uW / (cm^2 * nm * sr). AVIRIS radiometric calibration factors are calculated by measuring the response of AVIRIS to an integrating sphere (a known target illuminated by a known light source). This calibration is accurate to within 7%, absolute, over time. Intra-flight accuracy is within 2%. For more details regarding the orthocorrection process, please view the following document AVIRIS Orthocorrection Processing
L2 data products are atmospherically corrected surface reflectance data products. The atmospheric correction is based on the Atmosphere Removal Algorithm (ATREM) program developed by Gao. It retrieves scaled surface reflectance by modeling the absorption due to water vapor, carbon dioxide, ozone, nitrous oxide, carbon monoxide, methane, and oxygen. These absorption models are combined with observation geometry to estimate atmospheric transmittance from 400-2500 nanometers. Scattering effects are modeled with the 6S code. The algorithm retrieves the integrated water vapor amount on a per-pixel basis using 940 and 1140nm water absorption features, and then estimates the Lambertian surface reflectance.
The packaged L1B data products, L2 data products, RGB quicklooks, and KML files are made available via the AVIRIS Data Products Portal within 48-72 hours after collection.
The SDS Hardware:
- Data Processing Servers: Windows 2008 Server (305 TB Disk Space) and High-performance Beowulf Cluster (145 TB Disk Space)
- FTP Server : Windows 2008 Server
- Backup System : LTO Tape Library (712 TB Disk Space)
- Several media devices
The SDS software is as follows:
- Proprietary software written in Fortran, Python, IDL, and C programming languages.
- IDL (Interactive Display Language) from Exelis VIS
- ENVI (ENvironment for the Visualization of Images), also from Exelis VIS
Data Product Availability and Latency:
AVIRIS data management is governed by NASA's Earth Science Data Policy (adopted in 1991). NASA has developed policy implementation, practices, and nomenclature that Earth science missions use to comply with policy tenets.
The AVIRIS data latency requirements on delivering data products to the community is:
- Level 1 products (within 48 hours of acquisition)
- Level 2 products (within 72 hours of acquisition)