gvar - GOES NEXT GVAR reception and processing

PROCESS HIERARCHY

gvarsched
 |  |  |
 |  |  +--- rdframe
 |  |       |  |  |
 |  |       |  |  |
 |  |       |  |  |
 |  |       |  |  +-- fsgvar.lod/gvarR.bit
 |  |       |  |
 |  |       |  +----- wrframe
 |  |       |
 |  |       +-------- monitrx
 |  |
 |  +--- gvar_sounder_proc
 |        |  |
 |        |  +--- scrub
 |        |
 |        +------ gvarin
 |
 +------ gvar_pproc/gvar_pproc_wait
          |  |
          |  +--- gvar_VISONLY_pproc
          |        |  |
          |        |  +--- diskhogs
          |        |
          |        +------ gvarin
          |                 |
          |                 +---gvar_VISONLY_subproc
          |                      |  |
          |                      |  +--- gsubset
          |                      |
          |                      +------ gvar_cleanup
          |
          +------ gvar_LOWRES_pproc
                   |  |
                   |  +--- diskhogs
                   |
                   +------ gvarin
                            |
                            +---gvar_LOWRES_subproc
                                 |  |  |
                                 |  |  +--- gsubset
                                 |  |
                                 |  +------ gvar_cleanup
                                 |
                                 +--------- assemble

PROCESS VARIATIONS

Replace gsubset and gvar_cleanup by gvsend and either diskhogs or scrub.

LOG FILES

$PASSDIR/schedlog                    - framing status
$PASSDIR/schedjob.[A-J]              - framing details
$PASSDIR/logfiles/ppd.[A-Z]          - imager/sounder process startup
$PASSDIR/logfiles/LOWRES.PPROC.?     - LOWRES ingest details
$PASSDIR/logfiles/LOWRES.SUBSET.?    - LOWRES subset details
$PASSDIR/logfiles/VISONLY.PPROC.?    - VISONLY ingest details
$PASSDIR/logfiles/VISONLY.SUBSET.?   - VISONLY subset details

PARAMETER FILES

$DATADIR/ExtNorthHem/RangePairs      - Extended Northern Hemisphere
$DATADIR/SouthSH/RangePairs          - South Southern Hemisphere
$DATADIR/FullDisk/RangePairs         - Full globe, outer space cropped
$DATADIR/NorthHem/RangePairs         - Northern Hemisphere
$DATADIR/ContUS_IR/RangePairs        - Continental United States

GVAR DATASETS

$DATADIR/LowRes/g?.*                 - IR res, all channels, all areas
$DATADIR/VisOnly/g?.*                - HI res, visible only, Cont US

$DATADIR/ExtNorthHem/g?.*            - LowRes subsets, all channels
$DATADIR/SouthSH/g?.*                - LowRes subsets, all channels
$DATADIR/FullDisk/g?.*               - LowRes subsets, all channels
$DATADIR/NorthHem/g?.*               - LowRes subsets, all channels
$DATADIR/ContUS_IR/g?.*              - LowRes subsets, no visible

$DATADIR/ContUS_VIS/g?.*             - VisOnly subsets
$DATADIR/ContUS/g?.*                 - ContUS_IR, ContUS_VIS assemblies

RAW DATA FILES

$PASSDIR/antenna/BAST - list of disk pass partitions, which can be
                        raw partitions, UNIX files, or /dev/null
$SOUNDER_FILE         - raw sounder data file, ring buffer with header

STATUS FILES

$PASSDIR/onlinecat    - list of imager passes received or being received
$PASSDIR/SHMEM_ACC    - application shared memory, status and raw telemetry

FUNCTIONS

gvarsched: This is started up in $PASSDIR/tscan.rc.local, which should be run during normal system bootup.; gvarsched initializes application shared memory, which is where reception status is maintained, and also where GVAR telemetry is spooled.; gvarsched makes sure that framing is running properly; if not, it stops the current framing process (rdframe), and starts a new one.; gvarsched starts the imager processing script gvar_pproc_wait whenever it starts rdframe. Also, whenever the end of an imager pass is detected by wrframe, gvarsched starts up gvar_pproc.; Whenever the end of a sounder pass is detected by wrframe, gvarsched starts up gvar_sounder_proc.; gvarsched writes its text output in the form of messages to $PASSDIR/schedlog.

rdframe: This is the telemetry framer. It runs until it gets stuck (no new frames received in two minutes), or until explicitly killed.; On SPARC machines, rdframe downloads two pieces of software onto the DSP card: fsgvar.lod and gvarR.bit. gvarR.bit frames the data; fsgvar.lod transfers the data to the SPARC.; rdframe also has two subprocesses, wrframe and monitrx, which it starts up almost immediately.; rdframe reads data from framing hardware and writes it to application shared memory. It updates a record counter every time it writes.; rdframe ignores uninteresting sounder records, as well as idle records.; rdframe stdout and stderr are directed to log files $PASSDIR/schedjob.? rdframe also logs messages in $PASSDIR/schedlog

monitrx: This process programs the receiver and bit sync hardware. It also monitors the receiver and bit sync as long as rdframe continues to run.; If the receiver and bit sync are preset to handle GVAR data, monitrx can be replaced by /bin/true.; monitrx stdout and stderr are directed to log files $PASSDIR/schedjob.? monitrx also logs messages in $PASSDIR/schedlog

wrframe: This process reads raw telemetry data from application shared memory and writes it to disk. Imager data is written to pass partitions defined in $PASSRSVP/BAST. Sounder data is written to the disk files defined by $SOUNDER_FILE.; Each imager pass is stored in a different pass partition. Several sounder passes are stored in the same $SOUNDER_FILE.; wrframe determines when imager and sounder passes end, and signals gvarsched with SIGUSR1 or SIGUSR2 respectively.; Also, wrframe updates the online catalog every time an imager pass completes and new one begins. Imager passes end with either there is a consistent change in a pass number embedded in the telemetry, or when there is a 30 second time gap after a sustained flow of imager frames.; wrframe stdout and stderr are directed to log files $PASSDIR/schedjob.? wrframe also logs messages in $PASSDIR/schedlog

gvar_sounder_proc: This process is started up by gvarsched every time wrframe detects an end of sounder pass. gvar_sounder_proc processes the last sounder pass received. gvar_sounder_proc reads from $SOUNDER_FILE which has a simple header that shows the layout of sounder passes.; Output datasets are written to $DATADIR/Sounder. Function text output is written to $DATADIR/Sounder/SOUNDER.LOG. The log file is truncated to 5000 lines at the end of each run.

gvar_pproc_wait/gvar_pproc: Every time gvarsched starts up rdframe, gvar_pproc_wait is started up at the same time. gvar_pproc_wait waits for reception to begin before attempting to read any GVAR data. gvar_pproc_wait is the same as gvar_pproc, except for this initial wait step.; gvar_pproc is the standard GVAR image data processor. It starts every time as end-of-pass is detected by wrframe.; gvar_pproc is very simple. It starts to processing scripts, gvar_LOWRES_pproc and gvar_VISONLY_pproc, and puts them in the background.; gvar_pproc and gvar_pproc_wait depend on $LASTPASS, the 1-relative number of the pass currently being received. This number is set by gvarsched.

gvar_VISONLY_pproc: This process ingests only GVAR visible data that intersects the continental US. It stores data in the directory $DATADIR/VisOnly.; If there is no intersection, no output dataset is generated.; Before and after the ingest step, gvar_VISONLY_pproc runs diskhogs, which removes the oldest file matching a given template until a disk usage limit is met.; gvar_VISONLY_pproc indirectly starts a co-process, gvar_VISONLY_subproc, whenever it opens an output dataset.; gvar_VISONLY_pproc depends on $LASTPASS to know which GVAR pass to read.; gvar_VISONLY_pproc text output to written to $PASSDIR/logfiles/VISONLY.PPROC.$LASTPASS . The log file is truncated to 1000 lines at the end of each run.

gvar_LOWRES_pproc: This process ingests all GVAR data at IR resolution (4,1,1,1,1), and clips it so there is a minimum of 'outer space' coverage. It stores data in the directory $DATADIR/LowRes.; Before and after the ingest step, gvar_LOWRES_pproc runs diskhogs, which removes the oldest file matching a given template until a disk usage limit is met.; gvar_LOWRES_pproc indirectly starts a co-process, gvar_LOWRES_subproc, whenever it opens an output dataset.; gvar_LOWRES_pproc depends on $LASTPASS to know which GVAR pass to read.; gvar_LOWRES_pproc text output to written to $PASSDIR/logfiles/LOWRES.PPROC.$LASTPASS . The log file is truncated to 1000 lines at the end of each run.

gvar_VISONLY_subproc dataset: This process generates subsets of the specified $DATADIR/VisOnly dataset in the directory $DATADIR/ContUS_VIS. There must be a file $DATADIR/ContUS_IR/RangePairs containing 4 numbers which are interpreted as 1-relative GVAR visible channel coordinates: start line, last line, start sample, and last sample. (Note, there is no file ContUS_VIS/RangePairs !); gvar_VISONLY_subproc text output is written to $PASSDIR/logfiles/VISONLY.SUBSET.$LASTPASS. This file is truncated to 500 lines after each run.

gvar_LOWRES_subproc dataset: This process generates subsets of the specified $DATADIR/LowRes dataset in any subdirectory of $DATADIR that includes the file RangePairs. This file must contain 4 numbers, which are interpreted as 1-relative GVAR visible channel coordinates: start line, last line, start sample, and last sample.; These subdirectories are ContUS_IR, ExtNorthHem, NorthHem, SouthSH, and FullDisk.; Subsets are generated only if the subarea is entirely included in the input dataset, e.g., an extended NH subset cannot be generated if the input covers only the continental US.; Except in the case of ContUS_IR, gvar_LOWRES_subproc includes all input variables in the output subsets.; In the case of ContUS_IR, gvar_LOWRES_subproc includes all non-visible channel variables.; gvar_LOWRES_subproc assembles the latest ContUS_IR and ContUS_VIS subsets and puts the assembly in $DATADIR/ContUS.; gvar_LOWRES_subproc text output is written to $PASSDIR/logfiles/LOWRES.SUBSET.$LASTPASS. This file is truncated to 500 lines after each run.

gvarin: gvarin is the GVAR imager and sounder ingestor. It reads GVAR imager data directly from application shared memory when reception is live; otherwise it reads imager data from pass partitions. gvarin reads sounder data from $SOUNDER_FILE.

gsubset: gsubsets creates a subset from a input dataset, based on earth transform coordinates. In the case of GVAR data, earth transform coordinates are absolute visible channel coordinates. If the area defined by the absolute visible channel coordinates in not completely covered by the input, no subset is generated.; Normally, subsets created by gsubset can be 'hollow' links that simply point back to the corresponding input datasets, or they can be instantiated. In this processing scenario, each subset must be a link, because it is created before the corresponding input dataset is completely full,

diskhogs: diskhogs removes the oldest files matching a given wildcard template until a given disk space limit is reached. Age can be based on dataset pass_date and start time attributes, or on file last modify time.

gvar_cleanup: gvar_cleanup removes link subsets and assemblies that have been orphaned, e.g., they are linked to datasets in $DATADIR/LowRes or VisOnly that no longer exist.

assemble: assemble combines variables from several input datasets to produce a single output dataset.; Normally, assemblies can be 'hollow' links that simply point back to the corresponding input datasets, or they can be instantiated. In this processing scenario, each assembly must be a link, because it is created before the corresponding input datasets are completely full,

gvsend: gvsend creates one or more instantiated subsets from an input dataset that is currently being filled by gvarin. It is intended that gvsend output be directed to remote NFS directories; otherwise it is better to use gsubset.
gvsend uses the same criteria as gsubset to define subset areas, namely absolute visible channel coordinates. If an area is not entirely covered by the input dataset, no subset can be generated.

scrub: scrub removes oldest files matching a given wildcard template based on age or count criteria.

NOTES

The following range_pairs are good choices for GVAR image subsets:

 2985  6495 9260 17730 - Continental US
 2815 10110 8180 22010 - Extended Northern Hemisphere
 2595 13390 5900 24810 - Full Disk
 2815 7790  8180 22015 - Northern Hemisphere
10045 12480 8180 22010 - South Southern Hemisphere

Last Update: $Date: 1999/05/10 21:16:31 $