6. Reformat MET Point Stat Data

6.1. Description

The METreformat module is used to reformat MET point_stat .stat files into a format that can be read by METplotpy. The reformatted data contains additional columns:

Two columns for statistics

stat_name
stat_value

Four columns for confidence levels (normal and bootstrap)

stat_bcl
- upper level bootstrap confidence limit
stat_bcu
- lower level bootstrap confidence limit
stat_ncl
- lower level normal confidence limit
stat_ncu
- upper level normal confidence limit

These columns are needed for the METplotpy plots that are used by METviewer (these plots can also be generated from the command line).

The following MET line types* can be reformatted:

FHO
CNT
CTC
CTS
SL1L2

Other line types will be supported in the future.

6.2. Required Components

Some METdbLoad modules are used to find and collect data from the individual .stat files into one data structure.

Two files are required:

an XML specification file (a simplified version of what is used by METdbLoad to read in the .stat files)
a YAML configuration file

The XML specification file is used to indicate the directory where the .stat file(s) are located and the type of MET files to read. The YAML configuration file is used to indicate the name and location of the output file, and the location of the XML specification file.

6.3. Example

set up a base directory, where the METdataio source code is located

bash:
export BASE_DIR=/path/to/METdataio

csh:
setenv BASE_DIR /path/to/METdataio

replace /path/to with an actual path
set up a working directory, where the XML and YAML config files will be located

bash:
export WORKING_DIR=/path/to/working_dir

csh:
setenv WORKING_DIR /path/to/working_dir

copy the point_stat.xml and point_stat.yaml files to the WORKING_DIR

cp $BASE_DIR/METdataio/METreformat/point_stat.xml $WORKING_DIR/
cp $BASE_DIR/METdataio/METreformat/point_stat.yaml $WORKING_DIR/

modify the XML specification file

replace the content between the <folder_tmpl> </folder_tmpl> tags with the full path to the .stat files to be reformatted:

for example:

<folder_tmpl>/path/to/METdataio/METreformat/test/data/point_stat</folder_tmpl>

replacing the /path/to with the directory where you saved the source code (NOTE: Do NOT use environment variables, you must specify the full path to the METdataio source code).
this example uses the sample data in the METdataio/METreformat/test/data/point_stat
the following code does not need to be modified, it is used by METdbLoad to determine what types of files to look for and process. Currently, only point_stat is supported, with support for the other MET tools to be added in the future.

<load_val>
        <field name="met_tool">
                <val>ensemble_stat</val>
                <val>grid_stat</val>
                <val>mode</val>
                <val>point_stat</val>
                <val>stat_analysis</val>
                <val>wavelet_stat</val>
        </field>
</load_val>

modify the point_stat.yaml file to specify the output directory, output filename, and location of the XML specification file:

output_dir: /path/to/output_dir

output_filename: point_stat_reformatted.txt

xml_spec_file: /path/to/xml_spec_file/<xml filename>.xml

For output_dir, replace /path/to with the full path to the outputdir
For xml_spec_file, replace /path/to with the full path to the XML spec file you edited in the step above and replace the <xml filename> with point_stat.
NOTE: Do NOT use environment variables for /path/to, specify the actual path.
set the PYTHONPATH:

bash
export PYTHONPATH=$BASE_DIR:/$BASE_DIR/METdbLoad:$BASE_DIR/METdbLoad/ush:$BASE_DIR/METreformat

csh
setenv PYTHONPATH $BASE_DIR:/$BASE_DIR/METdbLoad:$BASE_DIR/METdbLoad/ush:$BASE_DIR/METreformat

Generate the reformatted file:
- from the command line (from any directory):

python $BASE_DIR/METreformat/write_stat_ascii.py $WORKING_DIR/point_stat.yaml

A text file will be created in the output directory with the file name as specified in the yaml file.