6. Reformat MET Point Stat Data
6.1. Description
The METreformat module is used to reformat MET point_stat .stat files into a format that can be read by METplotpy. The reformatted data contains additional columns:
Two columns for statistics
stat_name
stat_value
Four columns for confidence levels (normal and bootstrap)
stat_bcl
upper level bootstrap confidence limit
stat_bcu
lower level bootstrap confidence limit
stat_ncl
lower level normal confidence limit
stat_ncu
upper level normal confidence limit
These columns are needed for the METplotpy plots that are used by METviewer (these plots can also be generated from the command line).
The following MET line types* can be reformatted:
FHO
CNT
CTC
CTS
SL1L2
Other line types will be supported in the future.
6.2. Required Components
Some METdbLoad modules are used to find and collect data from the individual .stat files into one data structure.
Two files are required:
an XML specification file (a simplified version of what is used by METdbLoad to read in the .stat files)
a YAML configuration file
The XML specification file is used to indicate the directory where the .stat file(s) are located and the type of MET files to read. The YAML configuration file is used to indicate the name and location of the output file, and the location of the XML specification file.
6.3. Example
set up a base directory, where the METdataio source code is located
bash:
export BASE_DIR=/path/to/METdataio
csh:
setenv BASE_DIR /path/to/METdataio
replace /path/to with an actual path
set up a working directory, where the XML and YAML config files will be located
bash:
export WORKING_DIR=/path/to/working_dir
csh:
setenv WORKING_DIR /path/to/working_dir
copy the point_stat.xml and point_stat.yaml files to the WORKING_DIR
cp $BASE_DIR/METdataio/METreformat/point_stat.xml $WORKING_DIR/
cp $BASE_DIR/METdataio/METreformat/point_stat.yaml $WORKING_DIR/
modify the XML specification file
replace the content between the <folder_tmpl> </folder_tmpl> tags with the full path to the .stat files to be reformatted:
for example:
<folder_tmpl>/path/to/METdataio/METreformat/test/data/point_stat</folder_tmpl>
replacing the /path/to with the directory where you saved the source code (NOTE: Do NOT use environment variables, you must specify the full path to the METdataio source code).
this example uses the sample data in the METdataio/METreformat/test/data/point_stat
the following code does not need to be modified, it is used by METdbLoad to determine what types of files to look for and process. Currently, only point_stat is supported, with support for the other MET tools to be added in the future.
<load_val>
<field name="met_tool">
<val>ensemble_stat</val>
<val>grid_stat</val>
<val>mode</val>
<val>point_stat</val>
<val>stat_analysis</val>
<val>wavelet_stat</val>
</field>
</load_val>
modify the point_stat.yaml file to specify the output directory, output filename, and location of the XML specification file:
output_dir: /path/to/output_dir
output_filename: point_stat_reformatted.txt
xml_spec_file: /path/to/xml_spec_file/<xml filename>.xml
For output_dir, replace /path/to with the full path to the outputdir
For xml_spec_file, replace /path/to with the full path to the XML spec file you edited in the step above and replace the <xml filename> with point_stat.
NOTE: Do NOT use environment variables for /path/to, specify the actual path.
set the PYTHONPATH:
bash
export PYTHONPATH=$BASE_DIR:/$BASE_DIR/METdbLoad:$BASE_DIR/METdbLoad/ush:$BASE_DIR/METreformat
csh
setenv PYTHONPATH $BASE_DIR:/$BASE_DIR/METdbLoad:$BASE_DIR/METdbLoad/ush:$BASE_DIR/METreformat
Generate the reformatted file:
from the command line (from any directory):
python $BASE_DIR/METreformat/write_stat_ascii.py $WORKING_DIR/point_stat.yaml
A text file will be created in the output directory with the file name as specified in the yaml file.