6. Reformat MET Point Stat Data

6.1. Description

The METreformat module is used to reformat MET point_stat .stat files into a format that can be read by METplotpy. The reformatted data contains additional columns:

Two columns for statistics

  • stat_name

  • stat_value

Four columns for confidence levels (normal and bootstrap)

  • stat_bcl

    • upper level bootstrap confidence limit

  • stat_bcu

    • lower level bootstrap confidence limit

  • stat_ncl

    • lower level normal confidence limit

  • stat_ncu

    • upper level normal confidence limit

These columns are needed for the METplotpy plots that are used by METviewer (these plots can also be generated from the command line).

The following MET line types* can be reformatted:

  • FHO

  • CNT

  • CTC

  • CTS

  • SL1L2

Other line types will be supported in the future.

6.2. Required Components

Some METdbLoad modules are used to find and collect data from the individual .stat files into one data structure.

Two files are required:

  • an XML specification file (a simplified version of what is used by METdbLoad to read in the .stat files)

  • a YAML configuration file

The XML specification file is used to indicate the directory where the .stat file(s) are located and the type of MET files to read. The YAML configuration file is used to indicate the name and location of the output file, and the location of the XML specification file.

6.3. Example

  • set up a base directory, where the METdataio source code is located

export BASE_DIR=/path/to/METdataio

setenv BASE_DIR /path/to/METdataio
  • replace /path/to with an actual path

  • set up a working directory, where the XML and YAML config files will be located

export WORKING_DIR=/path/to/working_dir

setenv WORKING_DIR /path/to/working_dir
  • copy the point_stat.xml and point_stat.yaml files to the WORKING_DIR

cp $BASE_DIR/METdataio/METreformat/point_stat.xml $WORKING_DIR/
cp $BASE_DIR/METdataio/METreformat/point_stat.yaml $WORKING_DIR/
  • modify the XML specification file

    replace the content between the <folder_tmpl> </folder_tmpl> tags with the full path to the .stat files to be reformatted:

    for example:

  • replacing the /path/to with the directory where you saved the source code (NOTE: Do NOT use environment variables, you must specify the full path to the METdataio source code).

  • this example uses the sample data in the METdataio/METreformat/test/data/point_stat

  • the following code does not need to be modified, it is used by METdbLoad to determine what types of files to look for and process. Currently, only point_stat is supported, with support for the other MET tools to be added in the future.

        <field name="met_tool">
  • modify the point_stat.yaml file to specify the output directory, output filename, and location of the XML specification file:

output_dir: /path/to/output_dir

output_filename: point_stat_reformatted.txt

xml_spec_file: /path/to/xml_spec_file/<xml filename>.xml
  • For output_dir, replace /path/to with the full path to the outputdir

  • For xml_spec_file, replace /path/to with the full path to the XML spec file you edited in the step above and replace the <xml filename> with point_stat.

  • NOTE: Do NOT use environment variables for /path/to, specify the actual path.

  • set the PYTHONPATH:


  • Generate the reformatted file:

    • from the command line (from any directory):

python $BASE_DIR/METreformat/write_stat_ascii.py $WORKING_DIR/point_stat.yaml
  • A text file will be created in the output directory with the file name as specified in the yaml file.