LIVVkit

Usage

LIVVkit is intended to be integrated into a model’s development cycle. A public-private version of GitFlow is a common development cycle used by many scientific modeling groups. LIVVkit would be integrated into this development process like:

Development cycle

Schematic of the public-private development cycle used by many scientfic modeling groups [Kennedy2017]. A feature branch is created (d124, green path) from the current development branch (light purple path) and incremental changes are made (f001f003). These changes frequently undergo the standard set of LIVVkit verification tests (dashed cyan circles). Once the feature is complete and tested (f003), it is ready to be merged into develop (which includes the new features d125d126) and a pull request will be opened by the developer (dashed arrow). At this point, the integrators merge the feature into a local copy of the development branch and the full set of LIVVkit integration tests are started (filled cyan circle). Once passed, the feature is merged into develop (d127) and the pull request is closed. After enough features have been developed, the development branch will be merged into the private master branch (m120, right purple path), given a version tag (v1.2.0), and pushed to the mirrored master branch on the public repository (left purple path).

Standard verification and integration tests

The standard verification and integration tests are performed by comparing a regression test dataset, $TEST, against a reference dataset, $REF. The datasets follows a somewhat rigid leaf-node structure which allows the testing dataset to be more human-friendly than just a bunch of files and still fully describe the tests. This directory structure is, for example, created by CISM’s Build and Test Structure (BATS).

Model Output/LIVVkit Input Hierarchy

The directory layout of the model output should be as follows:

BUNDLE
    ├── [METADATA]
    └── TEST
        ├── VARIANT
        │    ├── sRESO*
        │    │   ├── pPROC
        │    │   │   ├── [zSIZE*]
        │    │   │   ├── ...
        ...

where [] indicates an optional directory, BUNDLE indicate the specific ice-sheet model used (including variant names; e.g., CISM_glissade), METADATA indicates any directories that contain information in addition to the testing data (e.g., job submission scripts, CMakeCache.txt), TEST indicates a particular type of test (e.g., shelf, dome, ismip-hom), VARIANT indicates any variant of that test (e.g., ismip-hom-a), sRESO indicates the grid resolution, pPROC indicate the number of processors used to run the test, and zSIZE indicates the domain size used in the test.

Note

The percise meaning of any *-ed directory name may be variable from test to test, but will generally follow the definition given here.

Standard verification analysis

The standard verification analysis typically compares a regression test dataset with a reference dataset. Because these datasets can be quickly generated and analyzed (\(\Delta t \lesssim\) a coffee break), they can (and should) be run frequently.

When a developer makes a new feature branch, they would first generate a reference dataset on their development machine.

For a CISM developer working on a Linux laptop/desktop, using the GNU compiler, and outputting the datasets to a reg_ref directory, this would look like:

cd $CISM
git checkout -b feature-branch

cd tests/regression/
./build_and_test.py -p linux-gnu -o reg_ref
export REF=$PWD/reg_ref/linux-gnu/CISM_glissade

When a change is made to the model and the developer is ready to test their code, they will then generate a test dataset on their development machine.

Again for a CISM developer working on a Linux laptop/desktop, using the GNU compiler, and outputting the datasets to a reg_test directory, this would look like:

# ... A change to CISM ...

cd $CISM/tests/regression/
./build_and_test.py -p linux-gnu -o reg_test
export TEST=$PWD/reg_test/linux-gnu/CISM_glissade

Then, the testing results can be compared to a reference dataset with LIVVkit:

cd $LIVV
./livv --verify $TEST $REF -o vv_test -s

LIVVkit will run the verification suite, report a summary of the results on the command line, produce an output website in the created vv_test directory, and launch an HTTP server to easily view the output website. LIVVkit will tell you the address to view the website at on the command line, which will typically look like http://0.0.0.0:8000/vv_test/index.html.

Once the feature is developed, and the developer is happy with the testing results, she/he would push the feature branch to the development repository and open a pull request, kicking off a integration analysis by the model integrator(s). Additionally, the output directory may compressed and uploaded to Github for viewing by the integration team (every output website is portable).

Integration analysis

Upon receiving a pull request for a new feature, a model’s integration team will typically initiate a more substantial test of the new feature, often on the target production machine(s).

First, the integrator would checkout the code base on the production machine, and generate a reference dataset for the current state of the development branch:

For a CISM integrator working on the supercomputer Titan at OLCF, using the GNU compiler, and outputting the datasets to a reg_ref directory, this would look like:

cd $CISM
git checkout develop

cd tests/regression/
./build_and_test.py -b ref_build -p titan-gnu --timing -o reg_ref
export REF=$PWD/reg_ref/linux-gnu/CISM_glissade

cd reg_ref/titan-gnu/CISM_glissade
./submit-all-jobs.bash

Note

For CISM, BATS recognizes a set of platforms that requires job submission scripts, automatically sets up the jobs, and creates a submission script. Also, by specifying the --timing option, a much larger set of tests are run, including repeat runs for performance variability.

Once those jobs are submitted, the integrator can attempt to merge in the feature branch. After all conflicts are resolved, a test dataset would be generated.

Again, for a CISM integrator working on the supercomputer Titan at OLCF, using the GNU compiler, and outputting the datasets to a reg_test directory, this would look like:

cd $CISM
git merge feature-branch

# ... resolve any conflicts ...

cd $CISM/tests/regression
./build_and_test.py -b test_build -p titan-gnu --timing -o reg_test
export TEST=$PWD/reg_test/linux-gnu/CISM_glissade

cd reg_test/titan-gnu/CISM_glissade
./submit-all-jobs.bash

Then once all the jobs were finished, the testing results can be compared to a reference dataset.

cd $LIVV
    ./livv --verify $TEST $REF -o vv_test -s

LIVVkit will run the verification suite, report a summary of the results on the command line, produce an output website in the created vv_test directory, and launch an HTTP server to easily view the output website. LIVVkit will tell you the address to view the website at on the command line, which will typically look like http://0.0.0.0:8000/vv_test/index.html.

Additionally, the output directory may compressed and uploaded to Github for viewing by the rest of the integration team and the feature developer (every output website is portable). If test results aren’t satisfactory, this provides a valuable resource for the developer to make the necessary changes to their feature.

Once testing results are satisfactory, the integration team may do a similar comparison to the latest release in order to track changes over a longer period of development and analyze the model for creep, or run a series of extended validation analyses.

Extended and/or validation analyses

Note

A set of standard ice sheet model validation analyses are currently being developed and will be released soon (along with the necessary observational data). Currently, there is a preliminary set of validation analyses developed for E3SM (Energy Exascale Earth System Model), CESM (Community Earth System Model), and CISM (Community Ice Sheet Model), which are contained in the LIVVkit Extensions (LEX) repository. See the LIVVkit Extensions (LEX) page for how to get and use LEX as well as how develop your own validation analysis.