Extended Verification and Validation for Earth System Models

evv4esm is a Python-based toolkit for extended verification and validation of Earth system models (ESMs). Currently, it provides a number of tests to determine if modifications either to the code or parameters of an ESM alter its simulated climate.

Test descriptions

• Multi-variate Kolmogorov-Smirnov, MVK: Tests the null hypothesis that the reference (n) and modified (m) model Short Independent Simulation Ensembles (SISE) represent the same climate state, based on the equality of distribution of each variable's annual global average in the standard monthly model output between the two simulations.
• Time Step Convergence, TSC: Tests the null hypothesis that the convergence of the time stepping error for a set of key atmospheric variables is the same for a reference ensemble and a test ensemble. Both the reference and test ensemble are generated with a two-second time step, and for each variable the RMSD between each ensemble and a truth ensemble, generated with a one-second time step, is calculated. RMSD is calculated globally and over two domains, the land and the ocean. The land/ocean domains contain just the atmosphere points that are over land/ocean cells.
• Perturbation Growth, PGN: Tests the null hypothesis that the reference (n) and modified (m) model ensembles represent the same atmospheric state after each physics parameterization is applied within a single time-step using the two-sample (n and m) T-test for equal averages at a 95% confidence level. Ensembles are generated by repeating the simulation for many initial conditions, with each initial condition subject to multiple perturbations.

Kolmogorov-Smirnov test

Perturbation growth test

Time step convergence test