Evaluation

Once training is done, we can perform evaluation depending on the use case. We recommend using GPUs for evaluation.

1. Autoregressive if we have an autoregressive model, we can simply run:

# Evaluating autoregressive model, e.g., 
# --model_name 'unet_s2s'
# --eval_years 2022
# --version_num 0       ## Checkpoint versions autogenerated in logs/
# --lra5 't2m' 'tp'     ## Additional LRA5 vars to be evaluated
# --oras5 'sosstsst'    ## Additional ORAS5 vars to be evaluated

$ python eval_iter.py --model_name <str> --eval_years <int> --version_num <int> --lra5 [...] --oras5 [...]

2. Direct if we have a collection of models trained specifically for unique lead_time, we can run:

# Evaluating direct model with the default sequence of
# lead_time = [1, 5, 10, 15, 20, 25, 30, 35, 40, 44] e.g., 
# --model_name 'unet_s2s'
# --eval_years 2022
# --version_nums 0 4 5 6 7 8 9 10 11 12 
# --lra5 't2m' 'tp'     ## Additional LRA5 vars to be evaluated
# --oras5 'sosstsst'    ## Additional ORAS5 vars to be evaluated

$ python eval_direct.py --model_name <str> --eval_years <int> --version_nums [...] --lra5 [...] --oras5 [...]

3. Ensemble if we have e.g., probabilistic model that generates ensemble forecasts and are supposed to be evaluated with additional probabilistic metrics, we can run:

# Evaluating ensembles with additional probabilistic metrics e.g., 
# --model_name 'vae_s2s'
# --eval_years 2022
# --version_num 0
# --lra5 't2m' 'tp'     ## Additional LRA5 vars to be evaluated
# --oras5 'sosstsst'    ## Additional ORAS5 vars to be evaluated

$ python eval_ensemble.py --model_name <str> --eval_years <int> --version_num <int> --lra5 [...] --oras5 [...]

Accessing Baseline Scores

You can access the complete scores (in .csv format) for climatology, persistence, physics-based (control + perturbed), and data-driven models here. Below is a snippet from logs/climatology/eval/rmse_climatology.csv, where each row represents e.g., <METRIC> == RMSE, at each future timestep.

z-10	z-50	z-100	z-200	z-300	…	w-1000
539.7944	285.9499	215.14742	186.43161	166.28784	…	0.07912156
538.9591	285.43832	214.82317	186.23743	166.16902	…	0.07907272
538.1366	284.96063	214.51791	186.04941	166.04732	…	0.07903882
…	…	…	…	…	…	…