Spinup-Evaluation

A Physics-Based Benchmarking Platform for ML-Based Spinup of the NEMO Ocean Model

Matt Archer

Isaac Akanho

Surbhi Goel

Simon Sadler

Etienne Meunier

Guillaume Gachon

Julie Deshayes

2026-02-24

Spinup-evaluation

Accelerating Spinup

• Ocean spinup is slow. Upwards of a 1000 simulation years to achieve equilibrium state!

• Many machine learning emulators exist for ocean modelling that are well validated:
  • How do we know if they are correct physically?
  • Errors can easily propagate during rollout.

• Our focus is on the physical evaluation of ocean models:
  • Including equation resolving, pure AI emulators and hybrid approaches.

• Implement sustainable software that is easy to extend, robust and reusable.
  • Generalised evaluation benchmark for climate.

• Temperature and salinity in deep ocean is slow to converge.
• Is RMSE sufficient? AI experts tend to use RMSE over some standard fields. Which means that we tend towards an averaged state over long roll out times
• I’ve emphasized hybrid as well here, because it’s likely that ML will not be be able to get you all the way there. Hy brid interplay between both techniques.
  • Tools to reshape data from emulators to climate model
• Generalised benchmark for ocean modelling
  • We focus on NEMO - this has been our first objective - the end goal is to generalise evaluation to lots of ocean models.

NEMO/DINO ocean model

/vopikamm/DINO

We use the DINO configuration (Kamm, Deshayes, and Madec 2025) for development purposes.

• not important to go into detail about the DINO test problem. I believe David is here presenting work on this so please do ask him questions on this - not me.
• It is important to note that although this has been our development
• maybe skip this slide

Is RMSE enough?

Unconstrained

Constrained

Reference

	Unconstrained	Constrained	Reference
\(T\) DW	2.9	2.6	2.6
\(\rho\) error	26.8	0.4	0.4

Evaluate even when resolution changes.

Meunier et al. (2025), Gorce et al. (2025)

• These are results from a diffusion model to sample from a equilibrated distribution. The justification is that there may not be just one valid spinup state.
• Horizontal surface plots
• Depth slices.
• I’m not going to describe the work - you can find it by following the link.
• Suffice it to say that in this case better diffusion outputs may lead to higher RMSE. This is because
• critically only one leads to a successful restart of NEMO even though the RMSE is relatively close in each case.
• the middle figure captures the high energy modes such as eddies
• it also caputres the stratification that is essential for nemo to run without a physical shock

Is RMSE enough?

Unconstrained

Constrained

Reference

	Unconstrained	Constrained	Reference
\(T\) DW	2.9	2.6	2.6
\(\rho\) error	26.8	0.4	0.4

Evaluate even when resolution changes.

Meunier et al. (2025), Gorce et al. (2025)

Physical evaluation

Metrics

Figure from Learning to generate physical ocean states. Meunier et al. (2025)

Bring in physical knowledge of problem

i.e. metrics specific to tracking equilibrium progress.

• No ground truth

• worth emphasizing that there is no ground truth - at best we can compare only against a pre-existing reference state.

Evaluation-Benchmark

/m2lines/nemo-spinup-forecast

AI emulator or traditional time integrator of physical quantities.

AI Emulator
/ Hybrid

Ocean
Model

Physical Evaluation

• Physical evaluation of instantaneous and temporal fields [GridT, restart].nc

Translator
Toolbox

• Create restart state from AI emulator
  
• Simple checks
• ‘Glue’ module for future time-stepping

• What does the evaluation-benchmark look like?
• We start with an equation resolving or AI based model
• We can go straight from the ocean model to physical evaluation.
• For the hybrid or AI model things we have a few options.
  • could go directly to physical evaluation but usually better to go through a restart / translator toolbox
  • this massage the outputs into a compatible format and does some simple checks.
    • We currently only support NEMO. But long term we’d like to extend this.
  • this ensures we’re assessing like for like between the ocean model and the ai emulator
• this is also the glue that prepares us for evaluating the state by plugging it back into the numerical model

Evaluation-Benchmark

• How do we evaluate hybrid ocean emulators?

• PILLAR 1:
  • Compute metrics at end time only.
  • Compute during roll-out

• PILLAR 2:
  • Feed emulated state back into ocean model to check stability and
    • evaluate metrics over time history

• PILLAR 3:
  • Sample from a distribution of possible emulated states

Pillars 1, 2 and 3 enabled with the use of two software components:

/m2lines/spinup-evaluation

version 0.2

• mention the correction of the model
• We have produced a piece of software that evaluates spin-up

Developer highlights

Spinup-evaluation (1)

• Computes metrics on raw emulator outputs (.npy), instantaneous (restart.nc) and time averaged (grid.nc) output files & statistical comparisons
• Strict mapping of ocean variables to restart, grid (output) and mesh_data names
• Provides a framework to easily add new metrics
• Implements temporal downsampling – all possible metrics computed even if data output at different cadences.

reference files

↓

restart.nc

→

Spinup
Evaluation

→

metrics.csv

output.nc

→

mmask.nc

→

raw.npy

→

/m2lines/spinup-evaluation

version 0.2

• Very much in development
• We want the software to be easily adaptable for new science cases actively working on this.
• Temporal downsampling - ensure that all metrics can be computed regardless of the temporal frequency of the data. I.e. if we have a metric that’s a function of salinity, SSH and temperature, but only ssh is output every month, we’ll still compute that metric.

Developer highlights

Spinup-restart/translator (2)

• Take an emulator state and converts to NEMO compatible restart.nc file
• Basic validation gate; does grid encoding and sanity checks (existence of NaNs etc)
• Regrid, upscale, downscale resolution;
• Encode other physical assumptions (geostrophic velocities)

reference files

↓

restart.nc

→

Spinup
Translator

→

Spinup
Evaluation

→

metrics.csv

output.nc

→

mmask.nc

→

raw.npy

→

Translate between other ocean model restart files - potentially difficult due to grids (in progress..)

• this is the module or ‘glue’ to enable physical timestepping
• to establish structural uncertainty between different ocean models.
• Often the case that we do our machine learning at a lower resolution because the compute demands are so high - and then upscale

Software engineering

Modular & Generalizable

• Reusable components across domains
• Extensible to new models (e.g., MOM6, Oceananigans)

Quality & Standards

• Robust testing (Unit, Integration, Coverage) with CI
• Well documented end-to-end examples (hybrid)
• Data products adhere to FAIR (see Wilkinson et al. 2016)
• Collaborative Design: Bridging Science & RSE

graph TD
    A(Scientific Expertise) --- C{RSE Bridge}
    B(Software Engineering) --- C
    C --> D[Sustainable Tools]
    
    classDef default fill:#1a365d,stroke:#fff,stroke-width:2px,color:#fff,font-size:18px;
    classDef highlight fill:#84BD00,stroke:#fff,stroke-width:2px,color:#fff,font-size:18px;
    classDef bridge fill:#003E51,stroke:#fff,stroke-width:2px,color:#fff,font-size:18px;

    class A,B default;
    class C bridge;
    class D highlight;
    
    linkStyle default stroke:#fff,stroke-width:2px;

Eliminating silos through cross-disciplinary partnership.

• I’ve mentioned spinup but the long term plan to to extend this to other domains.
• hopefully bug free. Working with netcdf files and xarray it’s surprising how easy it is for errors to creep in.
• We often see ‘Scientific Expertise’ on one side and ‘Software Engineering’ on the other, but without a formal bridge, the results are often fragile scripts that break when a postdoc leaves. By embedding RSEs into the design process, we aren’t just writing code; we are building Sustainable Tools

Conclusions

Key Takeaways

1. Extend evaluation of ocean models:
   • Evaluation of AI emulators must go beyond simple RMSE metrics.
   • Enable evaluation in conjunction with equation resolving techniques.

2. Software enabled research:
   • Research software engineering is a critical component of research bringing generalisability, extensibility and robustness to software products.

Next Steps

• Extend to other ocean models and AI emulators
• Translation between different ocean checkpoint files

• key assumption is that Evaluation should not proceed in isolation of the numerical approach it’s emulating.
• RSEs bring skills that scientists are unable to prioritise.
• Come and talk to me or message if there’s something you’d like us to prioritise or a science case that would be a good fit here.

Thanks for Listening

Get in touch:

 Matt Archer

 ma595.github.io

 ma595[AT]cam.ac.uk

 iccs[AT]cam.ac.uk

 ma595

ma595.github.io/spineval-agu - Slides

The ICCS received support from

/m2lines/spinup-evaluation

References

Gorce, Blandine, Luther Ollier, David Kamm, and Etienne Meunier. 2025. “Physically Consistent Sampling for Ocean Model Initialization.” In Advances in Neural Information Processing Systems. Vol. 38. https://neurips.cc/virtual/2025/loc/san-diego/poster/126936.

Kamm, D., J. Deshayes, and G. Madec. 2025. “DINO: A Diabatic Model of Pole-to-Pole Ocean Dynamics to Assess Subgrid Parameterizations Across Horizontal Scales.” EGUsphere 2025: 1–26. https://doi.org/10.5194/egusphere-2025-1100.

Meunier, Etienne, David Kamm, Guillaume Gachon, Redouane Lguensat, and Julie Deshayes. 2025. “Learning to Generate Physical Ocean States: Towards Hybrid Climate Modeling.” arXiv. https://doi.org/10.48550/arXiv.2502.02499.

Wilkinson, Mark D., Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, et al. 2016. “The FAIR Guiding Principles for Scientific Data Management and Stewardship.” Scientific Data 3 (1): 160018. https://doi.org/10.1038/sdata.2016.18.