AWI-CM3 run crashing after change in nproc

[fernandadialzira]

I am running pre-industrial simulations with annual restarts on JUWELS with AWI-CM3.

Before, I have set the transcript with fesom.nproc=48, oifs.nproc=48, and oifs.omp_num_threads=4, and it was resulting in long waiting time because I would only get 7 nodes allocated.

Simulation years: 2010-2100

So I started a new simulation with cold start for oifs, and restarting fesom from the previous simulation (year 2100). I changed fesom.nproc=432, oifs.nproc=360, and oifs.omp_num_threads = 6, which gives me 57 nodes.
The simulation runs for the first 2 years and on the 3rd year I get the following MPI_abort error.

(oasis_advance_map) ERROR: gsspos sumdst is zero but sumsrc is not
 (oasis_abort) ABORT: file      = mod_oasis_advance.F90
 (oasis_abort) ABORT: line      =         2035
 (oasis_abort) ABORT: on model  = OpenIFS
 (oasis_abort) ABORT: on global rank =          432
 (oasis_abort) ABORT: on local  rank =            0
 (oasis_abort) ABORT: CALLING ABORT FROM OASIS LAYER NOW

runscript: /p/project/chhb20/fmatos/runtime/awicm3-master/runscript_testing/pi_ghg_ece4.nc
log: /p/scratch/chhb20/fmatos/runtime/awicm3-master/experiments_testing/pi_ghg_ece4_005/run_21020101-21021231/log/pi_ghg_ece4_005_awicm3_compute_6625115.log
debug.root message: /p/scratch/chhb20/fmatos/runtime/awicm3-master/experiments_testing/pi_ghg_ece4_005/run_21020101-21021231/work.debug.root.02
Has anyone stumbled across such an error?

[JanStreffing]

Hey Fernanda, since the model is cancelled from the oasis layer, the relevant error message should be in the last edited debug.root.** file.

[JanStreffing]

Ah nvm, thats what you posted. I'll have a closer look

[JanStreffing]

You may have run into a rare OASIS error. GSSPOS build the global sign wise sums on the source and destination grids. The difference is multiplied onto the value of the destination grid in order to maintain global flux conservation. If the sum on the destination grid is 0 while the one on the source grid is not, you probably would run into divide by 0 somewhere, and instead you get this message + crash.

This happens in line 2034 of lib/psmile/src/mod_oasis_advance.F90

             ! compute correction and apply
             do m = 1,fsize
                if (av_sumd(m) == 0.0_ip_r8_p .and. av_sums(m) /= 0.0_ip_r8_p) then
                   WRITE(nulprt,*) subname,estr,'gsspos sumdst is zero but sumsrc is not'
                   call oasis_abort(file=__FILE__,line=__LINE__)
                else

If this was a common feature it might be grounds to worry, but the chances of hitting a 0.0000000000 with a real(8) are very low. I would suggest to replace the call oasis aboard with something like adding 0.0000000001 and taking else part out of the if logic.

Meanwhile you should probably buy a lottery ticket :D

[fernandadialzira]

Thank you! I will do it and come back with the result here to leave it documented :)

[JanStreffing]

When you do this, make sure you add some big fat warning to the std io. This should not happen frequently. In fact it should be exceedingly rare. If we see the message more often, we need to investigate further.

Also, I'm not sure I would trust that everything is properly dynamically linked. It should be, but it might still be saver to just recomp the whole model instead of just comp over oasis.

[fernandadialzira]

The message appeared actually 3 times when I was running for the year 2102. First I did the simulation and got the error.

What I do mainly when this happens and I don't know the error is to delete the run folder and restart. But I got the same.

Then started another simulation with the same settings, and got the same error for the same year. It wasn't very clear.

About the recomp: I am always also skeptical, so I never recomp only one component :D