OSZICAR never updated during training a Force field

[exk301]

Hello, VASP exeprts!

I have been training a force field for a metal cluster. So far, I have ML_AB file with roughly around 2000 number of configurations, and a corresponding force field file ML_FFN.
I'm trying to contnue training a force field using VASP version 6.4.3 (with -Duse_shmem" precompiler option as suggested in VASP website to decrease memory consumption). However, the job is stuck at initializing machine learning.. and OSZICAR is never updated. In contrary, when i used a vasp without shmem option, the job runns perfectly fine (I used the same input files, the same machine, with the same number of cores).

I also tested running geometry optimization, production run with a currently existing Force field using VASP with shmem, then I see OSZICAR is updated. It looks like the problem (that the output is never updated) with shmem option appears only with training a force field.

Do you have any advice regarding this?

Thank you!!

[ferenc_karsai]

I wanted to reproduce your hangup, but you didn't send your ML_AB file. Please send that file plus the exact INCAR that you used there.

I also need to know on how many cores that happened. The best would be if you can also send the stdout and OUTCAR file from the hangup job.

[exk301]

Hi Dr. Ferenc Karsai,

Thank you for your response. Here Attachment has links to files. I had big sized files so I just created links you can easily access to.
The first link leads to directory where output is not updated when I started to continue training a force field with an existing force field.
The second link is where I tried to train a force field from scratch (i.e., without a pre-existing force field) and the OSZICAR is stuck during self consistency run.
Each direcotry has my job subimssion script, INCAR, OUTCAR, output files you can take a look at.

Please let me know if you need further information from me. Thank you for your help again in advance!!

[ferenc_karsai]

I have tried your calculations and cannot observe the problems you have. Both calculations run through without problem with and without shared memory on 128 cores. It looks like your mpi installation for shared memory has a problem. I used gnu compiler with 11.2 and openmpi 4.1.2. If possible please try with this toolchain. Also you can try both shared memory implementations:
MPI3: -Duse_shmem
SysV: -Duse_shmem -Dsysv

[exk301]

Hello Dr. Ferenc Karsai,

Thank you so much! I'll try your advice!
Since we do not have gcc 11.2, do you think would gcc 13.2 + openmpi 5.x would work?

[ferenc_karsai]

Later versions can sometimes introduce bugs, but it is rare.
So without trying out I can't say for sure which version works, but at least we haven't heard anything negative about later versions of the toolchains.

So please go ahead and use that toolchain.

[exk301]

Hi Dr. Ferenc Karsai,
Following your advice, now I see there's no problem updating an output !

Thanks a lot!!!