### Starting TaskPrologue of job 237145 on a0531 at Fri May 13 19:50:56 CEST 2022 Running on cores 0-127 with govenor performance Fri May 13 19:50:56 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA A100-SXM... On | 00000000:0E:00.0 Off | 0 | | N/A 39C P0 83W / 400W | 0MiB / 81920MiB | 0% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 1 NVIDIA A100-SXM... On | 00000000:13:00.0 Off | 0 | | N/A 37C P0 85W / 400W | 0MiB / 81920MiB | 0% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 2 NVIDIA A100-SXM... On | 00000000:49:00.0 Off | 0 | | N/A 36C P0 86W / 400W | 0MiB / 81920MiB | 0% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 3 NVIDIA A100-SXM... On | 00000000:4F:00.0 Off | 0 | | N/A 42C P0 89W / 400W | 0MiB / 81920MiB | 0% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 4 NVIDIA A100-SXM... On | 00000000:90:00.0 Off | 0 | | N/A 42C P0 86W / 400W | 0MiB / 81920MiB | 0% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 5 NVIDIA A100-SXM... On | 00000000:96:00.0 Off | 0 | | N/A 36C P0 86W / 400W | 0MiB / 81920MiB | 0% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 6 NVIDIA A100-SXM... On | 00000000:CC:00.0 Off | 0 | | N/A 36C P0 81W / 400W | 0MiB / 81920MiB | 0% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ | 7 NVIDIA A100-SXM... On | 00000000:D1:00.0 Off | 0 | | N/A 39C P0 82W / 400W | 0MiB / 81920MiB | 0% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ ### Finished TaskPrologue + RUNTIME=3800 + '[' 0 -ge 1 ']' + ADDMOD=cuda + APP=/home/hpc/hpcg/mtec003h/hpcg/xhpcg_exp + module load hpcx/2.11 ++ /usr/bin/tclsh /apps/modules/modulecmd.tcl bash load hpcx/2.11 + eval 'CPATH=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/include:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/include:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/include:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/sharp/include:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll/include;' export 'CPATH;' 'LD_LIBRARY_PATH=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/nccl_rdma_sharp_plugin/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/sharp/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/lib/ucc:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/lib/ucx:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/lib;' export 'LD_LIBRARY_PATH;' 'OSHMEM_HOME=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi;' export 'OSHMEM_HOME;' 'PMIX_INSTALL_PREFIX=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi;' export 'PMIX_INSTALL_PREFIX;' 'MANPATH=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/share/man::;' export 'MANPATH;' 'LIBRARY_PATH=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/nccl_rdma_sharp_plugin/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/sharp/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/lib;' export 'LIBRARY_PATH;' '_LMFILES_=/apps/modules/data/development/hpcx/2.11;' export '_LMFILES_;' 'LOADEDMODULES=hpcx/2.11;' export 'LOADEDMODULES;' 'SHMEM_HOME=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi;' export 'SHMEM_HOME;' 'HPCX_CLUSTERKIT_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/clusterkit;' export 'HPCX_CLUSTERKIT_DIR;' 'PKG_CONFIG_PATH=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/lib/pkgconfig:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/lib/pkgconfig:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/sharp/lib/pkgconfig:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll/lib/pkgconfig;' export 'PKG_CONFIG_PATH;' 'HPCX_HOME=/apps/hpcx/2.11-gcc-inbox-gdrcopy2;' export 'HPCX_HOME;' 'HPCX_UCC_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc;' export 'HPCX_UCC_DIR;' 'OMPI_HOME=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi;' export 'OMPI_HOME;' 'MPI_HOME=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi;' export 'MPI_HOME;' 'HPCX_NCCL_RDMA_SHARP_PLUGIN_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/nccl_rdma_sharp_plugin;' export 'HPCX_NCCL_RDMA_SHARP_PLUGIN_DIR;' 'HPCX_MPI_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi;' export 'HPCX_MPI_DIR;' 'HPCX_IPM_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/tests/ipm-2.0.6;' export 'HPCX_IPM_DIR;' 'HPCX_MPI_TESTS_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/tests;' export 'HPCX_MPI_TESTS_DIR;' 'HPCX_UCX_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx;' export 'HPCX_UCX_DIR;' '__MODULES_LMCONFLICT=hpcx/2.11\&hpcx\&openmpi\&intelmpi\&ucx\&gcc;' export '__MODULES_LMCONFLICT;' 'OPAL_PREFIX=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi;' export 'OPAL_PREFIX;' '__MODULES_SHARE_MANPATH=:1;' export '__MODULES_SHARE_MANPATH;' 'HPCX_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2;' export 'HPCX_DIR;' 'HPCX_OSHMEM_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi;' export 'HPCX_OSHMEM_DIR;' 'HPCX_HCOLL_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll;' export 'HPCX_HCOLL_DIR;' 'HPCX_OSU_CUDA_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/tests/osu-micro-benchmarks-5.8-cuda;' export 'HPCX_OSU_CUDA_DIR;' 'PATH=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/clusterkit/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/tests/imb:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin;' export 'PATH;' 'HPCX_SHARP_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/sharp;' export 'HPCX_SHARP_DIR;' 'HPCX_IPM_LIB=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/tests/ipm-2.0.6/lib/libipm.so;' export 'HPCX_IPM_LIB;' 'HPCX_OSU_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/tests/osu-micro-benchmarks-5.8;' export 'HPCX_OSU_DIR;' test '0;' ++ CPATH=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/include:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/include:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/include:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/sharp/include:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll/include ++ export CPATH ++ LD_LIBRARY_PATH=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/nccl_rdma_sharp_plugin/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/sharp/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/lib/ucc:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/lib/ucx:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/lib ++ export LD_LIBRARY_PATH ++ OSHMEM_HOME=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi ++ export OSHMEM_HOME ++ PMIX_INSTALL_PREFIX=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi ++ export PMIX_INSTALL_PREFIX ++ MANPATH=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/share/man:: ++ export MANPATH ++ LIBRARY_PATH=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/nccl_rdma_sharp_plugin/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/sharp/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/lib ++ export LIBRARY_PATH ++ _LMFILES_=/apps/modules/data/development/hpcx/2.11 ++ export _LMFILES_ ++ LOADEDMODULES=hpcx/2.11 ++ export LOADEDMODULES ++ SHMEM_HOME=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi ++ export SHMEM_HOME ++ HPCX_CLUSTERKIT_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/clusterkit ++ export HPCX_CLUSTERKIT_DIR ++ PKG_CONFIG_PATH=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/lib/pkgconfig:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/lib/pkgconfig:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/sharp/lib/pkgconfig:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll/lib/pkgconfig ++ export PKG_CONFIG_PATH ++ HPCX_HOME=/apps/hpcx/2.11-gcc-inbox-gdrcopy2 ++ export HPCX_HOME ++ HPCX_UCC_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc ++ export HPCX_UCC_DIR ++ OMPI_HOME=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi ++ export OMPI_HOME ++ MPI_HOME=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi ++ export MPI_HOME ++ HPCX_NCCL_RDMA_SHARP_PLUGIN_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/nccl_rdma_sharp_plugin ++ export HPCX_NCCL_RDMA_SHARP_PLUGIN_DIR ++ HPCX_MPI_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi ++ export HPCX_MPI_DIR ++ HPCX_IPM_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/tests/ipm-2.0.6 ++ export HPCX_IPM_DIR ++ HPCX_MPI_TESTS_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/tests ++ export HPCX_MPI_TESTS_DIR ++ HPCX_UCX_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx ++ export HPCX_UCX_DIR ++ __MODULES_LMCONFLICT='hpcx/2.11&hpcx&openmpi&intelmpi&ucx&gcc' ++ export __MODULES_LMCONFLICT ++ OPAL_PREFIX=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi ++ export OPAL_PREFIX ++ __MODULES_SHARE_MANPATH=:1 ++ export __MODULES_SHARE_MANPATH ++ HPCX_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2 ++ export HPCX_DIR ++ HPCX_OSHMEM_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi ++ export HPCX_OSHMEM_DIR ++ HPCX_HCOLL_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll ++ export HPCX_HCOLL_DIR ++ HPCX_OSU_CUDA_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/tests/osu-micro-benchmarks-5.8-cuda ++ export HPCX_OSU_CUDA_DIR ++ PATH=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/clusterkit/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/tests/imb:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin ++ export PATH ++ HPCX_SHARP_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/sharp ++ export HPCX_SHARP_DIR ++ HPCX_IPM_LIB=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/tests/ipm-2.0.6/lib/libipm.so ++ export HPCX_IPM_LIB ++ HPCX_OSU_DIR=/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/tests/osu-micro-benchmarks-5.8 ++ export HPCX_OSU_DIR ++ test 0 + _mlstatus=0 + return 0 + module load cuda ++ /usr/bin/tclsh /apps/modules/modulecmd.tcl bash load cuda + eval 'CUDA_ROOT=/apps/SPACK/0.17.1/opt/linux-almalinux8-zen/gcc-8.4.1/cuda-11.6.1-hfhdb3z2cobwlalqrf5ist25hlklipuu;' export 'CUDA_ROOT;' 'LD_LIBRARY_PATH=/apps/SPACK/0.17.1/opt/linux-almalinux8-zen/gcc-8.4.1/cuda-11.6.1-hfhdb3z2cobwlalqrf5ist25hlklipuu/lib64:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/nccl_rdma_sharp_plugin/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/sharp/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/lib/ucc:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/lib/ucx:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/lib;' export 'LD_LIBRARY_PATH;' '__MODULES_LMALTNAME=cuda/11.6.1\&as\|cuda/default\&as\|cuda/latest;' export '__MODULES_LMALTNAME;' '__MODULES_LMCONFLICT=hpcx/2.11\&hpcx\&openmpi\&intelmpi\&ucx\&gcc:cuda/11.6.1\&cuda;' export '__MODULES_LMCONFLICT;' 'CUDA_HOME=/apps/SPACK/0.17.1/opt/linux-almalinux8-zen/gcc-8.4.1/cuda-11.6.1-hfhdb3z2cobwlalqrf5ist25hlklipuu;' export 'CUDA_HOME;' 'PATH=/apps/SPACK/0.17.1/opt/linux-almalinux8-zen/gcc-8.4.1/cuda-11.6.1-hfhdb3z2cobwlalqrf5ist25hlklipuu/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/clusterkit/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/tests/imb:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin;' export 'PATH;' '_LMFILES_=/apps/modules/data/development/hpcx/2.11:/apps/modules/data/development/cuda/11.6.1;' export '_LMFILES_;' 'LOADEDMODULES=hpcx/2.11:cuda/11.6.1;' export 'LOADEDMODULES;' 'CMAKE_PREFIX_PATH=/apps/SPACK/0.17.1/opt/linux-almalinux8-zen/gcc-8.4.1/cuda-11.6.1-hfhdb3z2cobwlalqrf5ist25hlklipuu/;' export 'CMAKE_PREFIX_PATH;' test '0;' ++ CUDA_ROOT=/apps/SPACK/0.17.1/opt/linux-almalinux8-zen/gcc-8.4.1/cuda-11.6.1-hfhdb3z2cobwlalqrf5ist25hlklipuu ++ export CUDA_ROOT ++ LD_LIBRARY_PATH=/apps/SPACK/0.17.1/opt/linux-almalinux8-zen/gcc-8.4.1/cuda-11.6.1-hfhdb3z2cobwlalqrf5ist25hlklipuu/lib64:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/nccl_rdma_sharp_plugin/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/sharp/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/lib/ucc:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/lib/ucx:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/lib ++ export LD_LIBRARY_PATH ++ __MODULES_LMALTNAME='cuda/11.6.1&as|cuda/default&as|cuda/latest' ++ export __MODULES_LMALTNAME ++ __MODULES_LMCONFLICT='hpcx/2.11&hpcx&openmpi&intelmpi&ucx&gcc:cuda/11.6.1&cuda' ++ export __MODULES_LMCONFLICT ++ CUDA_HOME=/apps/SPACK/0.17.1/opt/linux-almalinux8-zen/gcc-8.4.1/cuda-11.6.1-hfhdb3z2cobwlalqrf5ist25hlklipuu ++ export CUDA_HOME ++ PATH=/apps/SPACK/0.17.1/opt/linux-almalinux8-zen/gcc-8.4.1/cuda-11.6.1-hfhdb3z2cobwlalqrf5ist25hlklipuu/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/clusterkit/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/tests/imb:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin ++ export PATH ++ _LMFILES_=/apps/modules/data/development/hpcx/2.11:/apps/modules/data/development/cuda/11.6.1 ++ export _LMFILES_ ++ LOADEDMODULES=hpcx/2.11:cuda/11.6.1 ++ export LOADEDMODULES ++ CMAKE_PREFIX_PATH=/apps/SPACK/0.17.1/opt/linux-almalinux8-zen/gcc-8.4.1/cuda-11.6.1-hfhdb3z2cobwlalqrf5ist25hlklipuu/ ++ export CMAKE_PREFIX_PATH ++ test 0 + _mlstatus=0 + return 0 + '[' -z 8 ']' + export LD_LIBRARY_PATH=/usr/mpi/gcc/openmpi-4.1.4rc1/lib64:/apps/SPACK/0.17.1/opt/linux-almalinux8-zen/gcc-8.4.1/cuda-11.6.1-hfhdb3z2cobwlalqrf5ist25hlklipuu/lib64:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/nccl_rdma_sharp_plugin/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/sharp/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/lib/ucc:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/lib/ucx:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/lib + LD_LIBRARY_PATH=/usr/mpi/gcc/openmpi-4.1.4rc1/lib64:/apps/SPACK/0.17.1/opt/linux-almalinux8-zen/gcc-8.4.1/cuda-11.6.1-hfhdb3z2cobwlalqrf5ist25hlklipuu/lib64:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/nccl_rdma_sharp_plugin/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/sharp/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/lib/ucc:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/lib:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/lib/ucx:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/lib + export PATH=/usr/mpi/gcc/openmpi-4.1.4rc1/bin:/apps/SPACK/0.17.1/opt/linux-almalinux8-zen/gcc-8.4.1/cuda-11.6.1-hfhdb3z2cobwlalqrf5ist25hlklipuu/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/clusterkit/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/tests/imb:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin + PATH=/usr/mpi/gcc/openmpi-4.1.4rc1/bin:/apps/SPACK/0.17.1/opt/linux-almalinux8-zen/gcc-8.4.1/cuda-11.6.1-hfhdb3z2cobwlalqrf5ist25hlklipuu/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/clusterkit/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/tests/imb:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/hcoll/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucc/bin:/apps/hpcx/2.11-gcc-inbox-gdrcopy2/ucx/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin ++ cat /proc/cpuinfo ++ awk 'BEGIN{cpus=0}/processor/{cpus++}/cpu cores/{cpu_cores=$4}/siblings/{siblings=$3}END{print cpus*cpu_cores/siblings}' + CORES=128 + HOSTS=32 ++ date +%y-%m-%d_%H%M + DATE=22-05-13_1950 ++ /usr/sbin/ibstat ++ grep Rate ++ cut -d ' ' -f2 + MYIBRATE='200 25 40 200' ++ ibv_devinfo ++ grep hca_id ++ cut -f2 ++ head -n1 + MYIBHCA=mlx5_0 ++ cat /proc/cpuinfo ++ grep CPU ++ sort ++ uniq ++ cut -d ' ' -f 6,7 ++ tr -d ' ' ++ tr -d '\n' + MYCPU= ++ lscpu ++ grep '^CPU(s)' ++ tr -s ' ' ++ cut -d ' ' -f 2 ++ head -n1 + MYCORES=128 ++ numactl -H ++ grep available ++ cut -d ' ' -f 2 ++ head -n1 + MYNUMANODES=8 ++ free -g ++ grep Mem ++ awk '-F ' '{print $2}' + MYMEM=2015 ++ nvidia-smi -q ++ grep 'Product Name' ++ cut -d : -f 2 ++ tr -d ' ' ++ head -n1 + MYNVGPU=NVIDIAA100-SXM4-80GB ++ nvidia-smi -q ++ grep 'Attached GPU' ++ cut -d : -f 2 ++ tr -d ' ' + MYNUMGPUS=8 ++ nvidia-smi -q ++ grep Total ++ cut -d : -f 2 ++ head -n 1 ++ cut -d ' ' -f 2 + MYGPUMEM=81920 ++ nvidia-smi -q --display=CLOCK ++ grep -A 1 '\s\sApplications' ++ grep MHz ++ awk '{print $3}' ++ head -n1 + MYNVGPUMHZ=1155 ++ nvidia-smi -q --display=CLOCK ++ grep -A 2 '\s\sApplications' ++ tail -n 1 ++ grep MHz ++ awk '{print $3}' + MYNVGPUMEMMHZ=1593 + RUNDIR=/home/hpc/hpcg/mtec003h/hpcg/run/NVIDIAA100-SXM4-80GB/8GPUs--128CORES-8NUMAN-2015MEM/32N-256NP-237145.22-05-13_1950/ + mkdir -p /home/hpc/hpcg/mtec003h/hpcg/run/NVIDIAA100-SXM4-80GB/8GPUs--128CORES-8NUMAN-2015MEM/32N-256NP-237145.22-05-13_1950/ + cd /home/hpc/hpcg/mtec003h/hpcg/run/NVIDIAA100-SXM4-80GB/8GPUs--128CORES-8NUMAN-2015MEM/32N-256NP-237145.22-05-13_1950/ + HPCGLOG=HPCG-CUDA.a0531.nhr.fau.de.8xNVIDIAA100-SXM4-80GB-1155-1593.237145.22-05-13_1950.log + cat + cat + chmod a+x ./run_hpcg + FLAGSSINGLENODE='-mca pml ^ucx --mca osc ^ucx,rdma --mca btl self,vader --mca coll_hcoll_enable 0' + FLAGS='--mca pml ucx --mca osc ucx --mca spml ucx --mca btl ^self,vader,openib --mca coll_hcoll_enable 1 -x UCX_TLS=self,sm,shm,cuda,rc_x' + tee HPCG-CUDA.a0531.nhr.fau.de.8xNVIDIAA100-SXM4-80GB-1155-1593.237145.22-05-13_1950.log ++ which mpirun ++ alias ++ eval declare -f ++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot mpirun +++ declare -f + /apps/hpcx/2.11-gcc-inbox-gdrcopy2/ompi/bin/mpirun --mca pml ucx --mca osc ucx --mca spml ucx --mca btl '^self,vader,openib' --mca coll_hcoll_enable 1 -x UCX_TLS=self,sm,shm,cuda,rc_x ./run_hpcg All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. All done. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:0E:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:13:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:49:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:4F:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:90:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:96:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:CC:00.0. Treating as warning and moving on. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. Setting locked Memory clocks is not supported for GPU 00000000:D1:00.0. Treating as warning and moving on. All done. monitor_gpu from environment variable 1 gpu_temp_warning from environment variable 80 gpu_power_warning from environment variable 500 start of application (8 OMP threads)... 2022-05-13 19:51:28.565 Problem setup... Generic memory compression support is available Setup time: 0.554439 sec GPU: 'NVIDIA A100-SXM4-80GB' Memory use: 37029 MB / 81069 MB 8x8x4 process grid 512x512x288 local domain Reference SpMV+MG... Reference CG... Initial Residual: 6.518502e+04 Max_err: 1.000000e+00 tot_err: 1.390229e+05 REF Iter = 1 Scaled Residual: 1.922365e-01 Max error: 1.000000e+00 tot_error: 9.964809e-01 REF Iter = 2 Scaled Residual: 1.064445e-01 Max error: 1.000000e+00 tot_error: 9.932350e-01 REF Iter = 3 Scaled Residual: 7.364236e-02 Max error: 1.000000e+00 tot_error: 9.900251e-01 REF Iter = 4 Scaled Residual: 5.629806e-02 Max error: 1.000000e+00 tot_error: 9.868230e-01 REF Iter = 5 Scaled Residual: 4.551838e-02 Max error: 1.000000e+00 tot_error: 9.836207e-01 REF Iter = 6 Scaled Residual: 3.818126e-02 Max error: 1.000000e+00 tot_error: 9.804175e-01 REF Iter = 7 Scaled Residual: 3.288782e-02 Max error: 1.000000e+00 tot_error: 9.772131e-01 REF Iter = 8 Scaled Residual: 2.891680e-02 Max error: 1.000000e+00 tot_error: 9.740087e-01 REF Iter = 9 Scaled Residual: 2.584581e-02 Max error: 1.000000e+00 tot_error: 9.708051e-01 REF Iter = 10 Scaled Residual: 2.340020e-02 Max error: 1.000000e+00 tot_error: 9.676033e-01 REF Iter = 11 Scaled Residual: 2.139298e-02 Max error: 1.000000e+00 tot_error: 9.644032e-01 REF Iter = 12 Scaled Residual: 1.969574e-02 Max error: 1.000000e+00 tot_error: 9.612040e-01 REF Iter = 13 Scaled Residual: 1.822278e-02 Max error: 1.000000e+00 tot_error: 9.580045e-01 REF Iter = 14 Scaled Residual: 1.692741e-02 Max error: 1.000000e+00 tot_error: 9.548034e-01 REF Iter = 15 Scaled Residual: 1.578716e-02 Max error: 1.000000e+00 tot_error: 9.516003e-01 REF Iter = 16 Scaled Residual: 1.478867e-02 Max error: 1.000000e+00 tot_error: 9.483955e-01 REF Iter = 17 Scaled Residual: 1.392181e-02 Max error: 1.000000e+00 tot_error: 9.451900e-01 REF Iter = 18 Scaled Residual: 1.317247e-02 Max error: 1.000000e+00 tot_error: 9.419850e-01 REF Iter = 19 Scaled Residual: 1.252006e-02 Max error: 1.000000e+00 tot_error: 9.387817e-01 REF Iter = 20 Scaled Residual: 1.194201e-02 Max error: 1.000000e+00 tot_error: 9.355803e-01 REF Iter = 21 Scaled Residual: 1.141566e-02 Max error: 1.000000e+00 tot_error: 9.323807e-01 REF Iter = 22 Scaled Residual: 1.092320e-02 Max error: 1.000000e+00 tot_error: 9.291816e-01 REF Iter = 23 Scaled Residual: 1.045854e-02 Max error: 1.000000e+00 tot_error: 9.259818e-01 REF Iter = 24 Scaled Residual: 1.002485e-02 Max error: 1.000000e+00 tot_error: 9.227806e-01 REF Iter = 25 Scaled Residual: 9.629360e-03 Max error: 1.000000e+00 tot_error: 9.195781e-01 REF Iter = 26 Scaled Residual: 9.279960e-03 Max error: 1.000000e+00 tot_error: 9.163752e-01 REF Iter = 27 Scaled Residual: 8.978868e-03 Max error: 1.000000e+00 tot_error: 9.131737e-01 REF Iter = 28 Scaled Residual: 8.719987e-03 Max error: 1.000000e+00 tot_error: 9.099751e-01 REF Iter = 29 Scaled Residual: 8.490840e-03 Max error: 1.000000e+00 tot_error: 9.067803e-01 REF Iter = 30 Scaled Residual: 8.273829e-03 Max error: 1.000000e+00 tot_error: 9.035892e-01 REF Iter = 31 Scaled Residual: 8.053669e-03 Max error: 1.000000e+00 tot_error: 9.004007e-01 REF Iter = 32 Scaled Residual: 7.825930e-03 Max error: 1.000000e+00 tot_error: 8.972129e-01 REF Iter = 33 Scaled Residual: 7.595819e-03 Max error: 1.000000e+00 tot_error: 8.940252e-01 REF Iter = 34 Scaled Residual: 7.373390e-03 Max error: 1.000000e+00 tot_error: 8.908370e-01 REF Iter = 35 Scaled Residual: 7.166630e-03 Max error: 1.000000e+00 tot_error: 8.876495e-01 REF Iter = 36 Scaled Residual: 6.974439e-03 Max error: 1.000000e+00 tot_error: 8.844635e-01 REF Iter = 37 Scaled Residual: 6.788865e-03 Max error: 1.000000e+00 tot_error: 8.812789e-01 REF Iter = 38 Scaled Residual: 6.601562e-03 Max error: 1.000000e+00 tot_error: 8.780945e-01 REF Iter = 39 Scaled Residual: 6.409242e-03 Max error: 1.000000e+00 tot_error: 8.749084e-01 REF Iter = 40 Scaled Residual: 6.217941e-03 Max error: 1.000000e+00 tot_error: 8.717186e-01 REF Iter = 41 Scaled Residual: 6.039359e-03 Max error: 1.000000e+00 tot_error: 8.685250e-01 REF Iter = 42 Scaled Residual: 5.883270e-03 Max error: 1.000000e+00 tot_error: 8.653289e-01 REF Iter = 43 Scaled Residual: 5.754394e-03 Max error: 1.000000e+00 tot_error: 8.621323e-01 REF Iter = 44 Scaled Residual: 5.650262e-03 Max error: 1.000000e+00 tot_error: 8.589376e-01 REF Iter = 45 Scaled Residual: 5.560753e-03 Max error: 1.000000e+00 tot_error: 8.557456e-01 REF Iter = 46 Scaled Residual: 5.472830e-03 Max error: 1.000000e+00 tot_error: 8.525560e-01 REF Iter = 47 Scaled Residual: 5.376202e-03 Max error: 1.000000e+00 tot_error: 8.493677e-01 REF Iter = 48 Scaled Residual: 5.268264e-03 Max error: 1.000000e+00 tot_error: 8.461791e-01 REF Iter = 49 Scaled Residual: 5.155525e-03 Max error: 1.000000e+00 tot_error: 8.429893e-01 REF Iter = 50 Scaled Residual: 5.046795e-03 Max error: 1.000000e+00 tot_error: 8.397986e-01 Optimization... Optimization time: 4.808322e-01 sec Validation... Initial Residual: 6.518567e+10 Max_err: 1.000000e+00 tot_err: 1.390229e+05 Iteration = 1 Scaled Residual: 3.226868e-02 Max error: 4.768425e+00 tot_error: 9.992236e-01 Iteration = 2 Scaled Residual: 5.044270e-03 Max error: 1.128076e+00 tot_error: 9.992235e-01 Iteration = 3 Scaled Residual: 1.341135e-03 Max error: 1.000000e+00 tot_error: 9.992235e-01 Iteration = 4 Scaled Residual: 4.654539e-04 Max error: 1.000000e+00 tot_error: 9.992235e-01 Iteration = 5 Scaled Residual: 1.881065e-04 Max error: 1.000000e+00 tot_error: 9.992235e-01 Iteration = 6 Scaled Residual: 7.887786e-05 Max error: 1.000000e+00 tot_error: 9.992235e-01 Iteration = 7 Scaled Residual: 3.075648e-05 Max error: 1.000000e+00 tot_error: 9.992235e-01 Iteration = 8 Scaled Residual: 1.030210e-05 Max error: 1.000000e+00 tot_error: 9.992235e-01 Iteration = 9 Scaled Residual: 2.541873e-06 Max error: 1.000000e+00 tot_error: 9.992235e-01 Iteration = 10 Scaled Residual: 3.461314e-08 Max error: 1.000000e+00 tot_error: 9.992235e-01 Iteration = 11 Scaled Residual: 1.197965e-14 Max error: 1.000000e+00 tot_error: 9.992235e-01 Initial Residual: 6.518567e+10 Max_err: 1.000000e+00 tot_err: 1.390229e+05 Iteration = 1 Scaled Residual: 3.226868e-02 Max error: 4.768425e+00 tot_error: 9.992236e-01 Iteration = 2 Scaled Residual: 5.044270e-03 Max error: 1.128076e+00 tot_error: 9.992235e-01 Iteration = 3 Scaled Residual: 1.341135e-03 Max error: 1.000000e+00 tot_error: 9.992235e-01 Iteration = 4 Scaled Residual: 4.654539e-04 Max error: 1.000000e+00 tot_error: 9.992235e-01 Iteration = 5 Scaled Residual: 1.881065e-04 Max error: 1.000000e+00 tot_error: 9.992235e-01 Iteration = 6 Scaled Residual: 7.887786e-05 Max error: 1.000000e+00 tot_error: 9.992235e-01 Iteration = 7 Scaled Residual: 3.075648e-05 Max error: 1.000000e+00 tot_error: 9.992235e-01 Iteration = 8 Scaled Residual: 1.030210e-05 Max error: 1.000000e+00 tot_error: 9.992235e-01 Iteration = 9 Scaled Residual: 2.541873e-06 Max error: 1.000000e+00 tot_error: 9.992235e-01 Iteration = 10 Scaled Residual: 3.461314e-08 Max error: 1.000000e+00 tot_error: 9.992235e-01 Iteration = 11 Scaled Residual: 1.197965e-14 Max error: 1.000000e+00 tot_error: 9.992235e-01 Initial Residual: 6.518567e+10 Max_err: 1.000000e+00 tot_err: 1.390229e+05 Iteration = 1 Scaled Residual: 3.184539e-15 Max error: 1.000000e+00 tot_error: 9.992235e-01 Initial Residual: 6.518567e+10 Max_err: 1.000000e+00 tot_err: 1.390229e+05 Iteration = 1 Scaled Residual: 3.184539e-15 Max error: 1.000000e+00 tot_error: 9.992235e-01 Optimized CG Setup... Initial Residual: 6.518502e+04 Max_err: 1.000000e+00 tot_err: 1.390229e+05 Iteration = 1 Scaled Residual: 2.199502e-01 Max error: 1.000000e+00 tot_error: 9.964755e-01 Iteration = 2 Scaled Residual: 1.188462e-01 Max error: 1.000000e+00 tot_error: 9.931593e-01 Iteration = 3 Scaled Residual: 8.091535e-02 Max error: 1.000000e+00 tot_error: 9.899070e-01 Iteration = 4 Scaled Residual: 6.135929e-02 Max error: 1.000000e+00 tot_error: 9.866773e-01 Iteration = 5 Scaled Residual: 4.948738e-02 Max error: 1.000000e+00 tot_error: 9.834507e-01 Iteration = 6 Scaled Residual: 4.146986e-02 Max error: 1.000000e+00 tot_error: 9.802250e-01 Iteration = 7 Scaled Residual: 3.570629e-02 Max error: 1.000000e+00 tot_error: 9.769972e-01 Iteration = 8 Scaled Residual: 3.136710e-02 Max error: 1.000000e+00 tot_error: 9.737675e-01 Iteration = 9 Scaled Residual: 2.796789e-02 Max error: 1.000000e+00 tot_error: 9.705360e-01 Iteration = 10 Scaled Residual: 2.523905e-02 Max error: 1.000000e+00 tot_error: 9.673034e-01 Iteration = 11 Scaled Residual: 2.300132e-02 Max error: 1.000000e+00 tot_error: 9.640693e-01 Iteration = 12 Scaled Residual: 2.112796e-02 Max error: 1.000000e+00 tot_error: 9.608345e-01 Iteration = 13 Scaled Residual: 1.954098e-02 Max error: 1.000000e+00 tot_error: 9.575988e-01 Iteration = 14 Scaled Residual: 1.817824e-02 Max error: 1.000000e+00 tot_error: 9.543623e-01 Iteration = 15 Scaled Residual: 1.699473e-02 Max error: 1.000000e+00 tot_error: 9.511253e-01 Iteration = 16 Scaled Residual: 1.595962e-02 Max error: 1.000000e+00 tot_error: 9.478881e-01 Iteration = 17 Scaled Residual: 1.504609e-02 Max error: 1.000000e+00 tot_error: 9.446505e-01 Iteration = 18 Scaled Residual: 1.423472e-02 Max error: 1.000000e+00 tot_error: 9.414129e-01 Iteration = 19 Scaled Residual: 1.351047e-02 Max error: 1.000000e+00 tot_error: 9.381751e-01 Iteration = 20 Scaled Residual: 1.286019e-02 Max error: 1.000000e+00 tot_error: 9.349376e-01 Iteration = 21 Scaled Residual: 1.227394e-02 Max error: 1.000000e+00 tot_error: 9.317003e-01 Iteration = 22 Scaled Residual: 1.174288e-02 Max error: 1.000000e+00 tot_error: 9.284637e-01 Iteration = 23 Scaled Residual: 1.126028e-02 Max error: 1.000000e+00 tot_error: 9.252273e-01 Iteration = 24 Scaled Residual: 1.082003e-02 Max error: 1.000000e+00 tot_error: 9.219920e-01 Iteration = 25 Scaled Residual: 1.041578e-02 Max error: 1.000000e+00 tot_error: 9.187572e-01 Iteration = 26 Scaled Residual: 1.004327e-02 Max error: 1.000000e+00 tot_error: 9.155235e-01 Iteration = 27 Scaled Residual: 9.698156e-03 Max error: 1.000000e+00 tot_error: 9.122906e-01 Iteration = 28 Scaled Residual: 9.376031e-03 Max error: 1.000000e+00 tot_error: 9.090588e-01 Iteration = 29 Scaled Residual: 9.074010e-03 Max error: 1.000000e+00 tot_error: 9.058275e-01 Iteration = 30 Scaled Residual: 8.788677e-03 Max error: 1.000000e+00 tot_error: 9.025970e-01 Iteration = 31 Scaled Residual: 8.517456e-03 Max error: 1.000000e+00 tot_error: 8.993667e-01 Iteration = 32 Scaled Residual: 8.259303e-03 Max error: 1.000000e+00 tot_error: 8.961367e-01 Iteration = 33 Scaled Residual: 8.012526e-03 Max error: 1.000000e+00 tot_error: 8.929063e-01 Iteration = 34 Scaled Residual: 7.776241e-03 Max error: 1.000000e+00 tot_error: 8.896757e-01 Iteration = 35 Scaled Residual: 7.550538e-03 Max error: 1.000000e+00 tot_error: 8.864441e-01 Iteration = 36 Scaled Residual: 7.334628e-03 Max error: 1.000000e+00 tot_error: 8.832119e-01 Iteration = 37 Scaled Residual: 7.128275e-03 Max error: 1.000000e+00 tot_error: 8.799785e-01 Iteration = 38 Scaled Residual: 6.931712e-03 Max error: 1.000000e+00 tot_error: 8.767443e-01 Iteration = 39 Scaled Residual: 6.744167e-03 Max error: 1.000000e+00 tot_error: 8.735088e-01 Iteration = 40 Scaled Residual: 6.565368e-03 Max error: 1.000000e+00 tot_error: 8.702723e-01 Iteration = 41 Scaled Residual: 6.395456e-03 Max error: 1.000000e+00 tot_error: 8.670348e-01 Iteration = 42 Scaled Residual: 6.233812e-03 Max error: 1.000000e+00 tot_error: 8.637967e-01 Iteration = 43 Scaled Residual: 6.080526e-03 Max error: 1.000000e+00 tot_error: 8.605577e-01 Iteration = 44 Scaled Residual: 5.935919e-03 Max error: 1.000000e+00 tot_error: 8.573183e-01 Iteration = 45 Scaled Residual: 5.799550e-03 Max error: 1.000000e+00 tot_error: 8.540781e-01 Iteration = 46 Scaled Residual: 5.671669e-03 Max error: 1.000000e+00 tot_error: 8.508371e-01 Iteration = 47 Scaled Residual: 5.552240e-03 Max error: 1.000000e+00 tot_error: 8.475950e-01 Iteration = 48 Scaled Residual: 5.440239e-03 Max error: 1.000000e+00 tot_error: 8.443522e-01 Iteration = 49 Scaled Residual: 5.335545e-03 Max error: 1.000000e+00 tot_error: 8.411089e-01 Iteration = 50 Scaled Residual: 5.238043e-03 Max error: 1.000000e+00 tot_error: 8.378659e-01 Iteration = 51 Scaled Residual: 5.146581e-03 Max error: 1.000000e+00 tot_error: 8.346227e-01 Iteration = 52 Scaled Residual: 5.060014e-03 Max error: 1.000000e+00 tot_error: 8.313798e-01 Iteration = 53 Scaled Residual: 4.976792e-03 Max error: 1.000000e+00 tot_error: 8.281371e-01 Starting Benchmarking Phase... Performing 899 CG sets expected time: 3800.0 seconds expected Perf: 83585.0 GF (326.5 GF_per) 2022-05-13 20:05:39.306 progress = 1.0% 38.1 / 3800.0 sec elapsed 3761.9 sec remain 83496.502 GF 326.158 GF_per progress = 2.0% 76.3 / 3800.0 sec elapsed 3723.7 sec remain 83459.028 GF 326.012 GF_per progress = 3.0% 114.4 / 3800.0 sec elapsed 3685.6 sec remain 83471.780 GF 326.062 GF_per progress = 4.0% 152.5 / 3800.0 sec elapsed 3647.5 sec remain 83469.965 GF 326.055 GF_per progress = 5.0% 190.7 / 3800.0 sec elapsed 3609.3 sec remain 83462.515 GF 326.025 GF_per progress = 6.0% 228.8 / 3800.0 sec elapsed 3571.2 sec remain 83465.267 GF 326.036 GF_per progress = 7.0% 266.9 / 3800.0 sec elapsed 3533.1 sec remain 83462.558 GF 326.026 GF_per progress = 8.0% 305.1 / 3800.0 sec elapsed 3494.9 sec remain 83464.925 GF 326.035 GF_per progress = 9.0% 343.2 / 3800.0 sec elapsed 3456.8 sec remain 83466.014 GF 326.039 GF_per progress = 10.0% 381.3 / 3800.0 sec elapsed 3418.7 sec remain 83465.324 GF 326.036 GF_per progress = 11.0% 419.5 / 3800.0 sec elapsed 3380.5 sec remain 83467.659 GF 326.046 GF_per progress = 12.0% 457.6 / 3800.0 sec elapsed 3342.4 sec remain 83468.519 GF 326.049 GF_per progress = 13.0% 495.7 / 3800.0 sec elapsed 3304.3 sec remain 83469.628 GF 326.053 GF_per progress = 14.0% 533.8 / 3800.0 sec elapsed 3266.2 sec remain 83469.788 GF 326.054 GF_per progress = 15.1% 572.0 / 3800.0 sec elapsed 3228.0 sec remain 83466.798 GF 326.042 GF_per progress = 16.1% 610.1 / 3800.0 sec elapsed 3189.9 sec remain 83466.737 GF 326.042 GF_per progress = 17.1% 648.3 / 3800.0 sec elapsed 3151.7 sec remain 83465.541 GF 326.037 GF_per progress = 18.1% 686.4 / 3800.0 sec elapsed 3113.6 sec remain 83464.955 GF 326.035 GF_per progress = 19.1% 724.5 / 3800.0 sec elapsed 3075.5 sec remain 83465.392 GF 326.037 GF_per progress = 20.1% 762.7 / 3800.0 sec elapsed 3037.3 sec remain 83461.967 GF 326.023 GF_per progress = 21.1% 800.8 / 3800.0 sec elapsed 2999.2 sec remain 83462.700 GF 326.026 GF_per progress = 22.1% 839.0 / 3800.0 sec elapsed 2961.0 sec remain 83462.486 GF 326.025 GF_per progress = 23.1% 877.1 / 3800.0 sec elapsed 2922.9 sec remain 83461.678 GF 326.022 GF_per progress = 24.1% 915.3 / 3800.0 sec elapsed 2884.7 sec remain 83461.923 GF 326.023 GF_per progress = 25.1% 953.4 / 3800.0 sec elapsed 2846.6 sec remain 83461.554 GF 326.022 GF_per progress = 26.1% 991.5 / 3800.0 sec elapsed 2808.5 sec remain 83460.567 GF 326.018 GF_per progress = 27.1% 1029.7 / 3800.0 sec elapsed 2770.3 sec remain 83459.422 GF 326.013 GF_per progress = 28.1% 1067.8 / 3800.0 sec elapsed 2732.2 sec remain 83458.827 GF 326.011 GF_per progress = 29.1% 1106.0 / 3800.0 sec elapsed 2694.0 sec remain 83459.894 GF 326.015 GF_per progress = 30.1% 1144.1 / 3800.0 sec elapsed 2655.9 sec remain 83461.313 GF 326.021 GF_per progress = 31.1% 1182.2 / 3800.0 sec elapsed 2617.8 sec remain 83460.746 GF 326.019 GF_per progress = 32.1% 1220.4 / 3800.0 sec elapsed 2579.6 sec remain 83460.619 GF 326.018 GF_per progress = 33.1% 1258.5 / 3800.0 sec elapsed 2541.5 sec remain 83459.623 GF 326.014 GF_per progress = 34.1% 1296.7 / 3800.0 sec elapsed 2503.3 sec remain 83459.278 GF 326.013 GF_per progress = 35.1% 1334.8 / 3800.0 sec elapsed 2465.2 sec remain 83459.389 GF 326.013 GF_per progress = 36.1% 1372.9 / 3800.0 sec elapsed 2427.1 sec remain 83458.524 GF 326.010 GF_per progress = 37.1% 1411.1 / 3800.0 sec elapsed 2388.9 sec remain 83457.887 GF 326.007 GF_per progress = 38.1% 1449.2 / 3800.0 sec elapsed 2350.8 sec remain 83457.821 GF 326.007 GF_per progress = 39.1% 1487.4 / 3800.0 sec elapsed 2312.6 sec remain 83457.089 GF 326.004 GF_per progress = 40.1% 1525.5 / 3800.0 sec elapsed 2274.5 sec remain 83456.917 GF 326.004 GF_per progress = 41.1% 1563.6 / 3800.0 sec elapsed 2236.4 sec remain 83457.604 GF 326.006 GF_per progress = 42.2% 1601.8 / 3800.0 sec elapsed 2198.2 sec remain 83458.444 GF 326.010 GF_per progress = 43.2% 1639.9 / 3800.0 sec elapsed 2160.1 sec remain 83458.874 GF 326.011 GF_per progress = 44.2% 1678.0 / 3800.0 sec elapsed 2122.0 sec remain 83458.722 GF 326.011 GF_per progress = 45.2% 1716.2 / 3800.0 sec elapsed 2083.8 sec remain 83459.156 GF 326.012 GF_per progress = 46.2% 1754.3 / 3800.0 sec elapsed 2045.7 sec remain 83459.292 GF 326.013 GF_per progress = 47.2% 1792.4 / 3800.0 sec elapsed 2007.6 sec remain 83459.050 GF 326.012 GF_per progress = 48.2% 1830.6 / 3800.0 sec elapsed 1969.4 sec remain 83458.726 GF 326.011 GF_per progress = 49.2% 1868.7 / 3800.0 sec elapsed 1931.3 sec remain 83458.445 GF 326.010 GF_per progress = 50.2% 1906.9 / 3800.0 sec elapsed 1893.1 sec remain 83458.541 GF 326.010 GF_per progress = 51.2% 1945.0 / 3800.0 sec elapsed 1855.0 sec remain 83458.698 GF 326.011 GF_per progress = 52.2% 1983.1 / 3800.0 sec elapsed 1816.9 sec remain 83459.544 GF 326.014 GF_per progress = 53.2% 2021.3 / 3800.0 sec elapsed 1778.7 sec remain 83459.057 GF 326.012 GF_per progress = 54.2% 2059.4 / 3800.0 sec elapsed 1740.6 sec remain 83459.455 GF 326.013 GF_per progress = 55.2% 2097.5 / 3800.0 sec elapsed 1702.5 sec remain 83459.388 GF 326.013 GF_per progress = 56.2% 2135.7 / 3800.0 sec elapsed 1664.3 sec remain 83459.489 GF 326.014 GF_per progress = 57.2% 2173.8 / 3800.0 sec elapsed 1626.2 sec remain 83459.061 GF 326.012 GF_per progress = 58.2% 2211.9 / 3800.0 sec elapsed 1588.1 sec remain 83459.269 GF 326.013 GF_per progress = 59.2% 2250.1 / 3800.0 sec elapsed 1549.9 sec remain 83458.286 GF 326.009 GF_per progress = 60.2% 2288.2 / 3800.0 sec elapsed 1511.8 sec remain 83458.072 GF 326.008 GF_per progress = 61.2% 2326.4 / 3800.0 sec elapsed 1473.6 sec remain 83458.078 GF 326.008 GF_per progress = 62.2% 2364.6 / 3800.0 sec elapsed 1435.4 sec remain 83457.092 GF 326.004 GF_per progress = 63.2% 2402.7 / 3800.0 sec elapsed 1397.3 sec remain 83457.189 GF 326.005 GF_per progress = 64.2% 2440.8 / 3800.0 sec elapsed 1359.2 sec remain 83457.557 GF 326.006 GF_per progress = 65.2% 2479.0 / 3800.0 sec elapsed 1321.0 sec remain 83457.271 GF 326.005 GF_per progress = 66.2% 2517.1 / 3800.0 sec elapsed 1282.9 sec remain 83456.996 GF 326.004 GF_per progress = 67.2% 2555.2 / 3800.0 sec elapsed 1244.8 sec remain 83456.961 GF 326.004 GF_per progress = 68.2% 2593.4 / 3800.0 sec elapsed 1206.6 sec remain 83456.948 GF 326.004 GF_per progress = 69.3% 2631.5 / 3800.0 sec elapsed 1168.5 sec remain 83457.061 GF 326.004 GF_per progress = 70.3% 2669.7 / 3800.0 sec elapsed 1130.3 sec remain 83457.250 GF 326.005 GF_per progress = 71.3% 2707.8 / 3800.0 sec elapsed 1092.2 sec remain 83457.391 GF 326.005 GF_per progress = 72.3% 2745.9 / 3800.0 sec elapsed 1054.1 sec remain 83458.134 GF 326.008 GF_per progress = 73.3% 2784.0 / 3800.0 sec elapsed 1016.0 sec remain 83458.193 GF 326.009 GF_per progress = 74.3% 2822.2 / 3800.0 sec elapsed 977.8 sec remain 83458.345 GF 326.009 GF_per progress = 75.3% 2860.3 / 3800.0 sec elapsed 939.7 sec remain 83458.100 GF 326.008 GF_per progress = 76.3% 2898.5 / 3800.0 sec elapsed 901.5 sec remain 83458.025 GF 326.008 GF_per progress = 77.3% 2936.6 / 3800.0 sec elapsed 863.4 sec remain 83458.550 GF 326.010 GF_per progress = 78.3% 2974.7 / 3800.0 sec elapsed 825.3 sec remain 83458.902 GF 326.011 GF_per progress = 79.3% 3012.8 / 3800.0 sec elapsed 787.2 sec remain 83458.718 GF 326.011 GF_per progress = 80.3% 3051.0 / 3800.0 sec elapsed 749.0 sec remain 83458.725 GF 326.011 GF_per progress = 81.3% 3089.1 / 3800.0 sec elapsed 710.9 sec remain 83459.255 GF 326.013 GF_per progress = 82.3% 3127.2 / 3800.0 sec elapsed 672.8 sec remain 83459.294 GF 326.013 GF_per progress = 83.3% 3165.4 / 3800.0 sec elapsed 634.6 sec remain 83459.421 GF 326.013 GF_per progress = 84.3% 3203.5 / 3800.0 sec elapsed 596.5 sec remain 83460.122 GF 326.016 GF_per progress = 85.3% 3241.6 / 3800.0 sec elapsed 558.4 sec remain 83460.499 GF 326.018 GF_per progress = 86.3% 3279.7 / 3800.0 sec elapsed 520.3 sec remain 83460.423 GF 326.017 GF_per progress = 87.3% 3317.8 / 3800.0 sec elapsed 482.2 sec remain 83461.211 GF 326.020 GF_per progress = 88.3% 3355.9 / 3800.0 sec elapsed 444.1 sec remain 83461.773 GF 326.023 GF_per progress = 89.3% 3394.1 / 3800.0 sec elapsed 405.9 sec remain 83462.055 GF 326.024 GF_per progress = 90.3% 3432.2 / 3800.0 sec elapsed 367.8 sec remain 83462.295 GF 326.025 GF_per progress = 91.3% 3470.3 / 3800.0 sec elapsed 329.7 sec remain 83462.057 GF 326.024 GF_per progress = 92.3% 3508.5 / 3800.0 sec elapsed 291.5 sec remain 83462.039 GF 326.024 GF_per progress = 93.3% 3546.6 / 3800.0 sec elapsed 253.4 sec remain 83461.998 GF 326.023 GF_per progress = 94.3% 3584.8 / 3800.0 sec elapsed 215.2 sec remain 83461.961 GF 326.023 GF_per progress = 95.3% 3622.9 / 3800.0 sec elapsed 177.1 sec remain 83462.005 GF 326.023 GF_per progress = 96.3% 3661.0 / 3800.0 sec elapsed 139.0 sec remain 83462.105 GF 326.024 GF_per progress = 97.3% 3699.2 / 3800.0 sec elapsed 100.8 sec remain 83461.876 GF 326.023 GF_per progress = 98.3% 3737.3 / 3800.0 sec elapsed 62.7 sec remain 83462.134 GF 326.024 GF_per progress = 99.4% 3775.4 / 3800.0 sec elapsed 24.6 sec remain 83461.794 GF 326.023 GF_per Completed Benchmarking Phase... elapsed time: 3809.3 seconds 2022-05-13 21:09:08.682 Number of CG sets: 899 Iterations per set: 53 scaled res mean: 4.976792e-03 scaled res variance: 0.000000e+00 Total Time: 3.809326e+03 sec Setup Overhead: 1.29% Optimization Overhead: 1.12% Convergence Overhead: 5.66% 8x8x4 process grid 512x512x288 local domain SpMV = 75397.4 GF (474744.5 GB/s Effective) 294.5 GF_per (1854.5 GB/s Effective) SymGS = 109684.9 GF (846501.3 GB/s Effective) 428.5 GF_per (3306.6 GB/s Effective) total = 90631.3 GF (687233.2 GB/s Effective) 354.0 GF_per (2684.5 GB/s Effective) final = 83462.0 GF (632870.7 GB/s Effective) 326.0 GF_per (2472.2 GB/s Effective) end of application... 2022-05-13 21:09:09.370 === JOB_STATISTICS === === current date : Fri May 13 21:09:15 CEST 2022 = Job-ID : 237145 on alex = Job-Name : cudahpcg = Job-Command : /home/hpc/hpcg/mtec003h/hpcg/hpcg-nvidia-mh.slurm = Initial workdir : /home/hpc/hpcg/mtec003h/hpcg = Partition (Res) : a100_all (hpl) = Slurm account : mtec with QOS=normal = Requested resources: cpu=4096,mem=30000G,node=32,billing=4096,gres/gpu=256,gres/gpu:a100=256 for 04:00:00 = Elapsed runtime : 01:18:20 = Total RAM usage : 433.1 GiB of assigned 30000 GiB (1.4%) = Node list : a[0531-0537,0601-0605,0701-0705,0801-0805,0831,0901-0905,0931-0934] = Subm/Elig/Start/End: 2022-05-13T19:45:56 / 2022-05-13T19:45:56 / 2022-05-13T19:50:55 / 2022-05-13T21:09:15 ====================== === Quota infos ====== Path Used SoftQ HardQ Gracetime Filec FileQ FiHaQ FileGrace /home/hpc 7.1G 52.4G 104.9G N/A 78K 500K 1,000K N/A /home/woody 4.0K 500.0G 999.0G N/A 1 N/A /home/vault 0.0K 524.3G 1048.6G N/A 1 200K 400K N/A ====================== === GPU utilization == gpu_name, gpu_bus_id, pid, gpu_utilization [%], mem_utilization [%], max_memory_usage [MiB], time [ms] NVIDIA A100-SXM4-80GB, 00000000:0E:00.0, 22072, 65 %, 51 %, 38433 MiB, 4676046 ms NVIDIA A100-SXM4-80GB, 00000000:13:00.0, 22003, 66 %, 51 %, 38497 MiB, 4678804 ms NVIDIA A100-SXM4-80GB, 00000000:49:00.0, 22006, 66 %, 51 %, 38497 MiB, 4677924 ms NVIDIA A100-SXM4-80GB, 00000000:4F:00.0, 22007, 66 %, 51 %, 38497 MiB, 4679225 ms NVIDIA A100-SXM4-80GB, 00000000:90:00.0, 22009, 66 %, 51 %, 38497 MiB, 4678077 ms NVIDIA A100-SXM4-80GB, 00000000:96:00.0, 22012, 66 %, 51 %, 38497 MiB, 4676667 ms NVIDIA A100-SXM4-80GB, 00000000:CC:00.0, 22013, 66 %, 51 %, 38497 MiB, 4679345 ms NVIDIA A100-SXM4-80GB, 00000000:D1:00.0, 22014, 66 %, 51 %, 38433 MiB, 4676731 ms