CUDA Toolkit Version: nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2020 NVIDIA Corporation Built on Thu_Jun_11_22:26:38_PDT_2020 Cuda compilation tools, release 11.0, V11.0.194 Build cuda_11.0_bu.TC445_37.28540450_0 Loaded HPCX: /home/mhill2/hpl-new/hpl-ep/apps/hpcx/hpcx-v2.7.4-gcc-MLNX_OFED_LINUX-5.1-0.6.6.0-redhat7.7-x86_64 UCX Version: # UCT version=1.9.0 revision 5c8e0f4 # configured with: --disable-logging --disable-debug --disable-assertions --disable-params-check --with-knem --with-xpmem=/hpc/local/oss/xpmem/90a95a4 --without-java --enable-devel-headers --with-cuda=/hpc/local/oss/cuda11.0 --with-gdrcopy --prefix=/build-result/hpcx-v2.7.4-gcc-MLNX_OFED_LINUX-5.1-0.6.6.0-redhat7.7-x86_64/ucx OMPI Version: Open MPI v4.0.4rc3 http://www.open-mpi.org/community/help/ HPCG HOSTS: c0800a-s11 c0800a-s17 c0800a-s23 c0800a-s29 c0800a-s35 c0801a-s11 c0801a-s17 c0801a-s23 c0801a-s29 c0801a-s35 c0803a-s11 c0803a-s17 c0803a-s23 c0803a-s29 c0803a-s35 c0804a-s11 c0804a-s17 c0804a-s23 c0804a-s29 c0804a-s35 c0900a-s11 c0900a-s17 c0900a-s23 c0900a-s29 c0900a-s35 c0901a-s11 c0901a-s17 c0901a-s23 c0901a-s29 c0901a-s35 c0903a-s11 c0903a-s17 c0903a-s23 c0903a-s29 c0903a-s35 c0904a-s11 c0904a-s17 c0904a-s23 c0904a-s29 c0904a-s35 c0906a-s11 c0906a-s17 c0906a-s23 c0906a-s29 c0906a-s35 c0907a-s11 c0907a-s17 c0907a-s23 c0907a-s29 c0907a-s35 c0909a-s11 c0909a-s17 c0909a-s23 c0909a-s29 c0909a-s35 c0910a-s11 c0910a-s17 c0910a-s23 c0910a-s29 c0910a-s35 c1000a-s11 c1000a-s17 c1000a-s23 c1000a-s29 c1000a-s35 c1001a-s11 c1001a-s17 c1001a-s23 c1001a-s29 c1001a-s35 c1003a-s11 c1003a-s17 c1003a-s23 c1003a-s29 c1003a-s35 c1004a-s11 c1004a-s17 c1004a-s23 c1004a-s29 c1004a-s35 c1006a-s11 c1006a-s17 c1006a-s23 c1006a-s29 c1006a-s35 c1007a-s11 c1007a-s17 c1007a-s23 c1007a-s29 c1007a-s35 c1009a-s11 c1009a-s17 c1009a-s23 c1009a-s29 c1009a-s35 c1010a-s11 c1010a-s17 c1010a-s23 c1010a-s29 c1010a-s35 c1100a-s11 c1100a-s17 c1100a-s23 c1100a-s29 c1100a-s35 c1101a-s11 c1101a-s17 c1101a-s23 c1101a-s29 c1101a-s35 c1103a-s11 c1103a-s17 c1103a-s23 c1103a-s29 c1103a-s35 c1104a-s11 c1104a-s17 c1104a-s23 c1104a-s29 c1104a-s35 c1106a-s11 c1106a-s17 c1106a-s23 c1106a-s29 c1106a-s35 c1107a-s11 c1107a-s17 c1107a-s23 c1107a-s29 c1107a-s35 c1109a-s11 c1109a-s17 c1109a-s23 c1109a-s29 c1109a-s35 c1110a-s11 c1110a-s17 c1110a-s23 c1110a-s29 c1110a-s35 monitor_gpu from environment variable 1 gpu_temp_warning from environment variable 78 gpu_clock_warning from environment variable 1275 gpu_power_warning from environment variable 410 gpu_pcie_gen_warning from environment variable 3 gpu_pcie_width_warning from environment variable 2 start of application (4 OMP threads)... 2021-05-13 22:08:33.745 Problem setup... Setup time: 0.486257 sec GPU: 'A100-SXM-80GB' Memory use: 36898 MB / 81252 MB 8x10x14 process grid 512x512x288 local domain Reference SpMV+MG... Reference CG... Initial Residual: 9.705909e+04 Max_err: 1.000000e+00 tot_err: 2.907872e+05 REF Iter = 1 Scaled Residual: 1.872259e-01 Max error: 1.000000e+00 tot_error: 9.982270e-01 REF Iter = 2 Scaled Residual: 1.029686e-01 Max error: 1.000000e+00 tot_error: 9.965742e-01 REF Iter = 3 Scaled Residual: 7.113964e-02 Max error: 1.000000e+00 tot_error: 9.949458e-01 REF Iter = 4 Scaled Residual: 5.438279e-02 Max error: 1.000000e+00 tot_error: 9.933237e-01 REF Iter = 5 Scaled Residual: 4.399943e-02 Max error: 1.000000e+00 tot_error: 9.917036e-01 REF Iter = 6 Scaled Residual: 3.693199e-02 Max error: 1.000000e+00 tot_error: 9.900846e-01 REF Iter = 7 Scaled Residual: 3.182128e-02 Max error: 1.000000e+00 tot_error: 9.884663e-01 REF Iter = 8 Scaled Residual: 2.796961e-02 Max error: 1.000000e+00 tot_error: 9.868489e-01 REF Iter = 9 Scaled Residual: 2.497477e-02 Max error: 1.000000e+00 tot_error: 9.852326e-01 REF Iter = 10 Scaled Residual: 2.258145e-02 Max error: 1.000000e+00 tot_error: 9.836177e-01 REF Iter = 11 Scaled Residual: 2.061859e-02 Max error: 1.000000e+00 tot_error: 9.820040e-01 REF Iter = 12 Scaled Residual: 1.896861e-02 Max error: 1.000000e+00 tot_error: 9.803913e-01 REF Iter = 13 Scaled Residual: 1.755060e-02 Max error: 1.000000e+00 tot_error: 9.787793e-01 REF Iter = 14 Scaled Residual: 1.631409e-02 Max error: 1.000000e+00 tot_error: 9.771675e-01 REF Iter = 15 Scaled Residual: 1.522898e-02 Max error: 1.000000e+00 tot_error: 9.755560e-01 REF Iter = 16 Scaled Residual: 1.427582e-02 Max error: 1.000000e+00 tot_error: 9.739447e-01 REF Iter = 17 Scaled Residual: 1.344081e-02 Max error: 1.000000e+00 tot_error: 9.723341e-01 REF Iter = 18 Scaled Residual: 1.271025e-02 Max error: 1.000000e+00 tot_error: 9.707244e-01 REF Iter = 19 Scaled Residual: 1.206800e-02 Max error: 1.000000e+00 tot_error: 9.691160e-01 REF Iter = 20 Scaled Residual: 1.149731e-02 Max error: 1.000000e+00 tot_error: 9.675090e-01 REF Iter = 21 Scaled Residual: 1.098171e-02 Max error: 1.000000e+00 tot_error: 9.659032e-01 REF Iter = 22 Scaled Residual: 1.050758e-02 Max error: 1.000000e+00 tot_error: 9.642983e-01 REF Iter = 23 Scaled Residual: 1.006798e-02 Max error: 1.000000e+00 tot_error: 9.626941e-01 REF Iter = 24 Scaled Residual: 9.661812e-03 Max error: 1.000000e+00 tot_error: 9.610902e-01 REF Iter = 25 Scaled Residual: 9.291139e-03 Max error: 1.000000e+00 tot_error: 9.594869e-01 REF Iter = 26 Scaled Residual: 8.959445e-03 Max error: 1.000000e+00 tot_error: 9.578845e-01 REF Iter = 27 Scaled Residual: 8.667862e-03 Max error: 1.000000e+00 tot_error: 9.562834e-01 REF Iter = 28 Scaled Residual: 8.413160e-03 Max error: 1.000000e+00 tot_error: 9.546842e-01 REF Iter = 29 Scaled Residual: 8.188407e-03 Max error: 1.000000e+00 tot_error: 9.530871e-01 REF Iter = 30 Scaled Residual: 7.983302e-03 Max error: 1.000000e+00 tot_error: 9.514923e-01 REF Iter = 31 Scaled Residual: 7.787530e-03 Max error: 1.000000e+00 tot_error: 9.498994e-01 REF Iter = 32 Scaled Residual: 7.595156e-03 Max error: 1.000000e+00 tot_error: 9.483081e-01 REF Iter = 33 Scaled Residual: 7.404661e-03 Max error: 1.000000e+00 tot_error: 9.467184e-01 REF Iter = 34 Scaled Residual: 7.217405e-03 Max error: 1.000000e+00 tot_error: 9.451303e-01 REF Iter = 35 Scaled Residual: 7.034672e-03 Max error: 1.000000e+00 tot_error: 9.435438e-01 REF Iter = 36 Scaled Residual: 6.854014e-03 Max error: 1.000000e+00 tot_error: 9.419592e-01 REF Iter = 37 Scaled Residual: 6.670556e-03 Max error: 1.000000e+00 tot_error: 9.403763e-01 REF Iter = 38 Scaled Residual: 6.480924e-03 Max error: 1.000000e+00 tot_error: 9.387943e-01 REF Iter = 39 Scaled Residual: 6.286129e-03 Max error: 1.000000e+00 tot_error: 9.372125e-01 REF Iter = 40 Scaled Residual: 6.093231e-03 Max error: 1.000000e+00 tot_error: 9.356302e-01 REF Iter = 41 Scaled Residual: 5.912381e-03 Max error: 1.000000e+00 tot_error: 9.340471e-01 REF Iter = 42 Scaled Residual: 5.751743e-03 Max error: 1.000000e+00 tot_error: 9.324638e-01 REF Iter = 43 Scaled Residual: 5.615333e-03 Max error: 1.000000e+00 tot_error: 9.308809e-01 REF Iter = 44 Scaled Residual: 5.502009e-03 Max error: 1.000000e+00 tot_error: 9.292991e-01 REF Iter = 45 Scaled Residual: 5.405587e-03 Max error: 1.000000e+00 tot_error: 9.277189e-01 REF Iter = 46 Scaled Residual: 5.317761e-03 Max error: 1.000000e+00 tot_error: 9.261403e-01 REF Iter = 47 Scaled Residual: 5.231127e-03 Max error: 1.000000e+00 tot_error: 9.245632e-01 REF Iter = 48 Scaled Residual: 5.141663e-03 Max error: 1.000000e+00 tot_error: 9.229873e-01 REF Iter = 49 Scaled Residual: 5.049893e-03 Max error: 1.000000e+00 tot_error: 9.214126e-01 REF Iter = 50 Scaled Residual: 4.958052e-03 Max error: 1.000000e+00 tot_error: 9.198392e-01 Optimization... Optimization time: 5.043291e-01 sec Validation... Initial Residual: 9.705953e+10 Max_err: 1.000000e+00 tot_err: 2.907872e+05 Iteration = 1 Scaled Residual: 2.167359e-02 Max error: 4.768868e+00 tot_error: 9.996065e-01 Iteration = 2 Scaled Residual: 3.387771e-03 Max error: 1.128079e+00 tot_error: 9.996065e-01 Iteration = 3 Scaled Residual: 9.007134e-04 Max error: 1.000000e+00 tot_error: 9.996065e-01 Iteration = 4 Scaled Residual: 3.126013e-04 Max error: 1.000000e+00 tot_error: 9.996065e-01 Iteration = 5 Scaled Residual: 1.263334e-04 Max error: 1.000000e+00 tot_error: 9.996065e-01 Iteration = 6 Scaled Residual: 5.297489e-05 Max error: 1.000000e+00 tot_error: 9.996065e-01 Iteration = 7 Scaled Residual: 2.065636e-05 Max error: 1.000000e+00 tot_error: 9.996065e-01 Iteration = 8 Scaled Residual: 6.919148e-06 Max error: 1.000000e+00 tot_error: 9.996065e-01 Iteration = 9 Scaled Residual: 1.707526e-06 Max error: 1.000000e+00 tot_error: 9.996065e-01 Iteration = 10 Scaled Residual: 3.461868e-08 Max error: 1.000000e+00 tot_error: 9.996065e-01 Iteration = 11 Scaled Residual: 1.198061e-14 Max error: 1.000000e+00 tot_error: 9.996065e-01 Initial Residual: 9.705953e+10 Max_err: 1.000000e+00 tot_err: 2.907872e+05 Iteration = 1 Scaled Residual: 2.167359e-02 Max error: 4.768868e+00 tot_error: 9.996065e-01 Iteration = 2 Scaled Residual: 3.387771e-03 Max error: 1.128079e+00 tot_error: 9.996065e-01 Iteration = 3 Scaled Residual: 9.007134e-04 Max error: 1.000000e+00 tot_error: 9.996065e-01 Iteration = 4 Scaled Residual: 3.126013e-04 Max error: 1.000000e+00 tot_error: 9.996065e-01 Iteration = 5 Scaled Residual: 1.263334e-04 Max error: 1.000000e+00 tot_error: 9.996065e-01 Iteration = 6 Scaled Residual: 5.297489e-05 Max error: 1.000000e+00 tot_error: 9.996065e-01 Iteration = 7 Scaled Residual: 2.065636e-05 Max error: 1.000000e+00 tot_error: 9.996065e-01 Iteration = 8 Scaled Residual: 6.919148e-06 Max error: 1.000000e+00 tot_error: 9.996065e-01 Iteration = 9 Scaled Residual: 1.707526e-06 Max error: 1.000000e+00 tot_error: 9.996065e-01 Iteration = 10 Scaled Residual: 3.461868e-08 Max error: 1.000000e+00 tot_error: 9.996065e-01 Iteration = 11 Scaled Residual: 1.198061e-14 Max error: 1.000000e+00 tot_error: 9.996065e-01 Initial Residual: 9.705953e+10 Max_err: 1.000000e+00 tot_err: 2.907872e+05 Iteration = 1 Scaled Residual: 3.891571e-15 Max error: 1.000000e+00 tot_error: 9.996065e-01 Initial Residual: 9.705953e+10 Max_err: 1.000000e+00 tot_err: 2.907872e+05 Iteration = 1 Scaled Residual: 3.891571e-15 Max error: 1.000000e+00 tot_error: 9.996065e-01 Optimized CG Setup... Initial Residual: 9.705909e+04 Max_err: 1.000000e+00 tot_err: 2.907872e+05 Iteration = 1 Scaled Residual: 2.208019e-01 Max error: 1.000000e+00 tot_error: 9.982153e-01 Iteration = 2 Scaled Residual: 1.196212e-01 Max error: 1.000000e+00 tot_error: 9.965378e-01 Iteration = 3 Scaled Residual: 8.145402e-02 Max error: 1.000000e+00 tot_error: 9.948928e-01 Iteration = 4 Scaled Residual: 6.174371e-02 Max error: 1.000000e+00 tot_error: 9.932597e-01 Iteration = 5 Scaled Residual: 4.977233e-02 Max error: 1.000000e+00 tot_error: 9.916290e-01 Iteration = 6 Scaled Residual: 4.169633e-02 Max error: 1.000000e+00 tot_error: 9.899994e-01 Iteration = 7 Scaled Residual: 3.589416e-02 Max error: 1.000000e+00 tot_error: 9.883696e-01 Iteration = 8 Scaled Residual: 3.152662e-02 Max error: 1.000000e+00 tot_error: 9.867397e-01 Iteration = 9 Scaled Residual: 2.810844e-02 Max error: 1.000000e+00 tot_error: 9.851095e-01 Iteration = 10 Scaled Residual: 2.536527e-02 Max error: 1.000000e+00 tot_error: 9.834796e-01 Iteration = 11 Scaled Residual: 2.311661e-02 Max error: 1.000000e+00 tot_error: 9.818497e-01 Iteration = 12 Scaled Residual: 2.123548e-02 Max error: 1.000000e+00 tot_error: 9.802203e-01 Iteration = 13 Scaled Residual: 1.964186e-02 Max error: 1.000000e+00 tot_error: 9.785912e-01 Iteration = 14 Scaled Residual: 1.827359e-02 Max error: 1.000000e+00 tot_error: 9.769626e-01 Iteration = 15 Scaled Residual: 1.708571e-02 Max error: 1.000000e+00 tot_error: 9.753345e-01 Iteration = 16 Scaled Residual: 1.604659e-02 Max error: 1.000000e+00 tot_error: 9.737071e-01 Iteration = 17 Scaled Residual: 1.512942e-02 Max error: 1.000000e+00 tot_error: 9.720803e-01 Iteration = 18 Scaled Residual: 1.431463e-02 Max error: 1.000000e+00 tot_error: 9.704544e-01 Iteration = 19 Scaled Residual: 1.358716e-02 Max error: 1.000000e+00 tot_error: 9.688292e-01 Iteration = 20 Scaled Residual: 1.293404e-02 Max error: 1.000000e+00 tot_error: 9.672049e-01 Iteration = 21 Scaled Residual: 1.234520e-02 Max error: 1.000000e+00 tot_error: 9.655816e-01 Iteration = 22 Scaled Residual: 1.181191e-02 Max error: 1.000000e+00 tot_error: 9.639595e-01 Iteration = 23 Scaled Residual: 1.132758e-02 Max error: 1.000000e+00 tot_error: 9.623383e-01 Iteration = 24 Scaled Residual: 1.088604e-02 Max error: 1.000000e+00 tot_error: 9.607186e-01 Iteration = 25 Scaled Residual: 1.048103e-02 Max error: 1.000000e+00 tot_error: 9.591000e-01 Iteration = 26 Scaled Residual: 1.010814e-02 Max error: 1.000000e+00 tot_error: 9.574829e-01 Iteration = 27 Scaled Residual: 9.762726e-03 Max error: 1.000000e+00 tot_error: 9.558671e-01 Iteration = 28 Scaled Residual: 9.440194e-03 Max error: 1.000000e+00 tot_error: 9.542527e-01 Iteration = 29 Scaled Residual: 9.137181e-03 Max error: 1.000000e+00 tot_error: 9.526395e-01 Iteration = 30 Scaled Residual: 8.849919e-03 Max error: 1.000000e+00 tot_error: 9.510276e-01 Iteration = 31 Scaled Residual: 8.575736e-03 Max error: 1.000000e+00 tot_error: 9.494166e-01 Iteration = 32 Scaled Residual: 8.313438e-03 Max error: 1.000000e+00 tot_error: 9.478066e-01 Iteration = 33 Scaled Residual: 8.061543e-03 Max error: 1.000000e+00 tot_error: 9.461973e-01 Iteration = 34 Scaled Residual: 7.819704e-03 Max error: 1.000000e+00 tot_error: 9.445886e-01 Iteration = 35 Scaled Residual: 7.588430e-03 Max error: 1.000000e+00 tot_error: 9.429803e-01 Iteration = 36 Scaled Residual: 7.367459e-03 Max error: 1.000000e+00 tot_error: 9.413724e-01 Iteration = 37 Scaled Residual: 7.157065e-03 Max error: 1.000000e+00 tot_error: 9.397647e-01 Iteration = 38 Scaled Residual: 6.957675e-03 Max error: 1.000000e+00 tot_error: 9.381575e-01 Iteration = 39 Scaled Residual: 6.768737e-03 Max error: 1.000000e+00 tot_error: 9.365506e-01 Iteration = 40 Scaled Residual: 6.590053e-03 Max error: 1.000000e+00 tot_error: 9.349441e-01 Iteration = 41 Scaled Residual: 6.421506e-03 Max error: 1.000000e+00 tot_error: 9.333381e-01 Iteration = 42 Scaled Residual: 6.262328e-03 Max error: 1.000000e+00 tot_error: 9.317327e-01 Iteration = 43 Scaled Residual: 6.112370e-03 Max error: 1.000000e+00 tot_error: 9.301280e-01 Iteration = 44 Scaled Residual: 5.971460e-03 Max error: 1.000000e+00 tot_error: 9.285241e-01 Iteration = 45 Scaled Residual: 5.838857e-03 Max error: 1.000000e+00 tot_error: 9.269209e-01 Iteration = 46 Scaled Residual: 5.714529e-03 Max error: 1.000000e+00 tot_error: 9.253185e-01 Iteration = 47 Scaled Residual: 5.598161e-03 Max error: 1.000000e+00 tot_error: 9.237170e-01 Iteration = 48 Scaled Residual: 5.488699e-03 Max error: 1.000000e+00 tot_error: 9.221164e-01 Iteration = 49 Scaled Residual: 5.385787e-03 Max error: 1.000000e+00 tot_error: 9.205168e-01 Iteration = 50 Scaled Residual: 5.288806e-03 Max error: 1.000000e+00 tot_error: 9.189185e-01 Iteration = 51 Scaled Residual: 5.196436e-03 Max error: 1.000000e+00 tot_error: 9.173214e-01 Iteration = 52 Scaled Residual: 5.107589e-03 Max error: 1.000000e+00 tot_error: 9.157255e-01 Iteration = 53 Scaled Residual: 5.020771e-03 Max error: 1.000000e+00 tot_error: 9.141309e-01 Iteration = 54 Scaled Residual: 4.934569e-03 Max error: 1.000000e+00 tot_error: 9.125375e-01 Starting Benchmarking Phase... Performing 502 CG sets expected time: 1900.0 seconds expected Perf: 407657.9 GF (364.0 GF_per) 2021-05-13 22:22:24.562 progress = 1.2% 22.0 / 1900.0 sec elapsed 1878.0 sec remain 421795.952 GF 376.604 GF_per progress = 2.3% 43.8 / 1900.0 sec elapsed 1856.2 sec remain 422546.739 GF 377.274 GF_per progress = 3.5% 65.7 / 1900.0 sec elapsed 1834.3 sec remain 423127.061 GF 377.792 GF_per progress = 4.6% 87.5 / 1900.0 sec elapsed 1812.5 sec remain 423484.521 GF 378.111 GF_per progress = 5.8% 109.3 / 1900.0 sec elapsed 1790.7 sec remain 423633.210 GF 378.244 GF_per progress = 6.9% 131.2 / 1900.0 sec elapsed 1768.8 sec remain 423679.351 GF 378.285 GF_per progress = 8.1% 153.0 / 1900.0 sec elapsed 1747.0 sec remain 423801.961 GF 378.395 GF_per progress = 9.2% 174.8 / 1900.0 sec elapsed 1725.2 sec remain 423900.069 GF 378.482 GF_per progress = 10.3% 196.6 / 1900.0 sec elapsed 1703.4 sec remain 423944.484 GF 378.522 GF_per progress = 11.5% 218.4 / 1900.0 sec elapsed 1681.6 sec remain 424013.240 GF 378.583 GF_per progress = 12.6% 240.3 / 1900.0 sec elapsed 1659.7 sec remain 424064.112 GF 378.629 GF_per progress = 13.8% 262.1 / 1900.0 sec elapsed 1637.9 sec remain 424101.412 GF 378.662 GF_per progress = 14.9% 283.9 / 1900.0 sec elapsed 1616.1 sec remain 424122.231 GF 378.681 GF_per progress = 16.1% 305.7 / 1900.0 sec elapsed 1594.3 sec remain 424158.165 GF 378.713 GF_per progress = 17.2% 327.5 / 1900.0 sec elapsed 1572.5 sec remain 424168.413 GF 378.722 GF_per progress = 18.4% 349.4 / 1900.0 sec elapsed 1550.6 sec remain 424190.223 GF 378.741 GF_per progress = 19.5% 371.2 / 1900.0 sec elapsed 1528.8 sec remain 424209.883 GF 378.759 GF_per progress = 20.7% 393.0 / 1900.0 sec elapsed 1507.0 sec remain 424199.840 GF 378.750 GF_per progress = 21.8% 414.8 / 1900.0 sec elapsed 1485.2 sec remain 424215.119 GF 378.763 GF_per progress = 23.0% 436.7 / 1900.0 sec elapsed 1463.3 sec remain 424233.052 GF 378.780 GF_per progress = 24.1% 458.5 / 1900.0 sec elapsed 1441.5 sec remain 424250.663 GF 378.795 GF_per progress = 25.3% 480.3 / 1900.0 sec elapsed 1419.7 sec remain 424260.055 GF 378.804 GF_per progress = 26.4% 502.1 / 1900.0 sec elapsed 1397.9 sec remain 424264.197 GF 378.807 GF_per progress = 27.6% 523.9 / 1900.0 sec elapsed 1376.1 sec remain 424275.778 GF 378.818 GF_per progress = 28.7% 545.8 / 1900.0 sec elapsed 1354.2 sec remain 424274.228 GF 378.816 GF_per progress = 29.9% 567.6 / 1900.0 sec elapsed 1332.4 sec remain 424278.981 GF 378.821 GF_per progress = 31.0% 589.4 / 1900.0 sec elapsed 1310.6 sec remain 424289.602 GF 378.830 GF_per progress = 32.2% 611.2 / 1900.0 sec elapsed 1288.8 sec remain 424299.424 GF 378.839 GF_per progress = 33.3% 633.0 / 1900.0 sec elapsed 1267.0 sec remain 424317.889 GF 378.855 GF_per progress = 34.5% 654.8 / 1900.0 sec elapsed 1245.2 sec remain 424327.164 GF 378.864 GF_per progress = 35.6% 676.6 / 1900.0 sec elapsed 1223.4 sec remain 424335.928 GF 378.871 GF_per progress = 36.8% 698.5 / 1900.0 sec elapsed 1201.5 sec remain 424343.146 GF 378.878 GF_per progress = 37.9% 720.3 / 1900.0 sec elapsed 1179.7 sec remain 424349.886 GF 378.884 GF_per progress = 39.1% 742.1 / 1900.0 sec elapsed 1157.9 sec remain 424355.639 GF 378.889 GF_per progress = 40.2% 763.9 / 1900.0 sec elapsed 1136.1 sec remain 424364.126 GF 378.897 GF_per progress = 41.4% 785.7 / 1900.0 sec elapsed 1114.3 sec remain 424369.934 GF 378.902 GF_per progress = 42.5% 807.5 / 1900.0 sec elapsed 1092.5 sec remain 424370.412 GF 378.902 GF_per progress = 43.7% 829.4 / 1900.0 sec elapsed 1070.6 sec remain 424374.075 GF 378.905 GF_per progress = 44.8% 851.2 / 1900.0 sec elapsed 1048.8 sec remain 424374.595 GF 378.906 GF_per progress = 45.9% 873.0 / 1900.0 sec elapsed 1027.0 sec remain 424379.839 GF 378.911 GF_per progress = 47.1% 894.8 / 1900.0 sec elapsed 1005.2 sec remain 424380.219 GF 378.911 GF_per progress = 48.2% 916.6 / 1900.0 sec elapsed 983.4 sec remain 424383.505 GF 378.914 GF_per progress = 49.4% 938.5 / 1900.0 sec elapsed 961.5 sec remain 424386.863 GF 378.917 GF_per progress = 50.5% 960.3 / 1900.0 sec elapsed 939.7 sec remain 424391.540 GF 378.921 GF_per progress = 51.7% 982.1 / 1900.0 sec elapsed 917.9 sec remain 424393.261 GF 378.923 GF_per progress = 52.8% 1003.9 / 1900.0 sec elapsed 896.1 sec remain 424397.802 GF 378.927 GF_per progress = 54.0% 1025.7 / 1900.0 sec elapsed 874.3 sec remain 424405.247 GF 378.933 GF_per progress = 55.1% 1047.5 / 1900.0 sec elapsed 852.5 sec remain 424405.384 GF 378.933 GF_per progress = 56.3% 1069.4 / 1900.0 sec elapsed 830.6 sec remain 424404.437 GF 378.933 GF_per progress = 57.4% 1091.2 / 1900.0 sec elapsed 808.8 sec remain 424409.623 GF 378.937 GF_per progress = 58.6% 1113.0 / 1900.0 sec elapsed 787.0 sec remain 424411.008 GF 378.938 GF_per progress = 59.7% 1134.8 / 1900.0 sec elapsed 765.2 sec remain 424412.651 GF 378.940 GF_per progress = 60.9% 1156.6 / 1900.0 sec elapsed 743.4 sec remain 424415.413 GF 378.942 GF_per progress = 62.0% 1178.4 / 1900.0 sec elapsed 721.6 sec remain 424418.219 GF 378.945 GF_per progress = 63.2% 1200.3 / 1900.0 sec elapsed 699.7 sec remain 424421.967 GF 378.948 GF_per progress = 64.3% 1222.1 / 1900.0 sec elapsed 677.9 sec remain 424423.473 GF 378.950 GF_per progress = 65.5% 1243.9 / 1900.0 sec elapsed 656.1 sec remain 424425.031 GF 378.951 GF_per progress = 66.6% 1265.7 / 1900.0 sec elapsed 634.3 sec remain 424425.652 GF 378.951 GF_per progress = 67.8% 1287.5 / 1900.0 sec elapsed 612.5 sec remain 424421.631 GF 378.948 GF_per progress = 68.9% 1309.4 / 1900.0 sec elapsed 590.6 sec remain 424419.867 GF 378.946 GF_per progress = 70.1% 1331.2 / 1900.0 sec elapsed 568.8 sec remain 424419.676 GF 378.946 GF_per progress = 71.2% 1353.0 / 1900.0 sec elapsed 547.0 sec remain 424419.194 GF 378.946 GF_per progress = 72.4% 1374.8 / 1900.0 sec elapsed 525.2 sec remain 424418.674 GF 378.945 GF_per progress = 73.5% 1396.7 / 1900.0 sec elapsed 503.3 sec remain 424419.723 GF 378.946 GF_per progress = 74.7% 1418.5 / 1900.0 sec elapsed 481.5 sec remain 424420.034 GF 378.946 GF_per progress = 75.8% 1440.3 / 1900.0 sec elapsed 459.7 sec remain 424420.407 GF 378.947 GF_per progress = 77.0% 1462.1 / 1900.0 sec elapsed 437.9 sec remain 424420.325 GF 378.947 GF_per progress = 78.1% 1484.0 / 1900.0 sec elapsed 416.0 sec remain 424421.059 GF 378.947 GF_per progress = 79.3% 1505.8 / 1900.0 sec elapsed 394.2 sec remain 424422.431 GF 378.949 GF_per progress = 80.4% 1527.6 / 1900.0 sec elapsed 372.4 sec remain 424424.432 GF 378.950 GF_per progress = 81.5% 1549.4 / 1900.0 sec elapsed 350.6 sec remain 424423.924 GF 378.950 GF_per progress = 82.7% 1571.2 / 1900.0 sec elapsed 328.8 sec remain 424423.916 GF 378.950 GF_per progress = 83.8% 1593.1 / 1900.0 sec elapsed 306.9 sec remain 424423.643 GF 378.950 GF_per progress = 85.0% 1614.9 / 1900.0 sec elapsed 285.1 sec remain 424426.131 GF 378.952 GF_per progress = 86.1% 1636.7 / 1900.0 sec elapsed 263.3 sec remain 424426.830 GF 378.953 GF_per progress = 87.3% 1658.5 / 1900.0 sec elapsed 241.5 sec remain 424428.323 GF 378.954 GF_per progress = 88.4% 1680.3 / 1900.0 sec elapsed 219.7 sec remain 424427.261 GF 378.953 GF_per progress = 89.6% 1702.2 / 1900.0 sec elapsed 197.8 sec remain 424428.829 GF 378.954 GF_per progress = 90.7% 1724.0 / 1900.0 sec elapsed 176.0 sec remain 424430.771 GF 378.956 GF_per progress = 91.9% 1745.8 / 1900.0 sec elapsed 154.2 sec remain 424430.344 GF 378.956 GF_per progress = 93.0% 1767.6 / 1900.0 sec elapsed 132.4 sec remain 424430.029 GF 378.955 GF_per progress = 94.2% 1789.4 / 1900.0 sec elapsed 110.6 sec remain 424431.397 GF 378.957 GF_per progress = 95.3% 1811.2 / 1900.0 sec elapsed 88.8 sec remain 424432.313 GF 378.957 GF_per Completed Benchmarking Phase... elapsed time: 1825.8 seconds 2021-05-13 22:52:50.525 Number of CG sets: 502 Iterations per set: 54 scaled res mean: 4.934569e-03 scaled res variance: 0.000000e+00 Total Time: 1.825788e+03 sec Setup Overhead: 1.32% Optimization Overhead: 1.37% Convergence Overhead: 7.41% 8x10x14 process grid 512x512x288 local domain SpMV = 321668.5 GF (2025364.1 GB/s Effective) 287.2 GF_per (1808.4 GB/s Effective) SymGS = 557639.0 GF (4303584.4 GB/s Effective) 497.9 GF_per (3842.5 GB/s Effective) total = 470873.9 GF (3570493.2 GB/s Effective) 420.4 GF_per (3187.9 GB/s Effective) final = 424434.4 GF (3218356.5 GB/s Effective) 379.0 GF_per (2873.5 GB/s Effective) end of application... 2021-05-13 22:52:51.496