Feature/remove salmon input rewrite

docs/guides/salmon-genoa-fx700-handoff.md

@@ -0,0 +1,121 @@
+# SALMON GENOA / FX700 Handoff
+
+This note is for an on-site agent continuing the SALMON integration work.
+
+## Scope and rules
+
+- Treat GENOA and FX700 as separate platform configurations. Do not infer MPI or network behavior from the CPU/compiler family.
+- Keep platform-specific settings in `programs/salmon/build.sh` and `programs/salmon/run.sh`.
+- Do not change the SALMON input archive or rewrite `theory` values unless the site owner confirms that the archive is wrong.
+- Do not mark a run successful only because the launcher returned zero. Check the application output and the GS-to-RT restart state.
+- Use WSL Ubuntu-22.04 for repository checks: `bash -n`, `shellcheck`, `git diff`, and patch validation. Use the target site for module, compiler, MPI, and runtime checks.
+- Do not remove existing platform branches or comments without confirming the platform-specific behavior with the site owner.
+- Commits must use `git commit --signoff` and explain the platform-specific reason. Do not push unrelated temporary files or generated SALMON sources.
+
+## Current repository state
+
+The current branch contains:
+
+- `b35f4fd`: SALMON Fujitsu topology guard patch and GENOA OpenMPI selection.
+- `5f99c5d`: GENOA AOCL utility dependency detection and linking.
+
+The untracked `.tmp-salmon-v222-check/` directory is only a local validation checkout and must not be committed.
+
+## GENOA
+
+### Expected platform configuration
+
+BenchPark's GENOA definition uses:
+
+- module: `system/genoa mpi/openmpi-x86_64`
+- MPI: OpenMPI, currently `/usr/lib64/openmpi`
+- compiler: GCC
+- BLAS/LAPACK baseline: OpenBLAS
+- scheduler setting: `SLURM_MPI_TYPE=pmix`
+
+The previous GENOA failure used `/usr/lib64/mpich/bin/mpicc` and `mpif90`; that was a configuration mismatch. The build configuration has been changed to OpenMPI.
+
+### AOCL configuration
+
+The site AOCL installation is expected under:
+
+```text
+/lvs0/rccs-nghpcadu/nakamura/aocl/install/
+```
+
+Relevant libraries are:
+
+```text
+amd-blis/lib/LP64/libblis-mt.so
+amd-libflame/lib/LP64/libflame.so
+amd-utils/lib/libaoclutils.so
+amd-utils/lib/libau_cpuid.so
+```
+
+The build script searches the AOCL root for the two utils libraries, adds their directory to `LD_LIBRARY_PATH`, and passes them with BLIS and libFLAME. The earlier linker failure was caused by missing `libaoclutils.so` and `au_cpuid_has_flags`.
+
+### Important pending check
+
+Verify `programs/salmon/run.sh` uses the same OpenMPI module as `build.sh`. A runtime branch still needs to be checked for an old MPICH module reference. Build and run must use the same MPI family.
+
+### GENOA completion criteria
+
+- Configure reports `/usr/lib64/openmpi/bin/mpicc` and `/usr/lib64/openmpi/bin/mpif90`.
+- AOCL configure/link output contains BLIS, libFLAME, `libaoclutils.so`, and `libau_cpuid.so`.
+- SALMON builds without undefined AOCL symbols.
+- GS and RT both complete with finite values and valid restart files.
+- The result contains valid `gs` and `rt` sections.
+
+## FX700
+
+### Expected platform configuration
+
+- module: `system/fx700 FJSVstclanga`
+- launcher/compiler: Fujitsu MPI wrappers `mpifcc` / `mpifrt`
+- run shape for the current test: `1 MPI process x 48 OpenMP threads`
+- `SLURM_MPI_TYPE=pmix`
+- BLAS/LAPACK: `-SSL2BLAMP`
+
+The Fujitsu topology patch is applied to SALMON v2.2.2, but `USE_FJMPI=OFF` is used for FX700. The patch separates the Fujitsu compiler macro from the Tofu-specific topology path. Do not turn `USE_FJMPI` on for FX700 without site confirmation.
+
+### Observed FX700 behavior
+
+- Build succeeds with `1 MPI x 48 threads`.
+- GS reaches `end SALMON`.
+- RT starts but reports `rbox1=0` and `Ne=NaN` at the first time step.
+- `run.sh` may also report a missing success marker because the output uses `end SALMON` rather than the current elapsed-time pattern.
+
+Do not solve this by accepting `end SALMON` alone. First establish why the RT restart/input state is invalid. Compare the input namelist, `restart/` files, and GS-to-RT handoff with a successful GH200 run.
+
+### FX700 investigation sequence
+
+1. Confirm the checkout contains the current branch commit and the Fujitsu topology patch.
+2. Confirm the build uses `mpifcc` / `mpifrt` from `FJSVstclanga`.
+3. Run GS and record the exit status, final output, and all generated restart files with sizes.
+4. Confirm `data_for_restart` is moved to `restart` before RT.
+5. Compare `Si-1-1-1-tddft.nml`, restart filenames, and working-directory contents with GH200.
+6. Run RT separately if needed and inspect the first occurrence of `NaN`, `rbox1`, and `Ne`.
+7. Only after valid GS/RT output exists, adjust the success-marker logic to recognize the platform's actual completion marker.
+
+## Required evidence for a change
+
+Record the following in the handoff or pull request:
+
+- system and queue
+- module list and `which mpicc mpif90 mpiexec`
+- MPI/compiler versions
+- SALMON source tag and commit
+- node/process/thread shape
+- relevant configure summary
+- relevant build or run error
+- GS/RT output and restart file listing
+- whether the result sections were emitted
+
+Run repository checks in WSL before committing:
+
+```bash
+bash -n programs/salmon/build.sh programs/salmon/run.sh
+shellcheck -S error programs/salmon/build.sh programs/salmon/run.sh
+```
+
+Do not combine a GENOA MPI/AOCL fix with an FX700 runtime fix in one commit unless both are independently verified.

programs/salmon/build.sh

@@ -8,13 +8,26 @@ REPO_DIR="SALMON2"
 VERSION_TAG="v.2.2.2"
 BUILD_DIR="build-benchkit"
 ARTIFACT_DIR="${PWD}/artifacts"
+RESULTS_DIR="${PWD}/results"
+BUILD_LOG_DIR="${RESULTS_DIR}/salmon_build_logs"
+AOCL_ROOT_DEFAULT="/lvs0/rccs-nghpcadu/nakamura/aocl/install"
+FJMPI_PATCH="${PWD}/programs/salmon/patches/fjmpi-topology-guard.patch"
 
 source scripts/bk_functions.sh
 
 mkdir -p "${ARTIFACT_DIR}"
+mkdir -p "${BUILD_LOG_DIR}"
 bk_fetch_source "${REPO_URL}" "${REPO_DIR}" "${VERSION_TAG}"
 
 cd "${REPO_DIR}"
+if git apply --check "${FJMPI_PATCH}"; then
+  git apply "${FJMPI_PATCH}"
+elif git apply --reverse --check "${FJMPI_PATCH}" >/dev/null 2>&1; then
+  echo "SALMON Fujitsu MPI topology patch is already applied"
+else
+  echo "SALMON Fujitsu MPI topology patch does not apply to ${VERSION_TAG}" >&2
+  exit 1
+fi
 rm -rf "${BUILD_DIR}"
 mkdir -p "${BUILD_DIR}"
 
@@ -24,22 +37,47 @@ common_cmake_args=(
   -DUSE_EIGENEXA=OFF
   -DUSE_LIBXC=OFF
   -DCMAKE_BUILD_TYPE=None
-  -DCMAKE_VERBOSE_MAKEFILE=ON
+  -DCMAKE_VERBOSE_MAKEFILE=OFF
 )
 
+print_log_summary() {
+  local logfile="$1"
+
+  grep -E "Build Netlib LAPACK|Set vendor-specific LAPACK|Found (BLAS|LAPACK|OpenMP|MPI)|Downloading|Built target salmon|Linking Fortran executable|error:|ERROR" "${logfile}" \
+    | tail -n 80 || true
+}
+
+run_logged() {
+  local label="$1"
+  local logfile="$2"
+  shift 2
+
+  echo "${label}; full log: ${logfile}"
+  if "$@" > "${logfile}" 2>&1; then
+    print_log_summary "${logfile}"
+    return 0
+  fi
+
+  echo "${label} failed; full log: ${logfile}" >&2
+  echo "---- ${logfile} tail ----" >&2
+  tail -n 160 "${logfile}" >&2 || true
+  exit 1
+}
+
 case "${system}" in
   Fugaku)
     cmake_args=(
       "${common_cmake_args[@]}"
       -DCMAKE_Fortran_COMPILER=mpifrtpx
       -DCMAKE_C_COMPILER=mpifccpx
-      -DCMAKE_Fortran_FLAGS="-Kfast -Kocl -Nlst=t -Koptmsg=2 -Ncheck_std=03s"
-      -DCMAKE_C_FLAGS="-Kfast -Kocl -Nlst=t -Koptmsg=2 -Xg -std=gnu99"
+      -DCMAKE_Fortran_FLAGS="-Kfast -Kocl -Ncheck_std=03s -Nalloc_assign"
+      -DCMAKE_C_FLAGS="-Kfast -Kocl -Xg -std=gnu99"
       -DCMAKE_C_STANDARD_COMPUTED_DEFAULT=Fujitsu
       "-DCMAKE_Fortran_MODDIR_FLAG=-M "
       -DOPENMP_FLAGS="-Kopenmp -Nfjomplib"
       -DLAPACK_VENDOR_FLAGS=-SSL2BLAMP
       -DFortran_PP_FLAGS=-Cpp
+      -DUSE_FJMPI=ON
     )
     ;;
   RC_GH200)
@@ -81,39 +119,64 @@ case "${system}" in
     ;;
   RC_GENOA)
     module purge
-    module load system/genoa mpi/mpich-x86_64
+    module load system/genoa mpi/openmpi-x86_64
+    aocl_root="${BK_SALMON_AOCL_ROOT:-${AOCL_ROOT_DEFAULT}}"
+    aocl_blis_lib="${aocl_root}/amd-blis/lib/LP64"
+    aocl_flame_lib="${aocl_root}/amd-libflame/lib/LP64"
+    aocl_utils_lib=""
+    aocl_utils_so="$(find "${aocl_root}" -type f -name libaoclutils.so -print -quit 2>/dev/null || true)"
+    aocl_cpuid_so="$(find "${aocl_root}" -type f -name libau_cpuid.so -print -quit 2>/dev/null || true)"
+    if [[ -n "${aocl_utils_so}" ]]; then
+      aocl_utils_lib="$(dirname "${aocl_utils_so}")"
+    fi
     cmake_args=(
       "${common_cmake_args[@]}"
       -DCMAKE_Fortran_COMPILER=mpif90
       -DCMAKE_C_COMPILER=mpicc
       -DCMAKE_Fortran_FLAGS="-O3 -ffree-line-length-none -fallow-argument-mismatch"
       -DCMAKE_C_FLAGS=-O3
     )
+    if [[ -f "${aocl_blis_lib}/libblis-mt.so" && -f "${aocl_flame_lib}/libflame.so" && -n "${aocl_utils_so}" && -n "${aocl_cpuid_so}" ]]; then
+      export LD_LIBRARY_PATH="${aocl_utils_lib}:${aocl_flame_lib}:${aocl_blis_lib}:${LD_LIBRARY_PATH:-}"
+      cmake_args+=("-DLAPACK_VENDOR_FLAGS=${aocl_flame_lib}/libflame.so ${aocl_blis_lib}/libblis-mt.so ${aocl_utils_so} ${aocl_cpuid_so}")
+    elif command -v pkg-config >/dev/null 2>&1 && pkg-config --exists openblas; then
+      cmake_args+=("-DLAPACK_VENDOR_FLAGS=$(pkg-config --libs openblas)")
+    elif ldconfig -p 2>/dev/null | grep -q 'libopenblas\.so'; then
+      cmake_args+=(-DLAPACK_VENDOR_FLAGS=-lopenblas)
+    else
+      echo "System OpenBLAS/LAPACK not found; SALMON may build bundled Netlib LAPACK." >&2
+    fi
     ;;
-  RC_FX700)
-    module purge
-    module load system/fx700 FJSVstclanga
-    cmake_args=(
-      "${common_cmake_args[@]}"
-      -DCMAKE_Fortran_COMPILER=mpifrt
-      -DCMAKE_C_COMPILER=mpifcc
-      -DCMAKE_Fortran_FLAGS="-Kfast -Kocl -Nlst=t -Koptmsg=2 -Ncheck_std=03s"
-      -DCMAKE_C_FLAGS="-Kfast -Kocl -Nlst=t -Koptmsg=2 -Xg -std=gnu99"
-      -DCMAKE_C_STANDARD_COMPUTED_DEFAULT=Fujitsu
-      "-DCMAKE_Fortran_MODDIR_FLAG=-M "
-      -DOPENMP_FLAGS="-Kopenmp -Nfjomplib"
-      -DLAPACK_VENDOR_FLAGS=-SSL2BLAMP
-      -DFortran_PP_FLAGS=-Cpp
-    )
-    ;;
+  # RC_FX700)
+  #   FX700 currently fails during GS initialization even with the Fujitsu
+  #   topology guard patch applied. Keep this route disabled until verified.
+  #   module purge
+  #   module load system/fx700 FJSVstclanga
+  #   export SLURM_MPI_TYPE=pmix
+  #   cmake_args=(
+  #     "${common_cmake_args[@]}"
+  #     -DCMAKE_Fortran_COMPILER=mpifrt
+  #     -DCMAKE_C_COMPILER=mpifcc
+  #     -DCMAKE_Fortran_FLAGS="-Kfast -Kocl -Ncheck_std=03s -Nalloc_assign"
+  #     -DCMAKE_C_FLAGS="-Kfast -Kocl -Xg -std=gnu99"
+  #     -DCMAKE_C_STANDARD_COMPUTED_DEFAULT=Fujitsu
+  #     "-DCMAKE_Fortran_MODDIR_FLAG=-M "
+  #     -DOPENMP_FLAGS="-Kopenmp -Nfjomplib"
+  #     -DLAPACK_VENDOR_FLAGS=-SSL2BLAMP
+  #     -DFortran_PP_FLAGS=-Cpp
+  #     -DUSE_FJMPI=OFF
+  #   )
+  #   ;;
   *)
     echo "Unknown system: ${system}" >&2
     exit 1
     ;;
 esac
 
-cmake -S . -B "${BUILD_DIR}" "${cmake_args[@]}"
-cmake --build "${BUILD_DIR}" --target salmon --parallel 8
+run_logged "Configuring SALMON" "${BUILD_LOG_DIR}/${system}_configure.log" \
+  cmake -Wno-dev -S . -B "${BUILD_DIR}" "${cmake_args[@]}"
+run_logged "Building SALMON" "${BUILD_LOG_DIR}/${system}_build.log" \
+  cmake --build "${BUILD_DIR}" --target salmon --parallel 8
 
 salmon_bin=$(find "${BUILD_DIR}" -type f -name salmon -perm -u+x | head -n 1)
 if [[ -z "${salmon_bin}" ]]; then

programs/salmon/list.csv

@@ -1,6 +1,6 @@
 system,enable,nodes,numproc_node,nthreads,elapse
 Fugaku,yes,1,4,12,0:10:00
 RC_GH200,yes,1,1,1,0:10:00
-RC_DGXSP,yes,1,1,1,0:10:00
-RC_GENOA,yes,1,1,48,0:10:00
-RC_FX700,yes,1,1,48,0:30:00
+RC_DGXSP,yes,1,1,1,1:00:00
+RC_GENOA,yes,1,1,48,2:00:00
+RC_FX700,no,1,1,48,0:30:00

programs/salmon/patches/fjmpi-topology-guard.patch

@@ -0,0 +1,58 @@
+diff --git a/CMakeLists.txt b/CMakeLists.txt
+--- a/CMakeLists.txt
++++ b/CMakeLists.txt
+@@ -43,6 +43,7 @@ option_set(USE_SCALAPACK "Use ScaLAPACK Library"   OFF)
+ option_set(USE_EIGENEXA  "Use EigenExa Library"    OFF)
+ option_set(USE_LIBXC     "Use Libxc library"       OFF)
+ option_set(USE_FFTW      "Use FFTW library"        OFF)
++option_set(USE_FJMPI     "Use FUJITSU MPI"         OFF)
+
+ ## Optimization for stencil compitations
+ option_set(USE_OPT_ARRAY_PADDING           "Enable array padding for the stencil"          ON)
+diff --git a/src/config.h.in b/src/config.h.in
+--- a/src/config.h.in
++++ b/src/config.h.in
+@@ -15,6 +15,7 @@
+ #cmakedefine SYSTEM_HAS_PATH_MAX_IN_LINUX_LIMITS_H
+
+ #cmakedefine USE_MPI
++#cmakedefine USE_FJMPI
+ #cmakedefine USE_SCALAPACK
+ #cmakedefine USE_EIGENEXA
+ #cmakedefine USE_LIBXC
+diff --git a/src/parallel/init_communicator.f90 b/src/parallel/init_communicator.f90
+--- a/src/parallel/init_communicator.f90
++++ b/src/parallel/init_communicator.f90
+@@ -14,4 +14,5 @@
+ !  limitations under the License.
+ !
++#include "config.h"
+ MODULE init_communicator
+   implicit none
+@@ -38,6 +39,6 @@ subroutine init_communicator_dft(info)
+   integer :: i1,i2,i3,i4,i5,ix,iy,iz,nl,io2,io3,io4
+   integer,allocatable :: iranklists(:)
+
+-#ifdef __FUJITSU
++#ifdef USE_FJMPI
+   integer :: iret
+ #endif
+@@ -62,5 +63,5 @@ subroutine init_communicator_dft(info)
+                      0:nproc_k-1))
+
+ ! communicator r,o,k,ro,ko
+-#ifdef __FUJITSU
++#ifdef USE_FJMPI
+   call tofu_network_oriented_mapping(iret)
+@@ -109,4 +110,4 @@ subroutine init_communicator_dft(info)
+     if (nl /= info%isize_rko-1) &
+       stop '[FATAL ERROR] init_communicator_dft'
+-#ifdef __FUJITSU
++#ifdef USE_FJMPI
+   end if
+@@ -460,4 +461,4 @@
+ #endif
+
+-#ifdef __FUJITSU
++#ifdef USE_FJMPI
+ contains