diff --git a/docs/guides/salmon-genoa-fx700-handoff.md b/docs/guides/salmon-genoa-fx700-handoff.md new file mode 100644 index 0000000..49f38e0 --- /dev/null +++ b/docs/guides/salmon-genoa-fx700-handoff.md @@ -0,0 +1,121 @@ +# SALMON GENOA / FX700 Handoff + +This note is for an on-site agent continuing the SALMON integration work. + +## Scope and rules + +- Treat GENOA and FX700 as separate platform configurations. Do not infer MPI or network behavior from the CPU/compiler family. +- Keep platform-specific settings in `programs/salmon/build.sh` and `programs/salmon/run.sh`. +- Do not change the SALMON input archive or rewrite `theory` values unless the site owner confirms that the archive is wrong. +- Do not mark a run successful only because the launcher returned zero. Check the application output and the GS-to-RT restart state. +- Use WSL Ubuntu-22.04 for repository checks: `bash -n`, `shellcheck`, `git diff`, and patch validation. Use the target site for module, compiler, MPI, and runtime checks. +- Do not remove existing platform branches or comments without confirming the platform-specific behavior with the site owner. +- Commits must use `git commit --signoff` and explain the platform-specific reason. Do not push unrelated temporary files or generated SALMON sources. + +## Current repository state + +The current branch contains: + +- `b35f4fd`: SALMON Fujitsu topology guard patch and GENOA OpenMPI selection. +- `5f99c5d`: GENOA AOCL utility dependency detection and linking. + +The untracked `.tmp-salmon-v222-check/` directory is only a local validation checkout and must not be committed. + +## GENOA + +### Expected platform configuration + +BenchPark's GENOA definition uses: + +- module: `system/genoa mpi/openmpi-x86_64` +- MPI: OpenMPI, currently `/usr/lib64/openmpi` +- compiler: GCC +- BLAS/LAPACK baseline: OpenBLAS +- scheduler setting: `SLURM_MPI_TYPE=pmix` + +The previous GENOA failure used `/usr/lib64/mpich/bin/mpicc` and `mpif90`; that was a configuration mismatch. The build configuration has been changed to OpenMPI. + +### AOCL configuration + +The site AOCL installation is expected under: + +```text +/lvs0/rccs-nghpcadu/nakamura/aocl/install/ +``` + +Relevant libraries are: + +```text +amd-blis/lib/LP64/libblis-mt.so +amd-libflame/lib/LP64/libflame.so +amd-utils/lib/libaoclutils.so +amd-utils/lib/libau_cpuid.so +``` + +The build script searches the AOCL root for the two utils libraries, adds their directory to `LD_LIBRARY_PATH`, and passes them with BLIS and libFLAME. The earlier linker failure was caused by missing `libaoclutils.so` and `au_cpuid_has_flags`. + +### Important pending check + +Verify `programs/salmon/run.sh` uses the same OpenMPI module as `build.sh`. A runtime branch still needs to be checked for an old MPICH module reference. Build and run must use the same MPI family. + +### GENOA completion criteria + +- Configure reports `/usr/lib64/openmpi/bin/mpicc` and `/usr/lib64/openmpi/bin/mpif90`. +- AOCL configure/link output contains BLIS, libFLAME, `libaoclutils.so`, and `libau_cpuid.so`. +- SALMON builds without undefined AOCL symbols. +- GS and RT both complete with finite values and valid restart files. +- The result contains valid `gs` and `rt` sections. + +## FX700 + +### Expected platform configuration + +- module: `system/fx700 FJSVstclanga` +- launcher/compiler: Fujitsu MPI wrappers `mpifcc` / `mpifrt` +- run shape for the current test: `1 MPI process x 48 OpenMP threads` +- `SLURM_MPI_TYPE=pmix` +- BLAS/LAPACK: `-SSL2BLAMP` + +The Fujitsu topology patch is applied to SALMON v2.2.2, but `USE_FJMPI=OFF` is used for FX700. The patch separates the Fujitsu compiler macro from the Tofu-specific topology path. Do not turn `USE_FJMPI` on for FX700 without site confirmation. + +### Observed FX700 behavior + +- Build succeeds with `1 MPI x 48 threads`. +- GS reaches `end SALMON`. +- RT starts but reports `rbox1=0` and `Ne=NaN` at the first time step. +- `run.sh` may also report a missing success marker because the output uses `end SALMON` rather than the current elapsed-time pattern. + +Do not solve this by accepting `end SALMON` alone. First establish why the RT restart/input state is invalid. Compare the input namelist, `restart/` files, and GS-to-RT handoff with a successful GH200 run. + +### FX700 investigation sequence + +1. Confirm the checkout contains the current branch commit and the Fujitsu topology patch. +2. Confirm the build uses `mpifcc` / `mpifrt` from `FJSVstclanga`. +3. Run GS and record the exit status, final output, and all generated restart files with sizes. +4. Confirm `data_for_restart` is moved to `restart` before RT. +5. Compare `Si-1-1-1-tddft.nml`, restart filenames, and working-directory contents with GH200. +6. Run RT separately if needed and inspect the first occurrence of `NaN`, `rbox1`, and `Ne`. +7. Only after valid GS/RT output exists, adjust the success-marker logic to recognize the platform's actual completion marker. + +## Required evidence for a change + +Record the following in the handoff or pull request: + +- system and queue +- module list and `which mpicc mpif90 mpiexec` +- MPI/compiler versions +- SALMON source tag and commit +- node/process/thread shape +- relevant configure summary +- relevant build or run error +- GS/RT output and restart file listing +- whether the result sections were emitted + +Run repository checks in WSL before committing: + +```bash +bash -n programs/salmon/build.sh programs/salmon/run.sh +shellcheck -S error programs/salmon/build.sh programs/salmon/run.sh +``` + +Do not combine a GENOA MPI/AOCL fix with an FX700 runtime fix in one commit unless both are independently verified. diff --git a/programs/salmon/build.sh b/programs/salmon/build.sh index ded1232..793cf0a 100644 --- a/programs/salmon/build.sh +++ b/programs/salmon/build.sh @@ -8,13 +8,26 @@ REPO_DIR="SALMON2" VERSION_TAG="v.2.2.2" BUILD_DIR="build-benchkit" ARTIFACT_DIR="${PWD}/artifacts" +RESULTS_DIR="${PWD}/results" +BUILD_LOG_DIR="${RESULTS_DIR}/salmon_build_logs" +AOCL_ROOT_DEFAULT="/lvs0/rccs-nghpcadu/nakamura/aocl/install" +FJMPI_PATCH="${PWD}/programs/salmon/patches/fjmpi-topology-guard.patch" source scripts/bk_functions.sh mkdir -p "${ARTIFACT_DIR}" +mkdir -p "${BUILD_LOG_DIR}" bk_fetch_source "${REPO_URL}" "${REPO_DIR}" "${VERSION_TAG}" cd "${REPO_DIR}" +if git apply --check "${FJMPI_PATCH}"; then + git apply "${FJMPI_PATCH}" +elif git apply --reverse --check "${FJMPI_PATCH}" >/dev/null 2>&1; then + echo "SALMON Fujitsu MPI topology patch is already applied" +else + echo "SALMON Fujitsu MPI topology patch does not apply to ${VERSION_TAG}" >&2 + exit 1 +fi rm -rf "${BUILD_DIR}" mkdir -p "${BUILD_DIR}" @@ -24,22 +37,47 @@ common_cmake_args=( -DUSE_EIGENEXA=OFF -DUSE_LIBXC=OFF -DCMAKE_BUILD_TYPE=None - -DCMAKE_VERBOSE_MAKEFILE=ON + -DCMAKE_VERBOSE_MAKEFILE=OFF ) +print_log_summary() { + local logfile="$1" + + grep -E "Build Netlib LAPACK|Set vendor-specific LAPACK|Found (BLAS|LAPACK|OpenMP|MPI)|Downloading|Built target salmon|Linking Fortran executable|error:|ERROR" "${logfile}" \ + | tail -n 80 || true +} + +run_logged() { + local label="$1" + local logfile="$2" + shift 2 + + echo "${label}; full log: ${logfile}" + if "$@" > "${logfile}" 2>&1; then + print_log_summary "${logfile}" + return 0 + fi + + echo "${label} failed; full log: ${logfile}" >&2 + echo "---- ${logfile} tail ----" >&2 + tail -n 160 "${logfile}" >&2 || true + exit 1 +} + case "${system}" in Fugaku) cmake_args=( "${common_cmake_args[@]}" -DCMAKE_Fortran_COMPILER=mpifrtpx -DCMAKE_C_COMPILER=mpifccpx - -DCMAKE_Fortran_FLAGS="-Kfast -Kocl -Nlst=t -Koptmsg=2 -Ncheck_std=03s" - -DCMAKE_C_FLAGS="-Kfast -Kocl -Nlst=t -Koptmsg=2 -Xg -std=gnu99" + -DCMAKE_Fortran_FLAGS="-Kfast -Kocl -Ncheck_std=03s -Nalloc_assign" + -DCMAKE_C_FLAGS="-Kfast -Kocl -Xg -std=gnu99" -DCMAKE_C_STANDARD_COMPUTED_DEFAULT=Fujitsu "-DCMAKE_Fortran_MODDIR_FLAG=-M " -DOPENMP_FLAGS="-Kopenmp -Nfjomplib" -DLAPACK_VENDOR_FLAGS=-SSL2BLAMP -DFortran_PP_FLAGS=-Cpp + -DUSE_FJMPI=ON ) ;; RC_GH200) @@ -81,7 +119,16 @@ case "${system}" in ;; RC_GENOA) module purge - module load system/genoa mpi/mpich-x86_64 + module load system/genoa mpi/openmpi-x86_64 + aocl_root="${BK_SALMON_AOCL_ROOT:-${AOCL_ROOT_DEFAULT}}" + aocl_blis_lib="${aocl_root}/amd-blis/lib/LP64" + aocl_flame_lib="${aocl_root}/amd-libflame/lib/LP64" + aocl_utils_lib="" + aocl_utils_so="$(find "${aocl_root}" -type f -name libaoclutils.so -print -quit 2>/dev/null || true)" + aocl_cpuid_so="$(find "${aocl_root}" -type f -name libau_cpuid.so -print -quit 2>/dev/null || true)" + if [[ -n "${aocl_utils_so}" ]]; then + aocl_utils_lib="$(dirname "${aocl_utils_so}")" + fi cmake_args=( "${common_cmake_args[@]}" -DCMAKE_Fortran_COMPILER=mpif90 @@ -89,31 +136,47 @@ case "${system}" in -DCMAKE_Fortran_FLAGS="-O3 -ffree-line-length-none -fallow-argument-mismatch" -DCMAKE_C_FLAGS=-O3 ) + if [[ -f "${aocl_blis_lib}/libblis-mt.so" && -f "${aocl_flame_lib}/libflame.so" && -n "${aocl_utils_so}" && -n "${aocl_cpuid_so}" ]]; then + export LD_LIBRARY_PATH="${aocl_utils_lib}:${aocl_flame_lib}:${aocl_blis_lib}:${LD_LIBRARY_PATH:-}" + cmake_args+=("-DLAPACK_VENDOR_FLAGS=${aocl_flame_lib}/libflame.so ${aocl_blis_lib}/libblis-mt.so ${aocl_utils_so} ${aocl_cpuid_so}") + elif command -v pkg-config >/dev/null 2>&1 && pkg-config --exists openblas; then + cmake_args+=("-DLAPACK_VENDOR_FLAGS=$(pkg-config --libs openblas)") + elif ldconfig -p 2>/dev/null | grep -q 'libopenblas\.so'; then + cmake_args+=(-DLAPACK_VENDOR_FLAGS=-lopenblas) + else + echo "System OpenBLAS/LAPACK not found; SALMON may build bundled Netlib LAPACK." >&2 + fi ;; - RC_FX700) - module purge - module load system/fx700 FJSVstclanga - cmake_args=( - "${common_cmake_args[@]}" - -DCMAKE_Fortran_COMPILER=mpifrt - -DCMAKE_C_COMPILER=mpifcc - -DCMAKE_Fortran_FLAGS="-Kfast -Kocl -Nlst=t -Koptmsg=2 -Ncheck_std=03s" - -DCMAKE_C_FLAGS="-Kfast -Kocl -Nlst=t -Koptmsg=2 -Xg -std=gnu99" - -DCMAKE_C_STANDARD_COMPUTED_DEFAULT=Fujitsu - "-DCMAKE_Fortran_MODDIR_FLAG=-M " - -DOPENMP_FLAGS="-Kopenmp -Nfjomplib" - -DLAPACK_VENDOR_FLAGS=-SSL2BLAMP - -DFortran_PP_FLAGS=-Cpp - ) - ;; + # RC_FX700) + # FX700 currently fails during GS initialization even with the Fujitsu + # topology guard patch applied. Keep this route disabled until verified. + # module purge + # module load system/fx700 FJSVstclanga + # export SLURM_MPI_TYPE=pmix + # cmake_args=( + # "${common_cmake_args[@]}" + # -DCMAKE_Fortran_COMPILER=mpifrt + # -DCMAKE_C_COMPILER=mpifcc + # -DCMAKE_Fortran_FLAGS="-Kfast -Kocl -Ncheck_std=03s -Nalloc_assign" + # -DCMAKE_C_FLAGS="-Kfast -Kocl -Xg -std=gnu99" + # -DCMAKE_C_STANDARD_COMPUTED_DEFAULT=Fujitsu + # "-DCMAKE_Fortran_MODDIR_FLAG=-M " + # -DOPENMP_FLAGS="-Kopenmp -Nfjomplib" + # -DLAPACK_VENDOR_FLAGS=-SSL2BLAMP + # -DFortran_PP_FLAGS=-Cpp + # -DUSE_FJMPI=OFF + # ) + # ;; *) echo "Unknown system: ${system}" >&2 exit 1 ;; esac -cmake -S . -B "${BUILD_DIR}" "${cmake_args[@]}" -cmake --build "${BUILD_DIR}" --target salmon --parallel 8 +run_logged "Configuring SALMON" "${BUILD_LOG_DIR}/${system}_configure.log" \ + cmake -Wno-dev -S . -B "${BUILD_DIR}" "${cmake_args[@]}" +run_logged "Building SALMON" "${BUILD_LOG_DIR}/${system}_build.log" \ + cmake --build "${BUILD_DIR}" --target salmon --parallel 8 salmon_bin=$(find "${BUILD_DIR}" -type f -name salmon -perm -u+x | head -n 1) if [[ -z "${salmon_bin}" ]]; then diff --git a/programs/salmon/list.csv b/programs/salmon/list.csv index fa1c447..386804d 100644 --- a/programs/salmon/list.csv +++ b/programs/salmon/list.csv @@ -1,6 +1,6 @@ system,enable,nodes,numproc_node,nthreads,elapse Fugaku,yes,1,4,12,0:10:00 RC_GH200,yes,1,1,1,0:10:00 -RC_DGXSP,yes,1,1,1,0:10:00 -RC_GENOA,yes,1,1,48,0:10:00 -RC_FX700,yes,1,1,48,0:30:00 +RC_DGXSP,yes,1,1,1,1:00:00 +RC_GENOA,yes,1,1,48,2:00:00 +RC_FX700,no,1,1,48,0:30:00 diff --git a/programs/salmon/patches/fjmpi-topology-guard.patch b/programs/salmon/patches/fjmpi-topology-guard.patch new file mode 100644 index 0000000..c13cff9 --- /dev/null +++ b/programs/salmon/patches/fjmpi-topology-guard.patch @@ -0,0 +1,58 @@ +diff --git a/CMakeLists.txt b/CMakeLists.txt +--- a/CMakeLists.txt ++++ b/CMakeLists.txt +@@ -43,6 +43,7 @@ option_set(USE_SCALAPACK "Use ScaLAPACK Library" OFF) + option_set(USE_EIGENEXA "Use EigenExa Library" OFF) + option_set(USE_LIBXC "Use Libxc library" OFF) + option_set(USE_FFTW "Use FFTW library" OFF) ++option_set(USE_FJMPI "Use FUJITSU MPI" OFF) + + ## Optimization for stencil compitations + option_set(USE_OPT_ARRAY_PADDING "Enable array padding for the stencil" ON) +diff --git a/src/config.h.in b/src/config.h.in +--- a/src/config.h.in ++++ b/src/config.h.in +@@ -15,6 +15,7 @@ + #cmakedefine SYSTEM_HAS_PATH_MAX_IN_LINUX_LIMITS_H + + #cmakedefine USE_MPI ++#cmakedefine USE_FJMPI + #cmakedefine USE_SCALAPACK + #cmakedefine USE_EIGENEXA + #cmakedefine USE_LIBXC +diff --git a/src/parallel/init_communicator.f90 b/src/parallel/init_communicator.f90 +--- a/src/parallel/init_communicator.f90 ++++ b/src/parallel/init_communicator.f90 +@@ -14,4 +14,5 @@ + ! limitations under the License. + ! ++#include "config.h" + MODULE init_communicator + implicit none +@@ -38,6 +39,6 @@ subroutine init_communicator_dft(info) + integer :: i1,i2,i3,i4,i5,ix,iy,iz,nl,io2,io3,io4 + integer,allocatable :: iranklists(:) + +-#ifdef __FUJITSU ++#ifdef USE_FJMPI + integer :: iret + #endif +@@ -62,5 +63,5 @@ subroutine init_communicator_dft(info) + 0:nproc_k-1)) + + ! communicator r,o,k,ro,ko +-#ifdef __FUJITSU ++#ifdef USE_FJMPI + call tofu_network_oriented_mapping(iret) +@@ -109,4 +110,4 @@ subroutine init_communicator_dft(info) + if (nl /= info%isize_rko-1) & + stop '[FATAL ERROR] init_communicator_dft' +-#ifdef __FUJITSU ++#ifdef USE_FJMPI + end if +@@ -460,4 +461,4 @@ + #endif + +-#ifdef __FUJITSU ++#ifdef USE_FJMPI + contains diff --git a/programs/salmon/run.sh b/programs/salmon/run.sh index 3dbf33c..a382136 100644 --- a/programs/salmon/run.sh +++ b/programs/salmon/run.sh @@ -13,19 +13,7 @@ RESULTS_DIR="${PWD}/results" WORK_DIR="${PWD}/salmon_run" INPUT_ARCHIVE_DEFAULT="/vol0003/rccs-sdt/data/a01010/benchmark_data/SALMON.tar.gz" INPUT_ARCHIVE_CLOUD="/lvs0/dne1/rccs-nghpcadu/CX_input/SALMON/SALMON.tar.gz" -GS_THEORY="${BK_SALMON_GS_THEORY:-dft}" -RT_THEORY="${BK_SALMON_RT_THEORY:-tddft_response}" - -salmon_normalize_legacy_theory() { - local input_file="$1" - local theory="$2" - - [[ -f "${input_file}" ]] || return 0 - if grep -Eiq '^[[:space:]]*theory[[:space:]]*=' "${input_file}"; then - sed -i -E "s/^([[:space:]]*theory[[:space:]]*=[[:space:]]*)['\"][^'\"]*['\"]([[:space:]]*,?.*)$/\1'${theory}'\2/I" "${input_file}" - sed -i -E "s/^([[:space:]]*theory[[:space:]]*=[[:space:]]*)[^[:space:],!]+([[:space:]]*,?.*)$/\1'${theory}'\2/I" "${input_file}" - fi -} +AOCL_ROOT_DEFAULT="/lvs0/rccs-nghpcadu/nakamura/aocl/install" mkdir -p "${RESULTS_DIR}" : > "${RESULTS_DIR}/result" @@ -46,11 +34,6 @@ case "${system}" in exec_gs=(./salmon) exec_rt=(./salmon) ;; - RC_FX700) - input_archive="${INPUT_ARCHIVE_CLOUD}" - exec_gs=(-stdin Si-1-1-1.nml ./salmon) - exec_rt=(-stdin Si-1-1-1-tddft.nml ./salmon) - ;; *) echo "Unknown system: ${system}" >&2 exit 1 @@ -62,6 +45,17 @@ if [[ ! -f "${input_archive}" ]]; then exit 1 fi +uses_stdin_input() { + case "$1" in + RC_GH200|RC_DGXSP|RC_GENOA) + return 0 + ;; + *) + return 1 + ;; + esac +} + rm -rf "${WORK_DIR}" mkdir -p "${WORK_DIR}/input" tar -xzf "${input_archive}" -C "${WORK_DIR}/input" @@ -78,8 +72,6 @@ fi cp artifacts/salmon "${WORK_DIR}/salmon" chmod +x "${WORK_DIR}/salmon" cp "${input_dir}"/* "${WORK_DIR}/" -salmon_normalize_legacy_theory "${WORK_DIR}/Si-1-1-1.nml" "${GS_THEORY}" -salmon_normalize_legacy_theory "${WORK_DIR}/Si-1-1-1-tddft.nml" "${RT_THEORY}" grep -Ein '^[[:space:]]*theory[[:space:]]*=' "${WORK_DIR}/Si-1-1-1.nml" "${WORK_DIR}/Si-1-1-1-tddft.nml" >&2 || true cd "${WORK_DIR}" @@ -100,20 +92,61 @@ case "${system}" in ;; RC_GENOA) module purge - module load system/genoa mpi/mpich-x86_64 - export OMP_NUM_THREADS="${nthreads}" - ;; - RC_FX700) - module purge - module load system/fx700 FJSVstclanga + module load system/genoa mpi/openmpi-x86_64 + export SLURM_MPI_TYPE=pmix + aocl_root="${BK_SALMON_AOCL_ROOT:-${AOCL_ROOT_DEFAULT}}" + aocl_blis_lib="${aocl_root}/amd-blis/lib/LP64" + aocl_flame_lib="${aocl_root}/amd-libflame/lib/LP64" + aocl_utils_lib="" + aocl_utils_so="$(find "${aocl_root}" -type f -name libaoclutils.so -print -quit 2>/dev/null || true)" + aocl_cpuid_so="$(find "${aocl_root}" -type f -name libau_cpuid.so -print -quit 2>/dev/null || true)" + if [[ -n "${aocl_utils_so}" ]]; then + aocl_utils_lib="$(dirname "${aocl_utils_so}")" + fi + if [[ -f "${aocl_blis_lib}/libblis-mt.so" && -f "${aocl_flame_lib}/libflame.so" && -n "${aocl_utils_so}" && -n "${aocl_cpuid_so}" ]]; then + export LD_LIBRARY_PATH="${aocl_utils_lib}:${aocl_flame_lib}:${aocl_blis_lib}:${LD_LIBRARY_PATH:-}" + fi export OMP_NUM_THREADS="${nthreads}" ;; + # RC_FX700) + # FX700 currently fails during GS initialization even with the Fujitsu + # topology guard patch applied. Keep this route disabled until verified. + # module purge + # module load system/fx700 FJSVstclanga + # export SLURM_MPI_TYPE=pmix + # export OMP_NUM_THREADS="${nthreads}" + # ;; esac run_salmon() { local logfile="$1" shift - mpiexec -n "${n_ranks}" "$@" > "${logfile}" 2>&1 + case "${system}" in + RC_GENOA) + mpirun -n "${n_ranks}" --bind-to core --map-by "ppr:${numproc_node}:node:PE=${nthreads}" "$@" > "${logfile}" 2>&1 + ;; + *) + mpiexec -n "${n_ranks}" "$@" > "${logfile}" 2>&1 + ;; + esac +} + +run_salmon_or_diagnose() { + local stage="$1" + local marker_file="$2" + local logfile="$3" + shift 3 + + if run_salmon "${logfile}" "$@"; then + return 0 + fi + + echo "SALMON ${stage} run failed" >&2 + echo "---- ${logfile} tail ----" >&2 + tail -n 80 "${logfile}" >&2 || true + echo "---- files updated since ${stage} start ----" >&2 + print_salmon_output_diagnostics "${marker_file}" + exit 1 } salmon_output_has_marker_since() { @@ -155,10 +188,10 @@ print_salmon_output_diagnostics() { touch .gs_start_marker gs_start=$(date +%s.%N) -if [[ "${system}" == "RC_GH200" || "${system}" == "RC_DGXSP" || "${system}" == "RC_GENOA" ]]; then - run_salmon gs.log "${exec_gs[@]}" < Si-1-1-1.nml +if uses_stdin_input "${system}"; then + run_salmon_or_diagnose GS .gs_start_marker gs.log "${exec_gs[@]}" < Si-1-1-1.nml else - run_salmon gs.log "${exec_gs[@]}" + run_salmon_or_diagnose GS .gs_start_marker gs.log "${exec_gs[@]}" fi gs_end=$(date +%s.%N) @@ -169,10 +202,10 @@ fi touch .rt_start_marker rt_start=$(date +%s.%N) -if [[ "${system}" == "RC_GH200" || "${system}" == "RC_DGXSP" || "${system}" == "RC_GENOA" ]]; then - run_salmon rt.log "${exec_rt[@]}" < Si-1-1-1-tddft.nml +if uses_stdin_input "${system}"; then + run_salmon_or_diagnose RT .rt_start_marker rt.log "${exec_rt[@]}" < Si-1-1-1-tddft.nml else - run_salmon rt.log "${exec_rt[@]}" + run_salmon_or_diagnose RT .rt_start_marker rt.log "${exec_rt[@]}" fi rt_end=$(date +%s.%N) diff --git a/scripts/test_submit_build.sh b/scripts/test_submit_build.sh new file mode 100755 index 0000000..2214c4d --- /dev/null +++ b/scripts/test_submit_build.sh @@ -0,0 +1,118 @@ +#!/bin/bash +# Submit an app build job for RC systems. + +if [ "$#" -ne 2 ]; then + echo "Usage: $0 " + echo " : program name (directory under programs/)" + echo " : line number in programs//list.csv (1-based, excluding header)" + echo "" + echo "Example: $0 salmon 4" + echo " This will submit a build job for the 4th configuration line from programs/salmon/list.csv" + exit 1 +fi + +code=$1 +list_csv_line_num=$2 + +if ! [[ "$list_csv_line_num" =~ ^[0-9]+$ ]]; then + echo "Error: must be a positive integer, got: '$list_csv_line_num'" + echo "Note: Use line number (1-based), not system name" + echo "Example: $0 $code 1" + exit 1 +fi + +if [ "$list_csv_line_num" -le 0 ]; then + echo "Error: must be greater than 0, got: $list_csv_line_num" + exit 1 +fi + +source ./scripts/job_functions.sh +SYSTEM_FILE="config/system.csv" +SYSTEM_INFO_FILE="config/system_info.csv" + +if [ ! -d "programs/$code" ]; then + echo "Error: programs/$code does not exist" + echo "Available programs:" + ls -1 programs/ 2>/dev/null | grep -v "^$" | head -10 + exit 1 +fi + +list_file="programs/$code/list.csv" +if [ ! -f "$list_file" ]; then + echo "Error: $list_file does not exist" + exit 1 +fi + +build_script="programs/$code/build.sh" +if [ ! -f "$build_script" ]; then + echo "Error: $build_script does not exist" + exit 1 +fi + +total_lines=$(tail -n +2 "$list_file" | wc -l) +if [ "$list_csv_line_num" -gt "$total_lines" ]; then + echo "Error: Line $list_csv_line_num does not exist in $list_file" + echo "Available lines: 1 to $total_lines" + echo "" + echo "Contents of $list_file:" + echo "Line# | Configuration" + echo "------|-------------" + echo " H | $(head -1 "$list_file")" + tail -n +2 "$list_file" | nl -v1 -w5 -s' | ' + exit 1 +fi + +line=$(tail -n +2 "$list_file" | sed -n "${list_csv_line_num}p") +if [ -z "$line" ]; then + echo "Error: Line $list_csv_line_num does not exist in $list_file" + exit 1 +fi + +IFS=, read -r -a cols <<< "$line" + +system="${cols[0]}" +enable="${cols[1]}" +run_nodes="${cols[2]}" +run_numproc_node="${cols[3]}" +run_nthreads="${cols[4]}" +elapse="${cols[5]}" + +if [[ "$enable" != "yes" ]]; then + echo "Notice: Line $list_csv_line_num has enable=$enable, skipping" + exit 1 +fi + +mode=$(get_system_mode "$system") +queue_group=$(get_system_queue_group "$system") + +if [[ -z "$mode" || -z "$queue_group" ]]; then + echo "Error: mode or queue_group not found for system=$system in $SYSTEM_FILE" + exit 1 +fi + +build_nodes=1 +build_cpus_per_task="${BK_BUILD_CPUS_PER_TASK:-8}" + +echo "Selected configuration from $list_file (line $list_csv_line_num):" +echo " $line" +echo "" +echo "Parsed values:" +echo " system=$system, enable=$enable, mode=$mode (from system.csv), queue_group=$queue_group (from system.csv)" +echo " run_nodes=$run_nodes, run_numproc_node=$run_numproc_node, run_nthreads=$run_nthreads, elapse=$elapse" +echo "" +echo "Build submission values:" +echo " nodes=$build_nodes, ntasks_per_node=1, cpus_per_task=$build_cpus_per_task" + +case "$system" in + RC_GH200|RC_DGXSP|RC_GENOA) + echo sbatch -p "$queue_group" -N "$build_nodes" -t "$elapse" --ntasks-per-node=1 --cpus-per-task="$build_cpus_per_task" \ + --wrap="bash programs/$code/build.sh $system" + sbatch -p "$queue_group" -N "$build_nodes" -t "$elapse" --ntasks-per-node=1 --cpus-per-task="$build_cpus_per_task" \ + --wrap="bash programs/${code}/build.sh $system" + ;; + *) + echo "Error: test_submit_build.sh currently supports RC systems only: RC_GH200, RC_DGXSP, RC_GENOA" + echo "Selected system: $system" + exit 1 + ;; +esac