Deployment¶
Platform-specific setup and job-submission guides for HPC systems. Both pages are rendered from the READMEs that live next to the install scripts, so they stay in sync with the scripts themselves.
OLCF Frontier¶
Setup¶
Clone the repo into your scratch directory and run the install script:
cd $MEMBERWORK
git clone https://github.com/AI-ModCon/BaseSIM_APEIRON.git
cd BaseSIM_APEIRON
source ./src/apeiron/deployment/frontier/install_venv.sh
install_venv.sh installs uv for your user if it isn’t already available, then uses it to create a virtual environment with exactly the dependency versions pinned in uv.lock. It then replaces the default CUDA builds of torch and torchvision with the ROCm builds of the same versions, read from uv.lock. The environment is saved to .venv in the project root and left activated. The script runs the following:
module load PrgEnv-gnu
module load python/3.13.0
module load gcc/12.2.0
module load rocm/7.1.0
command -v uv >/dev/null || python -m pip install --user uv
export PATH="$HOME/.local/bin:$PATH"
uv sync --locked --no-cache --python "$(command -v python)"
source ./.venv/bin/activate
ROCM=7.1
TORCH=$(uv export --locked --no-hashes --no-annotate --no-header --no-emit-project | sed -n 's/^torch==\([^ ;]*\).*/\1/p')
TORCHVISION=$(uv export --locked --no-hashes --no-annotate --no-header --no-emit-project | sed -n 's/^torchvision==\([^ ;]*\).*/\1/p')
uv pip install --no-cache \
"torch==${TORCH}+rocm${ROCM}" \
"torchvision==${TORCHVISION}+rocm${ROCM}" \
--index-url "https://download.pytorch.org/whl/rocm${ROCM}"
Warning: After the install, don’t run
uv runoruv syncin this environment. Both re-sync.venvtouv.lockand replace the ROCm builds of torch with the CUDA builds. Use the activated environment (source .venv/bin/activate) instead, oruv run --no-sync. To pick up a dependency change, re-runinstall_venv.sh.
Prior to running experiments, test ROCM support from the project root, in the activated environment:
pytest tests/test_rocm.py
Submitting a Job¶
Note: The MNIST example requires to the dataset, which is downloaded on first run. Download it before submitting a batch job:
python -c "from examples.mnist.utils import get_mnist_data; get_mnist_data()"
The virtual environment can be sourced directly at the top of your SLURM script (source .venv/bin/activate), so uv is not needed at runtime — jobs run against the installed environment.
From the project root:
sbatch -A xxx src/apeiron/deployment/frontier/mnist_example.sbatch
Troubleshooting¶
module load rocm/7.1.0fails — the ROCm 7.1 module on Frontier may use a different patch version. Runmodule avail rocmand load the 7.1.x module it lists. The PyTorch ROCm wheels for the lockedtorchneed ROCm 7.1.test_rocm.pyfails ortorch.version.hipisNone— the CUDA build of torch is installed, usually becauseuv runoruv syncwas used after the install. Re-runinstall_venv.sh.uv sync --lockedfails because the lockfile needs to be updated —pyproject.tomlwas changed without re-locking. Don’t re-lock on the cluster; pull the latestuv.lockfrom the repository, or runuv lockon your workstation and commit it.Disk quota errors in
$HOME— uv’s cache defaults to~/.cache/uv. The install script already passes--no-cache; if you runuvcommands yourself, add--no-cacheor pointUV_CACHE_DIRat project storage.invalid peer certificate: UnknownIssuer— the network intercepts TLS with a certificate uv’s bundled roots don’t trust. Setexport UV_SYSTEM_CERTS=1and re-run the script.uv: command not found—pip install --userputs uv in~/.local/bin; make sure it is on yourPATH.
NERSC Perlmutter¶
Setup¶
Clone the repo into your scratch directory and run the install script:
cd $SCRATCH
git clone https://github.com/AI-ModCon/BaseSIM_APEIRON.git
cd BaseSIM_APEIRON
source ./src/apeiron/deployment/perlmutter/install_venv.sh
install_venv.sh installs uv for your user if it isn’t already available, then uses it to create a virtual environment with exactly the dependency versions pinned in uv.lock. The environment is saved to .venv in the project root. The script runs the following:
module load python/3.13-26.1.0
command -v uv >/dev/null || python -m pip install --user uv
export PATH="$HOME/.local/bin:$PATH"
uv sync --locked --no-cache --python "$(command -v python)"
source .venv/bin/activate
Note: The MNIST example requires to the dataset, which is downloaded on first run. Download it before submitting a batch job:
uv run python -c "from examples.mnist.utils import get_mnist_data; get_mnist_data()"
Submitting a Job¶
The virtual environment can be sourced directly at the top of your SLURM script (source .venv/bin/activate), so uv is not needed at runtime — jobs run against the installed environment.
From the project root:
sbatch -A amsc002 src/apeiron/deployment/perlmutter/mnist_example.sbatch
Troubleshooting¶
uv sync --lockedfails because the lockfile needs to be updated —pyproject.tomlwas changed without re-locking. Don’t re-lock on the cluster; pull the latestuv.lockfrom the repository, or runuv lockon your workstation and commit it.Disk quota errors in
$HOME— uv’s cache defaults to~/.cache/uv. The install script already passes--no-cache; if you runuvcommands yourself, add--no-cacheor pointUV_CACHE_DIRat scratch storage (for exampleexport UV_CACHE_DIR=$SCRATCH/.cache/uv).uv: command not found—pip install --userputs uv in~/.local/bin; make sure it is on yourPATH.