Spack HPC应用环境部署
本文记录整理Spack(HPC系统应用软件管理工具)的使用说明。
Spack 简介
Spack 是一款强大的软件包管理工具
- 降低了应用软件的安装难度（源码编译，二进制包安装）
- 解决软件之间的依赖关系（确保构建的一致性）
- 软件层的构建系统（软件+选项/功能，区别于CMake，Make）
Spack 安装
## 安装 spack 代码包
> git clone https://github.com/spack/spack.git
## 初始化 spack 环境
> source spack/share/spack/setup-env.sh
Spack 环境配置
把如下内容添加到 ~/.bashrc 或者 ~/.bash_profile文件中。
# .bashrc
ulimit -l unlimited
ulimit -s unlimited
# Module package management
source /THL8/software/modules/3.2.10/Modules/3.2.10/init/bash
module purge
# Spack package management
export SPACK_ROOT=/THL8/software/opt/spack
export PATH=$SPACK_ROOT/bin:$PATH
source ${SPACK_ROOT}/share/spack/setup-env.sh
查看 Spack 已安装软件
> spack find -l [software name]
选项说明：
-p, --paths 显示软件安装路径
-d, --deps 显示软件完整的依赖关系图（有向无环图)
-l, --long 显示软件spec的hash值和版本号
-f, --show-flags 显示软件安装时的编译器选项
-x, --explicit 显示直接安装的软件
-X, --implicit 显示作为依赖安装的软件
-v, --variants 显示软件的安装选项
software name 指定软件名称，查看该软件详细信息（hash值，版本，依赖，安装选项，编译选项）
已安装软件的加载
把如下内容添加到 ~/.bashrc 或者 ~/.bash_profile文件中，即可完成对intel编译器19.0.5版本，miniconda2的 4.7.12.1版本，fftw的3.3.8版本和netcdf-fortran的4.5.2版本的安装。
# Spack package load
source <(spack module tcl loads --dependencies /w5ky76k) # intel@19.0.5
source <(spack module tcl loads --dependencies /s6pzrxk) # miniconda2@4.7.12.1
source <(spack module tcl loads --dependencies /pomc2sx) # fftw@3.3.8
source <(spack module tcl loads --dependencies /xpyelbb) # netcdf-fortran@4.5.2
其中 --dependencies 是指同时加载该软件的依赖，/xpyelbb 是netcdf-fortran@4.5.2的安装版本的hash值。配置完 ~/.bashrc 或者 ~/.bash_profile文件后，通过source 更新环境，即可完成相关软件的加载，可以通过module list查看
(base) demo@ubuntu:~/github/spack$ module list
Currently Loaded Modulefiles:
1) intel-19.0.5-gcc-8.3.0-w5ky76k 9) rdma-core-20-gcc-8.3.0-4hecl3d
2) miniconda2-4.7.12.1-intel-19.0.5-s6pzrxk 10) slurm-16.05.7-intel-19.0.5-rmgee6m
3) libpciaccess-0.13.5-intel-19.0.5-p4yolyz 11) mvapich2-2.3.4-intel-19.0.5-23edecg
4) libiconv-1.16-gcc-8.3.0-hgw6qbv 12) fftw-3.3.8-intel-19.0.5-pomc2sx
5) xz-5.2.5-intel-19.0.5-t7exvpq 13) mvapich2-2.3.4-intel-19.0.5-tgxj347
6) zlib-1.2.11-intel-19.0.5-onq52rg 14) hdf5-1.10.6-intel-19.0.5-3y4zybv
7) libxml2-2.9.10-intel-19.0.5-ososnes 15) netcdf-c-4.7.3-intel-19.0.5-mz4yici
8) libnl-3.3.0-gcc-8.3.0-esj5mge 16) netcdf-fortran-4.5.2-intel-19.0.5-xpyelbb
应用软件的安装
准备工作—：软件搜索
要使用 spack 安装一款科学计算软件，首选要确定 spack 软件库中是否已经包含这款软件，确认的办法就是使用 spack list 加软件名搜索，比如我们要安装 netcdf，使用 spack list netcdf 搜索得到如下结果：
(base) demo@ubuntu:~/github/spack$ spack list netcdf
==> 6 packages.
netcdf-c netcdf-cxx netcdf-cxx4 netcdf-fortran parallel-netcdf py-netcdf4
如果spack 还不支持该软件，也就是软件库中还没有，比如wrf，搜索结果会如下所示：
(base) demo@ubuntu:~/github/spack$ spack info wrf
==> Error: Package wrf not found.
不过有时搜索不到不是因为spack 库中没有，而是spack中修改了软件的名字。这时可以通过搜索软件描述来查找软件，比如我们查找 geophysical 相关 的软件：
(base) demo@ubuntu:~/github/spack$ spack list geophysical
==> 0 packages.
(base) demo@ubuntu:~/github/spack$ spack list --search-description geophysical
==> 1 packages.
spherepack
直接用 spack list 查找是没有，但是加上 --search-description 选项就可以找到一个叫做spherepack的软件包。
准备工作二：查看软件详细信息
有时即便软件库中包含了你想安装的软件，也不必着急安装，明智的做法是先使用 spack info package 命令查看spack中对软件的描述，还是以 netcdf-c 为例：
(base) demo@ubuntu:~/github/spack$ spack info netcdf-c
AutotoolsPackage: netcdf-c
Description:
NetCDF (network Common Data Form) is a set of software libraries and
machine-independent data formats that support the creation, access, and
sharing of array-oriented scientific data. This is the C distribution.
Homepage: http://www.unidata.ucar.edu/software/netcdf
Maintainers: @skosukhin @WardF
Tags:
None
Preferred version:
4.7.3 ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-c-4.7.3.tar.gz
Safe versions:
master [git] https://github.com/Unidata/netcdf-c.git on branch master
4.7.3 ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-c-4.7.3.tar.gz
4.7.2 ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-c-4.7.2.tar.gz
4.7.1 ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-c-4.7.1.tar.gz
4.7.0 ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-c-4.7.0.tar.gz
4.6.3 ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-c-4.6.3.tar.gz
4.6.2 ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-c-4.6.2.tar.gz
4.6.1 ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4.6.1.tar.gz
4.6.0 ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4.6.0.tar.gz
4.5.0 ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4.5.0.tar.gz
4.4.1.1 ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4.4.1.1.tar.gz
4.4.1 ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4.4.1.tar.gz
4.4.0 ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4.4.0.tar.gz
4.3.3.1 ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4.3.3.1.tar.gz
4.3.3 ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4.3.3.tar.gz
Variants:
Name [Default] Allowed values Description
===================== ============== ===============================
dap [off] on, off Enable DAP support
hdf4 [off] on, off Enable HDF4 support
jna [off] on, off Enable JNA support
mpi [on] on, off Enable parallel I/O for
netcdf-4
parallel-netcdf [off] on, off Enable parallel I/O for classic
files
pic [on] on, off Produce position-independent
code (for shared libs)
shared [on] on, off Enable shared library
Installation Phases:
autoreconf configure build install
Build Dependencies:
autoconf curl hdf5 m4 parallel-netcdf
automake hdf libtool mpi zlib
Link Dependencies:
curl hdf hdf5 mpi parallel-netcdf zlib
Run Dependencies:
None
Virtual Packages:
None
这里给出了netcdf-c软件的简要介绍，支持的版本，支持的安装选项，安装步骤，软件安装的依赖和链接的依赖。这里需要重点关注的是两个，一是软件的版本，这个根据用户需求确定；二是Variants，即安装的选项。例如是否支持mpi，是否支持并行I/O等等。
准备工作三：查看软件 spec
(base) demo@ubuntu:~/github/spack$ spack spec -Il netcdf-c %gcc@8.3.0 arch=linux-rhel7-haswell
Input spec
--------------------------------
- netcdf-c%gcc@8.3.0 arch=linux-rhel7-haswell
Concretized
--------------------------------
- fisfeqj netcdf-c@4.7.3%gcc@8.3.0 cflags="-O2 -fPIC" cxxflags="-O2 -fPIC" fflags="-O2 -fPIC" ~dap~hdf4~jna+mpi~parallel-netcdf+pic+shared arch=linux-rhel7-haswell
- p6z4wf2 ^hdf5@1.10.6%gcc@8.3.0 cflags="-O2 -fPIC" cxxflags="-O2 -fPIC" fflags="-O2 -fPIC" ~cxx~debug~fortran+hl+mpi+pic+shared~szip~threadsafe api=none arch=linux-rhel7-haswell
[+] 45jo6ul ^mpich@3.2%gcc@8.3.0 cflags="-O2 -fPIC" cxxflags="-O2 -fPIC" fflags="-O2 -fPIC" +hwloc+hydra+libxml2+pci+romio~slurm~verbs+wrapperrpath device=ch3 netmod=tcp pmi=pmi arch=linux-rhel7-haswell
[+] bpwxbl3 ^zlib@1.2.11%gcc@8.3.0 cflags="-O2 -fPIC" cxxflags="-O2 -fPIC" fflags="-O2 -fPIC" +optimize+pic+shared arch=linux-rhel7-haswell
[+] rahvynl ^m4@1.4.18%gcc@8.3.0 cflags="-O2 -fPIC" cxxflags="-O2 -fPIC" fflags="-O2 -fPIC" +sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=linux-rhel7-haswell
[+] zcmcitr ^libsigsegv@2.12%gcc@8.3.0 cflags="-O2 -fPIC" cxxflags="-O2 -fPIC" fflags="-O2 -fPIC" arch=linux-rhel7-haswell
这里的-I选项显示目标软件和依赖软件的安装状态，- 表示未安装，[+]表示已经安装；-l选项给出依赖的hash值以及版本信息；%gcc@8.3.0 指定使用gcc 8.3.0编译器编译安装，arch=linux-rhel7-haswell 指明系统的架构是haswell，操作系统是linux，发行版是rhel7。
查看spec 的目的是确认该版本的软件能否安装，以来是否满足，以及依赖软件的安装状态。
最终的工作：软件安装
(base) demo@ubuntu:~/github/spack$ spack install netcdf-c %gcc@8.3.0 arch=linux-rhel7-haswell
==> mpich@3.2 : externally installed in /THL8/software/opt/linux-rhel7-haswell/gcc-8.3.0/mpich-3.2-24zn2tm
[+] /THL8/software/opt/linux-rhel7-haswell/gcc-8.3.0/mpich-3.2-24zn2tm
[+] /THL8/software/opt/linux-rhel7-haswell/gcc-8.3.0/zlib-1.2.11-bpwxbl3
[+] /THL8/software/opt/linux-rhel7-haswell/gcc-8.3.0/libsigsegv-2.12-zcmcitr
==> Installing hdf5
==> No binary for hdf5 found: installing from source
==> hdf5: Executing phase: 'autoreconf'
==> hdf5: Executing phase: 'configure'
==> hdf5: Executing phase: 'build'
==> hdf5: Executing phase: 'install'
[+] /THL8/software/opt/linux-rhel7-haswell/gcc-8.3.0/hdf5-1.10.6-p6z4wf2
[+] /THL8/software/opt/linux-rhel7-haswell/gcc-8.3.0/m4-1.4.18-rahvynl
==> Installing netcdf-c
==> No binary for netcdf-c found: installing from source
==> netcdf-c: Executing phase: 'autoreconf'
==> netcdf-c: Executing phase: 'configure'
==> netcdf-c: Executing phase: 'build'
==> netcdf-c: Executing phase: 'install'
[+] /THL8/software/opt/linux-rhel7-haswell/gcc-8.3.0/netcdf-c-4.7.3-fisfeqj
这一步只需要将上面命令中的 spec -Il 替换成 install 即可启动软件的安装。之前查询显示已经安装的软件会自动跳过安装。
软件安装环境配置和选项设置
查找和添加编译器
(base) demo@ubuntu:~/github/spack$ spack compilers
==> Available compilers
-- gcc rhel7-x86_64 ---------------------------------------------
gcc@4.8.5
初次运行此命令会在~/.spack/ 目录下创建linux 文件夹，并写入compilers.yaml 文件。正如上面所示，在初始系统环境下只能检索到系统自带的编译器，版本一般都比较旧，无法满足应用部署的需求，需要安装较新版本的编译器，如GCC＠８.３.０。
(base) demo@ubuntu:~/github/spack$ spack spec -Il gcc@8.3.0 arch=linux-rhel7-haswell
(base) demo@ubuntu:~/github/spack$ spack install gcc@8.3.0 arch=linux-rhel7-haswell
编译器安装后可以通过spack find 检索
(base) demo@ubuntu:~/github/spack$ spack find -l
==> 16 installed packages
-- linux-rhel7-haswell / gcc@4.8.5 ------------------------------
nbrt3t7 autoconf@2.69 oywfhvk libsigsegv@2.12 yiirsm2 perl@5.30.3
rqa3ho7 automake@1.16.2 swiq7rt libtool@2.4.6 cexp2vp pkgconf@1.7.3
263modn gcc@8.3.0 dipchcn m4@1.4.18 a72rtxy readline@8.0
qrwijay gdbm@1.18.1 upj2gnv mpc@1.1.0 zolwez4 zlib@1.2.11
raqmdao gmp@6.1.2 vql3fcw mpfr@3.1.6
zwfd4x6 isl@0.18 cwq7uyp ncurses@6.2
然后加载新安装的GCC，并添加到配置环境中：
(base) demo@ubuntu:~/github/spack$ spack load gcc@8.3.0
(base) demo@ubuntu:~/github/spack$ spack compiler find
==> Added 1 new compiler to /THL7/home/maqz2/.spack/linux/compilers.yaml
gcc@8.3.0
==> Compilers are defined in the following files:
/THL7/home/maqz2/.spack/linux/compilers.yaml
查看compilers.yaml的内容，可以发现gcc@8.3.0已经添加到其中：
[maqz2@th-hpc2-ln0 ~]$ cat /THL7/home/maqz2/.spack/linux/compilers.yaml
compilers:
- compiler:
paths:
cc: /usr/bin/gcc
cxx: /usr/bin/g++
f77: /usr/bin/gfortran
fc: /usr/bin/gfortran
operating_system: rhel7
target: x86_64
modules: []
environment: {}
extra_rpaths: []
flags: {}
spec: gcc@4.8.5
- compiler:
paths:
cc: /THL6/software/opt/linux-rhel7-haswell/gcc-4.8.5/gcc-8.3.0-263modn/bin/gcc
cxx: /THL6/software/opt/linux-rhel7-haswell/gcc-4.8.5/gcc-8.3.0-263modn/bin/g++
f77: /THL6/software/opt/linux-rhel7-haswell/gcc-4.8.5/gcc-8.3.0-263modn/bin/gfortran
fc: /THL6/software/opt/linux-rhel7-haswell/gcc-4.8.5/gcc-8.3.0-263modn/bin/gfortran
operating_system: rhel7
target: x86_64
modules: []
environment: {}
extra_rpaths: []
flags: {}
spec: gcc@8.3.0
配置方式
spack 支持命令行参数和配置文件两种配置方式
检查配置参数是否正确
(base) demo@ubuntu:~/github/spack$ spack config blame config
--- config:
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:105 checksum: True
/THL6/software/opt/spack-0.15.4/etc/spack/config.yaml:69 build_jobs: 16
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:86 verify_ssl: True
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:110 dirty: False
_builtin debug: False
/THL6/software/opt/spack-0.15.4/etc/spack/config.yaml:57 build_stage:
/THL6/software/opt/spack-0.15.4/etc/spack/config.yaml:58 - ~/.spack/stage
/THL6/software/opt/spack-0.15.4/etc/spack/config.yaml:59 - /THL6/software/opt/tmp
/THL6/software/opt/spack-0.15.4/etc/spack/config.yaml:60 - /THL6/home/$user/tmp
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:63 - $tempdir/$user/spack-stage
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:63 # - $spack/var/spack/stage
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:63
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:63
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:81 # Cache directory for already downloaded source tarballs and archived
/THL6/software/opt/spack-0.15.4/etc/spack/config.yaml:19 # repositories. This can be purged with `spack clean --downloads`.
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:23 connect_timeout: 10
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:23 install_tree: /THL6/software/opt/
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:24 template_dirs:
/THL6/software/opt/spack-0.15.4/etc/spack/config.yaml:23 - $spack/share/spack/templates
/THL6/software/opt/spack-0.15.4/etc/spack/config.yaml:27
/THL6/software/opt/spack-0.15.4/etc/spack/config.yaml:28
/THL6/software/opt/spack-0.15.4/etc/spack/config.yaml:29 # Default directory layout
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:70 install_path_scheme: ${ARCHITECTURE}/${COMPILERNAME}-${COMPILERVER}/${PACKAGE}-${VERSION}-${HASH:7}
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:75 module_roots:
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:94 tcl: /THL6/software/opt/share/spack/modules
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:100 lmod: /THL6/software/opt/share/spack/lmod
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:118 source_cache: $spack/var/spack/cache
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:127 misc_cache: ~/.spack/cache
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:139 suppress_gpg_warnings: False
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:146 install_missing_compilers: False
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:154 build_language: C
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:159 locks: True
/THL6/software/opt/spack-0.15.4/etc/spack/defaults/config.yaml:163 ccache: False
问题排查及处理（待完成）
报错信息
查看日志 spack-build-out.txt
安装调试
We can also enter the build area and try to manually run the build:
$ spack build-env mpileaks bash
$ spack cd mpileaks
The spack build-env command spawned a new shell that contains the same environment that Spack used to build the mpileaks package (you can substitute bash for your favorite shell). The spack cd command changed our working dirctory to the last attempted build for mpileaks. From here we can manually re-run the build: