Research Note #12 - Installing WRF-CHEM on Oakforest-PACS

This is basically the same with normal installation except for some modifications.

1. NetCDF Libraries

Because Oakforest-PACS (O-PACS) has already had NetCDF libraries (C, Fortran and C++), we don't need to install them. We just to have load the modules into the environment variables.

Update (2018/12):
If the netcdf libraries and mpi libraries have been installed in the system, ignore this step and go straight to step #3.

$ module load netcdf

$ module load netcdf-fortran

$ module load netcdf-cxx

Since WRF installer needs all of them at once, it's better to copy the NetCDF files into a directory in parallel storage, then set the path to that directory and put it into bash_profile file. To copy the files, check the environment variables which contains entry "netcdf". Merge directories of all netcdf for each compiler into one, hence we'll get a single netcdf directory needed for WRF installer.

$ env | grep netcdf

$ cp <original netcdf dir> <new dir>

$ cp <original netcdf-fortran dir> <new dir>

$ cp <original netcdf-cxx dir> <new dir>

Let's say the new netcdf directory is at /work/gi55/c24223/libs/netcdf

export PACSLIB=/work/gi55/c24223/libs

export NETCDF=$PACSLIB/netcdf

export PATH=$NETCDF/bin:$PATH

2. MPI Library

O-PACS will load Intel MPI (impi) by default, thus we also don't need to install MPICH. Just set the directory of impi binaries (which contains mpicc, mpiexec etc) into environment variables and put it inside the bash_profile. The trick is basically same with NetCDF libraries, but this time, no need to copy the files because impi only has a single directory for all compiler.

$ env | grep impi

export mpi=/home/opt/local/cores/intel/impi/2018.1.163/bin64

export PATH=$mpi:$PATH

Now, reload the bash_profile script to get the changes.

3. Compiling WRF

It's better to compile WRF first, before installing any other libraries for WPS (zlib, libpng and JasPer). The main reason is, in order to install those WPS libraries, we should set up some environment variables which may mess up the WRF compilation (LD_LIBRARY_PATH, CPPFLAG and LDFLAGS).

Update (2018/12): 
If you have already set the .bash_profile file with shared_library variable (LD_LIBRARY_PATH), you don't need to delete/remark that line. Just remark CPPFLAG and LDFLAGS. Anyway, don't forget to remove the remarks (back to original state) after WRF installation completed, otherwise the WPS installation will fail.

$ export WRF_CHEM=1
$ export WRF_EM_CORE=1
$ export WRF_NMM_CORE=0

$ tar -xzvf WRFV3.8.1.TAR.gz

$ cd WRFV3/
$ tar -xzvf WRFV3-Chem-3.8.1.TAR.gz

$ ./clean -a

$ ./configure

Select the dmpar option with intel compiler Xeon AVX mods and basic nesting, then compile em_real.

18. (serial)  19. (smpar)  20. (dmpar)  21. (dm+sm)   INTEL (ifort/icc): Xeon (SNB with AVX mods)

$ ./compile em_real >& compile.log &

Make sure the compilation is successful.

==========================================================================
build started:   Tue Jan 16 13:26:53 JST 2018
build completed: Tue Jan 16 14:11:22 JST 2018

--->                  Executables successfully built                  <---

-rwxr-x--- 1 c24223 gi55 65532338 Jan 16 14:11 main/ndown.exe
-rwxr-x--- 1 c24223 gi55 65463925 Jan 16 14:11 main/real.exe
-rwxr-x--- 1 c24223 gi55 64466197 Jan 16 14:11 main/tc.exe
-rwxr-x--- 1 c24223 gi55 75598185 Jan 16 14:10 main/wrf.exe

==========================================================================

4. Zlib, Libpng and JasPer libraries

Because the libraries will be compiled and installed using intel compilers, we should set the environment variables.

Update (2018/12):
If all WPS library (zlib, libpng and jasper) has been installed or exists in the system, ignored this step and go straight to step #5. 

$ export CC=icc

$ export FC=ifort

$ export CXX=icpc

$ export F77=ifort

$ export FCFLAGS=-axMIC-AVX512

$ export FFLAGS=-axMIC-AVX512

Install all libraries in $PACSLIB directory.

$ tar -xzvf zlib-1.2.11.tar.gz

$ tar -xzvf libpng-1.6.34.tar.gz

$ tar -xzvf jasper-1.900.1.tar.gz

The configuration and installation of each libraries are almost same. Before installing zlib, set environment variables for dynamic library linker and c pre-processor.

$ export LDLIB=-L$PACSLIB/grib2/lib

$ export CPPFLAGS=-I$PACSLIB/grib2/include

Then install each library with the following steps (zlib first, then libpng and JasPer):

$ ./configure --prefix=$PACSLIB/grib2 

$ make

$ make install

5. Compiling WPS

Before compiling WPS, exit and re-login the shell. It's needed to return the environment variables of LDLIB and CPPFLAGS back to their original states with the entries loaded by O-PACS.

Set environment variables for JasPer:

export JASPERLIB=$PACSLIB/grib2/lib

export JASPERINC=$PACSLIB/grib2/include

Then add grib2 lib directory as well as NetCDF lib directory into the existed LD_LIBRARY_PATH.

export LD_LIBRARY_PATH=/home/opt/local/cores/intel/impi/2018.1.163/intel64/lib: <many directories>:/work/gi55/c24223/libs/grib2/lib:/work/gi55/c24223/libs/netcdf/lib

Then configure and compile using intel compiler with serial configuration (option #17):

$ tar -xzvf WPSV3.8.1.TAR.gz

$ cd WPS

$ ./clean -a

$ ./configure

$ ./compile

Check if the compilation successfully generates geogrid.exe, ungrib.exe and metgrid.exe.

lrwxrwxrwx 1 c24223 gi55 23 Jan 16 15:08 geogrid.exe -> geogrid/src/geogrid.exe

lrwxrwxrwx 1 c24223 gi55 23 Jan 16 15:09 metgrid.exe -> metgrid/src/metgrid.exe

lrwxrwxrwx 1 c24223 gi55 21 Jan 16 15:09 ungrib.exe -> ungrib/src/ungrib.exe

For the last measure, check the library dependency of ungrib.exe and the other two executables.

$ ldd ungrib.exe

 linux-vdso.so.1 =>  (0x00007fff239f0000)

        libpng16.so.16 => /work/gi55/c24223/libs/grib2/lib/libpng16.so.16 (0x00007f3555e64000)

        libz.so.1 => /work/gi55/c24223/libs/grib2/lib/libz.so.1 (0x00007f3555c45000)

        libm.so.6 => /lib64/libm.so.6 (0x00007f3555943000)

        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3555727000)

        libc.so.6 => /lib64/libc.so.6 (0x00007f3555366000)

        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f3555150000)

        libdl.so.2 => /lib64/libdl.so.2 (0x00007f3554f4c000)

        libimf.so => /home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libimf.so (0x00007f35549be000)

        libsvml.so => /home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libsvml.so (0x00007f355330b000)

        libirng.so => /home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libirng.so (0x00007f3552f97000)

        libintlc.so.5 => /home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libintlc.so.5 (0x00007f3552d2a000)

        /lib64/ld-linux-x86-64.so.2 (0x00007f35560b3000)

If all library found (no 'library not found'), then the compilation is successful.