Research Note #33 - Instalasi WRF-Chem 4.7.1 + KPP di HPC BRIN (Mahameru)

Kinetic PreProcessor (KPP) adalah tool untuk menerjemahkan reaksi kimia kompleks ke dalam kode program (misal Fortran atau C) dan umumnya digunakan pada pemodelan kimia atmosfer, termasuk WRF-Chem. Dengan sekian banyaknya pilihan opsi mekanisme kimia yang dapat digunakan pada WRF-Chem, KPP membantu developer untuk mengaplikasikan skema kimia kompleks yang mereka bangun tanpa harus 'mengganggu' kode program WRF-Chem itu sendiri. Beberapa mekanisme kimia yang menerapkan reaksi kimia kompleks, misalnya untuk simulasi fase gas pada secondary aerosol, antara lain: MOZART, MOZCART, RADM, MOSAIC dll. Dengan kata lain, bila ingin menggunakan mekanisme-mekanisme tersebut untuk simulasi, WRF-Chem yang digunakan wajib hukumnya dikompilasi dengan dukungan KPP.

Tulisan ini berasumsi bahwa environment yang digunakan adalah HPC BRIN, dengan bash shell, gcc 12.2.0 dan pembaca telah mengetahui dasar-dasar kompilasi WRF-Chem ver. 4.7.1 (misal, sudah memiliki library-library yang dibutuhkan, seperti NetCDF, MPICH dll). Bila belum, silakan membaca tulisan berikut: https://sketsa-langit.blogspot.com/2025/10/research-note-29-instalasi-wrf-chem-di.html.

Berikut langkah-langkah kompilasi WRF-Chem dengan KPP pada HPC BRIN. 

Temukan path library flex di sistem. Library flex ini akan digunakan untuk kompilasi KPP:

module load gcc flex

module show flex

------------------------------------------------------------------------------------------------------------------------------------------------------------

   /mgpfs/shared/apps/mod/library/flex/2.6.4:

------------------------------------------------------------------------------------------------------------------------------------------------------------

load("gcc/12.2.0")

whatis("flex 2.6.4 - The fast lexical analyzer generator")

setenv("FLEXHOME","/mgpfs/shared/apps/app/flex/2.6.4/GCC/12.2.0")

prepend_path("PATH","/mgpfs/shared/apps/app/flex/2.6.4/GCC/12.2.0/bin")

prepend_path("CPATH","/mgpfs/shared/apps/app/flex/2.6.4/GCC/12.2.0/include")

prepend_path("INCLUDE_PATH","/mgpfs/shared/apps/app/flex/2.6.4/GCC/12.2.0/include")

prepend_path("INCLUDE_LIBRARY_PATH","/mgpfs/shared/apps/app/flex/2.6.4/GCC/12.2.0/include")

prepend_path("LIBRARY_PATH","/mgpfs/shared/apps/app/flex/2.6.4/GCC/12.2.0/lib")

prepend_path("LD_LIBRARY_PATH","/mgpfs/shared/apps/app/flex/2.6.4/GCC/12.2.0/lib")

prepend_path("INFOPATH","/mgpfs/shared/apps/app/flex/2.6.4/GCC/12.2.0/share/info")

prepend_path("MANPATH","/mgpfs/shared/apps/app/flex/2.6.4/GCC/12.2.0/share/man")

Deklarasikan WRF_CHEM, WRF_KPP dan FLEX_LIB_DIR di environment bash:

export WRF_CHEM=1

export WRF_KPP=1

export FLEX_LIB_DIR=/mgpfs/shared/apps/app/flex/2.6.4/GCC/12.2.0/lib

Masuk ke interactive-node HPC karena proses kompilasi ini akan memakan memori sistem yang besar. Proses kompilasi pada login-node hampir bisa dipastikan akan gagal (akan di-kill oleh sistem).

srun --partition=interactive --pty /bin/bash

Setelah masuk ke interactive-node, masuk ke direktori WRF dan jalankan konfigurasi kompilasi wrf.

./clean -aa

./configure

Pilih 34 (dmpar) untuk GCC compiler. Untuk nesting, pilih 1 (basic).

Pastikan proses konfigurasi berjalan lancar, tanpa ada pesan masalah pada compiler. Bila ada masalah, bereskan dulu masalah compiler, lalu lakukan konfigurasi ulang. Bila tak ada masalah, buka dan edit file configure.wrf.

Hilangkan 'time' pada deklarasi fortran compile 'FC':

FC = time $(DM_FC)  menjadi  FC = $(DM_FC)

Simpan file dan keluar dari configure.wrf. Selanjutnya, pada direktori WRF, buka dan edit file scan.h dan scan.y yang terdapat pada direktori chem/KPP/kpp/kpp-2.1/src/. 

Pada file scan.h (dan scan.y), temukan kode baris void yyerror( char * str ) ; pada baris ke-101 (390 pada scan.y), dan ubah menjadi void yyerror( const char * str ) ;. Simpan perubahan dan keluar dari kedua file tersebut.

Buka file compile_wkc pada direktori chem/KPP dan ubahlah baris kode pada baris 121:

 $WKC_HOME/util/wkc/tuv_kpp FIRST ../../inc/

menjadi:

 $WKC_HOME/util/wkc/tuv_kpp FIRST ../../../../inc/

Simpan perubahan dan keluar dari file. Sekarang anda bisa melakukan kompilasi wrf.

./compile -j 2 em_real >& compile.log

Proses kompilasi akan memakan waktu kurang lebih 30 menit - 1 jam. Pastikan tidak ada Error yang tercatat pada file compile.log dan file-file exe berhasil dibuat.

--->                  Executables successfully built                  <---

-rwxr-xr-x 1 ardh003 researcher 63915712 Apr 30 10:35 main/ndown.exe

-rwxr-xr-x 1 ardh003 researcher 64055224 Apr 30 10:35 main/real.exe

-rwxr-xr-x 1 ardh003 researcher 57765544 Apr 30 10:35 main/tc.exe

-rwxr-xr-x 1 ardh003 researcher 85490080 Apr 30 10:29 main/wrf.exe

==========================================================================

Research Note #25 - Reading a Text File Line-by-Line with Fortran

Again, this is actually a very simple task which I usually deal with ... and forget, most of the time! That's why this post exists. The following is a basic Fortran code to enable you reading the contents of a text file, line-by-line. 

As an example, let's make a text file (named 'list.txt') containing a list of files with their complete paths on Linux:

$ ls ../gcom-c/aerosol/0524/pl/* > list.txt

$ cat list.txt

../gcom-c/aerosol/0524/pl/GC1SG1_20190901D01D_T0524_L2SG_ARPLK_1001.h5

../gcom-c/aerosol/0524/pl/GC1SG1_20190902D01D_T0524_L2SG_ARPLK_1001.h5

../gcom-c/aerosol/0524/pl/GC1SG1_20190903D01D_T0524_L2SG_ARPLK_1001.h5

../gcom-c/aerosol/0524/pl/GC1SG1_20190904D01D_T0524_L2SG_ARPLK_1001.h5

Now, this is the Fortran code to read file 'list.txt':

PROGRAM READTEXT

IMPLICIT NONE

CHARACTER(LEN=8) :: listfile

CHARACTER(LEN=70) :: filecontents

INTEGER :: io

listfile="list.txt"

OPEN(10, FILE=listfile, FORM="FORMATTED", STATUS="OLD",&

        ACTION="READ")

DO

   READ(10,"(A70)",IOSTAT=io) filecontents

   IF (io/=0) EXIT

   PRINT *, filecontents

END DO

CLOSE(10)

END PROGRAM READTEXT

How it works?

The program will open file 'list.txt', then read the first line of the file and put the (string) contents into variable 'filecontents'. It will then print the contents of variable onto the screen. Since there's a DO.. END DO statement, this process will repeat, with the next line being read, replacing the contents of variable 'filecontents'. This looping process will end once there are no more lines found in the file, invoking IOSTAT to a value other than 0 (0 means no I/O error occurred, other values mean there are I/O errors), then program will exit. That's all!

Some important notices:

• The absence of 'DO ... END DO' will make the program only reads the first line of the file.
• The absence of 'IF (io/=0) EXIT' will make the program runs endlessly, with the last line of the file kept being showed on the screen.
• If the output format is not determined (for example, FMT=* instead of A70), the program will not correctly show the file's contents. The (character) length of this format should be same or more than the one at variable declaration. Why 70? Because in this case, the full path of file list (in one line) has 70 characters. 

Research Note #24 - Showing GCOM-C/SLGI Dataset Header/Metadata

One of the most important things to do while working with binary satellite data is checking the header/metadata of the dataset. By doing so, you could know the dimensions, parameters, variables and attributes of the data which will be needed for analysis (with program, script, etc.). These are three methods to show the GCOM-C/SLGI dataset header and attributes. I made this post because I always forget the quickest way to do it and ended up wasting too much time just for knowing some attributes of the data.

1. Using HDF View

This is definitely the quickest way and the most convenient way to check the headers and attributes of HDF data file. It works perfectly with MODIS as well as GCOM-C/SLGI. Just open the HDF file, and browse into the dataset directories. Simple!

2. Using h5dump

Works in similar fashion with ncdump. Just type the command and you're good to go. Not so convenient for non-Linux user.

Example:

h5dump -H <filename> --> Show header contents of a datafile

h5dump -a /Image_data/LST/Slope <filename> --> Show the contents of attribute 'Slope' (scaling factor) of dataset 'LST' of a datafile

3. Using SGLI Tool

This is by far the most not-convenient way to see the headers. Not only it's slow and taking so much of system resources, it also shows the header only (no attributes). To see the header, open a HDF file and go to Menu->View->Meta Data. 

Research Note #23 - GEE Script to Overlay Sentinel 2A with MODIS Terra/Aqua Hotspots

The main purpose of this script is to overlay MODIS Terra/Aqua hotspots product MOD14A1/MYD14A1 (daily thermal anomalies, lev3, 1km grid) over Sentinel-2A Thermal Reflectance (lev2, 10-20m spatial resolution). The channels used for Sentinel data is 11, 8 and 4 (SWIR1, NIR, RED) hence creating false color image over area of Sri Mukhtar Sahib city in Punjab region, India (14x zoom) on Google Map. Observation time was set between November 1 - 15, 2018.

/**
 * Function to mask clouds using the Sentinel-2 QA band
 * @param {ee.Image} image Sentinel-2 image
 * @return {ee.Image} cloud masked Sentinel-2 image
 */
function maskS2clouds(image) {
  var qa = image.select('QA60');

  // Bits 10 and 11 are clouds and cirrus, respectively.
  var cloudBitMask = 1 << 10;
  var cirrusBitMask = 1 << 11;

  // Both flags should be set to zero, indicating clear conditions.
  var mask = qa.bitwiseAnd(cloudBitMask).eq(0)
      .and(qa.bitwiseAnd(cirrusBitMask).eq(0));

  return image.updateMask(mask).divide(10000);
}

// Map the function over one year of data and take the median.
// Load Sentinel-2 TOA reflectance data.
var dataset = ee.ImageCollection('COPERNICUS/S2')
                  .filterDate('2018-11-01', '2018-11-15')
                  // Pre-filter to get less cloudy granules.
                  .filter(ee.Filter.lt('CLOUDY_PIXEL_PERCENTAGE', 20))
                  .map(maskS2clouds);

var rgbVis = {
  min: 0.0,
  max: 0.3,
  bands: ['B11', 'B8', 'B4'],
};


Map.addLayer(dataset.median(), rgbVis, 'RGB');

Map.setCenter(74.516872, 30.475603, 14);

var aqua = ee.ImageCollection('MODIS/006/MYD14A1')
                  .filter(ee.Filter.date('2018-11-01', '2018-11-15'));
var fireMaskVis = {
  min: 9.936350984054926,
  max: 107.38312795164352,
  bands: ['MaxFRP', 'FireMask', 'FireMask'],
};

Map.addLayer(aqua, fireMaskVis, 'Fire Mask');

var terra = ee.ImageCollection('MODIS/006/MOD14A1')
                  .filter(ee.Filter.date('2018-11-01', '2018-11-15'));
var fireMaskVis = {
  min: 9.936350984054926,
  max: 107.38312795164352,
  bands: ['MaxFRP', 'FireMask', 'FireMask'],
};

Map.addLayer(terra, fireMaskVis, 'Fire Mask');

Research Note #22 - Reading GCOM-C 250m-Res HDF5 Data with Fortran

There were not so many references about 'simple' ways to open and read HDF5 data file with Fortran as well as compiling the codes. I said 'simple', because there are actually manuals and tutorials about that purpose, however, most of them were too complicated and refer to other sub-references, thus blah .. blah ... it takes too much time, especially if you just simply want to open, read and compile a HDF dataset. Here, I posted a very basic way to access HDF5 data, using very minimal Fortran code. Before starting, here's the convention:

• I used Linux OS. 
• I used Fortran 90 for programming.
• HDF5 (ver 1.8.17) libraries has been already installed in my system as module.
• I used JAXA's GCOM-C NDVI Monthly statistics (Level-2 data) with 250m spatial resolution with Sinusoidal/EQA projection. This data contains 4800x4800 pixels. For exemplary purpose, I renamed it to "ndvi.h5".

So, the following is a very basic example of Fortran code to open and read HDF5 data:

PROGRAM HDF5test

   USE HDF5

   IMPLICIT NONE

   INTEGER :: hdferr, i, j

   INTEGER(HID_T) :: fileid, dset

   CHARACTER(LEN=7), PARAMETER :: filename = "ndvi.h5"

   CHARACTER(LEN=19), PARAMETER :: dataset = "Image_data/NDVI_AVE"

   INTEGER, PARAMETER :: xdim = 4800

   INTEGER, PARAMETER :: ydim = 4800

   INTEGER, DIMENSION(1:xdim, 1:ydim) :: ndvi_ave

   INTEGER(HSIZE_T), DIMENSION(1:2) :: dims = (/xdim, ydim/)

   

   CALL h5open_f(hdferr)

   CALL h5fopen_f(filename, H5F_ACC_RDONLY_F, fileid, hdferr)

   CALL h5dopen_f(fileid, dataset, dset, hdferr)

   CALL h5dread_f(dset, H5T_NATIVE_INTEGER, ndvi_ave, dims, hdferr)

    DO j = 1, ydim

      DO i = 1, xdim

         print *, ndvi_ave(i,j)

      ENDDO

   ENDDO

  CALL h5dclose_f(dset, hdferr)

  CALL h5fclose_f(fileid, hdferr)

  CALL h5close_f(hdferr)

END PROGRAM HDFtest

The most important lines are:

• CALL h5open_f(hdferr). This function initializes HDF5 library for Fortran uses. Returns 0 (to hdferr variable) if it's successful, and -1 if it's failed.
• CALL h5fopen_f(filename, H5F_ACC_RDONLY_F, fileid, hdferr). This function opens an existing HDF5 file. In this example, I just want to read it, hence H5F_ACC_RDONLY_F was used as argument. This function will return integers for file identifier and status to fileid and hdferr variables, respectively.
• CALL h5dopen_f(fileid, dataset, dset, hdferr). This function opens an existing dataset in the opened file. Take notice that it needs file identifier (from h5fopen_f), dataset name, dataset identifier and status (as previous functions).
• CALL h5dread_f(dset, H5T_NATIVE_INTEGER, ndvi_ave, dims, hdferr). This function reads raw data from dataset of the file, into a buffer. The buffer means a space in the computer memory, this could be a variable or array. Take notice that it need dataset identifier (from h5dopen_f), memory data type (Native integer), buffer, dimension (to store buffer dimension) and as always, status (hdferr).

To make it simple, the program will firstly initialize HDF5 library, opens an existing HDF5 datafile, opens a dataset from that file (a file could consist of many datasets), read the raw data from the dataset, put the data into an array buffer, then prints the contents of the buffer into the screen. Finally, it closes the dataset, closes the file and ultimately deactivates HDF5 library (flush and cleans the memory). That's all.

Another important thing is variable declaration. HID_T and HSIZE_T are parameters for predefined datatype used by HDF5 library. So, don't be confused with them (for details, just read: https://support.hdfgroup.org/HDF5/doc/fortran/index.html).

Finally, save and give the codes a name (for example: test.f90). To compile the code, just use this command on (bash) shell:

$ h5fc -o test.exe test.f90

It's just similar with a normal Fortran compiler command. If you don't specify a name for the output, a file named 'a.out' will be generated by the compiler.

Research Note #21 - Compiling Fortran Script for NetCDF Data Processing

NetCDF data is become more and more important for my research, thus it's necessary to work with it with my own program, in Fortran (F90). The compilation anyway, a little bit tricky and slightly different with ordinary Fortran compilation. Here are the simple steps to compile it, using Intel Fortran compiler:

1. Make sure NetCDF module has been installed and declared in the system path.

The path for NetCDF module is : $NETCDF/include

If the path has not yet existed, make a soft link of the module file to current directory:

ln -sf $NETCDF/include/netcdf.mod

2. Compile the Fortran code to make an object file

Let's say the code file name is 'test.f90', then execute:

$ ifort -c test.f90 -I$NETCDF/include

Or, if the path of NetCDF module has existed, just simply:

$ ifort -c test.f90

Make sure no problems occur.

3. Finally, link the object file to an executable file:

$ ifort -o test.exe test.o -I$NETCDF/include -L$NETCDF/lib -lnetcdff -lnetcdf

Same as step 2, if the module path has existed:

$ ifort -o test.exe test.o -L$NETCDF/lib -lnetcdff -lnetcdf

If everything's OK, then an exe file will be created in the current directory. The most important thing, don forget to include '-lnetcdff', because since release 4.1.3, the Fortran library is no longer combined with C library. One hint is by checking if there are file named 'libnetcdff' in the $NETCDF/lib directory. If they are there, the argument '-lnetcdff' should be added in the compile command. 

Research Note #20 - Preparing Emission Data with PREP-CHEM-SRC and CONVERT_EMISS for WRF-Chem

It took quite a while before I could post this, mainly because everything becomes more and more complicated. The deeper I dig into WRF-Chem, the more complex the problems are. Successfully compiling PREP_CHEM_SRC didn't make things easier as more questions keep coming into my mind about how the system actually works, thanks to the oblivious manual. Well, at least the emission data could be prepared and seems to be simulated in the model now.

Here are the steps to prepare emission data for WRF-Chem using PREP-CHEM-SRC and CONVERT_EMISS utilities.

1. Download and Link the Emission Data

Download the latest emission data from this link. It was in the same directory with PREP-CHEM-SRC source codes. Extract the data inside any desired directory and it is recommended to make a soft link to the location to avoid long-directory listing in the input namelist file which will be explained soon. Locate the softlink in the same directory with the PREP-CHEM-SRC utility (and its namelist input file), and name it 'datain' (the name is actually up to you). 

$ mkdir wrf/EMISSION

$ cd wrf/EMISSION

$ tar -xzvf global_emissions_v3_24aug2015.tar.gz

$ cd PREP-CHEM-SRC-1.5/bin

$ ln -sf ../../Global_emissions_v3/ datain

The directory should similar like this (after executing ls command):

[c24223@ofp05 bin]$ ls -alh

total 3.9M

drwxr-s--- 4 c24223 gi55 4.0K Feb  7 19:06 .

drwxr-s--- 8 c24223 gi55 4.0K Aug 25  2015 ..

drwxr-s--- 3 c24223 gi55  12K Jan 31 20:24 build

lrwxrwxrwx 1 c24223 gi55   26 Jan 31 20:24 datain -> ../../Global_emissions_v3/

-rw-r----- 1 c24223 gi55 6.0K Feb  6 20:38 prep_chem_sources.inp

-rwxr-x--- 1 c24223 gi55 3.9M Jan 31 20:22 prep_chem_sources_RADM_WRF_FIM_.exe

drwxr-s--- 6 c24223 gi55 4.0K Apr 30  2015 .svn

2. Configure using Namelist Input File

This step is really important to determine what kind of emission data, map projections, grid configuration, output format etc. used by the utility. These configurations are stored inside namelist input file: prep_chem_sources.inp. 

Ideally, we could run the utility without any changes to namelist file, but unfortunately, it is not the case this time. The main contents of the file contain paths of emission data from step #1, and it seems the developers didn't update it according the latest path in the data. Therefore, you MUST re-check and confirm the path inside the namelist file by your own, otherwise you will get errors while executing the utility.

A sample of 'working' configuration for my case could be download here. 

Some important points are before using the namelist file above:

• The emission dataset used this post is published does not contain MEGAN emission data.
• I didn't use bbem/bbem plumerise and volcanic emission data.
• The map projection used in my simulation is mercator/ii. 
• For more detailed information about the configurations in the namelist file, read this guide (it is much better than the original emission guide of WRF-Chem).

3. Execute prep_chem_src utility

This step is quite straight forward, except that some nasty things could happen during the process. 

$ ./prep_chem_sources_RADM_WRF_FIM_.exe

You could just sit down nicely until the utility finishes preparing the emission data. Anyway, if you find that the utility crashes midway with some scary segmentation fault messages, you should check the compilation of it. One possible cause is, the compiler option used while compiling the utility. For more detail, check the previous post about installing prep_chem_src utility.

Another possible reason is the program overloads the memory because of compilation or grid settings in the namelist file, causing crash. If it's the case, execute 'ulimit -s unlimited' and try again.

Anyway, if the program successfully works, you will see files similar like the following in the directory:

-rw-r----- 1 c24223 gi55  27M Feb  7 20:03 matrixfire-T-2018-01-01-000000-g1-ab.bin

-rw-r----- 1 c24223 gi55  35M Feb  7 20:03 matrixfire-T-2018-01-01-000000-g1-bb.bin

-rw-r----- 1 c24223 gi55 6.5K Feb  7 20:02 matrixfire-T-2018-01-01-000000-g1.ctl

-rw-r----- 1 c24223 gi55 168M Feb  7 20:03 matrixfire-T-2018-01-01-000000-g1-gocartBG.bin

-rw-r----- 1 c24223 gi55  70M Feb  7 20:03 matrixfire-T-2018-01-01-000000-g1.gra

-rw-r----- 1 c24223 gi55  71M Feb  7 20:03 matrixfire-T-2018-01-01-000000-g1.vfm

So, here are some explanations for the files:

• g1-ab.bin file is anthropogenic emission data (+ biogenic emission data, if you use it) for WRF-Chem
• g1-bb.bin file is biomass burning (+ plume rise, if you use it) emission data for WRF-Chem
• g1-gocartBG.bin is GOCART background emission data for WRF-Chem
• g1.gra and g1.vfm are binary files in GrADS format to visualize the emission data
• g1.ctl is GrADS control/descriptor file

You can now check the emission data using GrADS (don't forget to add 'options big_endian' into the descriptor file).

 --- To be continued --- 

Research Note #19 - Installing prep_chem_sources Utility for WRF-Chem

This has been probably one of the trickiest installations I've ever experienced, thanks to the lack of necessary library, limited references and bug in the source code. 

In order to install (or rather, compile) prep_chem_sources tool (I'll call it PCS from now on), HDF5, NetCDF and ZLIB libraries are necessary. Since NetCDF and ZLIB have been already available from previous WRF installation, one should only care about HDF5. Be advised though, HDF5 library of Oakforest super computer module was not compatible with the installation script. Thus, newly fresh HDF5 library is necessary (latest version might be compatible, thus keep checking updates of the super computer).

1. HDF5 Library Installation

Download the latest version from HDF group website. For this case, the version used is HDF5-1.10.4. Extract the file in a directory and execute installation script. Since PCS needs ZLIB and Fortran library, do not forget to mention them during the configuration.

$ ./configure --prefix=/work/gi55/c24223/libs/hdf5 --enable-fortran --enable-cxx --with-zlib=/work/gi55/c24223/libs/grib2

$ ./make

$ ./make check

$ ./make install

Make sure no errors occurred during the installation. After installing HDF5, put its path and LD_LIBRARY_PATH in the .bash_profile and re-load the file.

export HDF5=$PACSLIB/hdf5

export PATH=$HDF5/bin:$PATH

export LD_LIBRARY_PATH= ... :/work/gi55/c24223/libs/hdf5/lib

2. Download PCS tool

Download the tool from this ftp site directory: ftp://aftp.fsl.noaa.gov/divisions/taq/global_emissions

By the time this note was written, the latest version of PCS tool is ver. 1.5. Extract the tools inside any desired directory. Once extracted, there will a directory PREP-CHEM-SRC-1.5 with several sub-directories such as: /bin, /src, /extra etc.

3. Edit  convert_edgar_to_RELACS_REAC.f90 Script

This step should be done, otherwise PCS installation will fail with following messages:

convert_edgar_to_RELACS_REAC.f90(17): error #6405: The same named entity from different modules and/or program units cannot be referenced. [CO]
,CO &
-----------------------------^
convert_edgar_to_RELACS_REAC.f90(20): error #6405: The same named entity from different modules and/or program units cannot be referenced. [CH4]
,CH4 &
-----------------------------^
convert_edgar_to_RELACS_REAC.f90(21): error #6405: The same named entity from different modules and/or program units cannot be referenced. [SO2]
,SO2 &
-----------------------------^
compilation aborted for convert_edgar_to_RELACS_REAC.f90 (code 1)

make: *** [convert_edgar_to_RELACS_REAC.o] Error 1

The cause of this error is ironically simple: the absence of remark symbol ('!') in front of the 9th line of script convert_edgar_to_RELACS_REAC.f90 in the /src sub-directory:

8 subroutine convert_edgar_to_relacs_reac(isp,iespc,ident,spc_name_dummy)  !kml
9 use chem1_list
10 !use chem1_list, only : alke, bio,ora2,aro,ket,alka,ald

Thus, edit the script by adding '!' symbol in front of the said line using vi or other text editor.

4. Edit include.mk.intel.wrf Option Script and Compile PCS

This step is briefly explained on emission guide of WRF-Chem, however no further explanation about the script options available since it will strictly depend on the machine/system architecture. Anyway, one useful hint is: use the same options with WRF-Chem installation used, which are stored inside file configure.wrf inside /WRFV3 sub-directory.

Go to /bin/build sub-directory of PCS, and edit include.mk.intel.wrf option script (since the compiler used is intel compiler). Change these settings as follows:

update 2019/01/31:
Changed F_OPTS because the "longer" version caused crash during execution of utility for GOCART background data.

NETCDF=/work/gi55/c24223/libs/netcdf

HDF5=/work/gi55/c24223/libs/hdf5

HDF5_LIB=-L$(HDF5)/lib -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -L/work/gi55/c24223/libs/grib2/lib -lz -ldl

F_COMP=mpif90 -f90=ifort

C_COMP=mpicc -cc=icc -DMPI2_SUPPORT

LOADER=$(F_COMP)

C_LOADER=$(C_COMP) -DFSEEKO64_OK

F_OPTS=-fpp -D$(CHEM) -O3 -convert big_endian
C_OPTS= -O2

LOADER_OPTS= -O2

Save the script and compile PCS (for GOCART chem option):

$ make OPT=intel.wrf CHEM=RADM_WRF_FIM

If everything's going well, an executable prep_chem_sources_RADM_WRF_FIM_.exe will be created in /bin sub-directory. Check with ldd command to make sure there are no missing shared library links of the executable.

$ ldd prep_chem_sources_RADM_WRF_FIM_.exe

        linux-vdso.so.1 =>  (0x00007ffc35bbe000)

        libnetcdff.so.6 => /work/gi55/c24223/libs/netcdf/lib/libnetcdff.so.6 (0x00007ff8b2825000)

        libnetcdf.so.11 => /work/gi55/c24223/libs/netcdf/lib/libnetcdf.so.11 (0x00007ff8b23cf000)

        libhdf5hl_fortran.so.100 => /work/gi55/c24223/libs/hdf5/lib/libhdf5hl_fortran.so.100 (0x00007ff8b21a8000)

        libhdf5_fortran.so.100 => /work/gi55/c24223/libs/hdf5/lib/libhdf5_fortran.so.100 (0x00007ff8b1f45000)

        libhdf5_hl.so.100 => /work/gi55/c24223/libs/hdf5/lib/libhdf5_hl.so.100 (0x00007ff8b1d1c000)

        libhdf5.so.103 => /work/gi55/c24223/libs/hdf5/lib/libhdf5.so.103 (0x00007ff8b1679000)

        libz.so.1 => /work/gi55/c24223/libs/grib2/lib/libz.so.1 (0x00007ff8b145a000)

        libdl.so.2 => /lib64/libdl.so.2 (0x00007ff8b1237000)

        libmpifort.so.12 => /home/opt/local/cores/intel/impi/2018.1.163/intel64/lib/libmpifort.so.12 (0x00007ff8b0e8d000)

        libmpi.so.12 => /home/opt/local/cores/intel/impi/2018.1.163/intel64/lib/libmpi.so.12 (0x00007ff8b0208000)

        librt.so.1 => /lib64/librt.so.1 (0x00007ff8b0000000)

        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff8afde3000)

        libm.so.6 => /lib64/libm.so.6 (0x00007ff8afae1000)

        libc.so.6 => /lib64/libc.so.6 (0x00007ff8af71e000)

        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ff8af507000)

        libhdf5_hl.so.10 => /work/opt/local/apps/intel/2017.1.132/hdf5/1.8.17/lib/libhdf5_hl.so.10 (0x00007ff8af2db000)

        libhdf5.so.10 => /work/opt/local/apps/intel/2017.1.132/hdf5/1.8.17/lib/libhdf5.so.10 (0x00007ff8aebb3000)

        libifport.so.5 => /home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libifport.so.5 (0x00007ff8ae984000)

        libifcoremt.so.5 => /home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libifcoremt.so.5 (0x00007ff8ae5f0000)

        libimf.so => /home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libimf.so (0x00007ff8ae061000)

        libsvml.so => /home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libsvml.so (0x00007ff8ac9ae000)

        libintlc.so.5 => /home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libintlc.so.5 (0x00007ff8ac740000)

        libirng.so => /home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/libirng.so (0x00007ff8ac3cc000)

        /lib64/ld-linux-x86-64.so.2 (0x00005572db0f7000)

   

Research Note #18 - Installing WRF-Chem with Kinetic Pre-Processor (KPP)

The installation process is basically similar with typical WRF-Chem, with the additional steps to set KPP environment variables and flex library installation. To make it simple, these are the steps to install WRF-Chem with KPP. It assumes that 'yacc' (yet another compiler) has been installed into the system.

1. Install Flex Library

Flex library could be free-downloaded easily from many sources on internet. For this case, GNU flex used could be obtained from this link.

After downloading, extract the files into any desired directory. Next, run configuration script, and compile as usual. Simple as that.

$ ./configure --prefix=$PACSLIB/flex

$ make

$ make check

$ make install

Make sure that file libfl.a exists in $PACSLIB/flex/lib directory after compilation.
More detailed steps could be found here.

2. Setting the Environment Variables

This step is available on WRF-Chem manual. Just set the environment variables on bash shell for these values:

$ export WRF_KPP=1

$ export YACC="/usr/bin/yacc -d"

$ export FLEX_LIB_DIR=$PACSLIB/flex/lib

The most important thing, add the path of flex shared library (libfl.so.2) to the LD_LIBRARY_PATH of .bash_profile, otherwise, the installation of WRF-Chem will fail. For example:

$ export LD_LIBRARY_PATH=/home/opt/local/cores/intel/impi/2018.1.163/intel64/lib:/home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64:/home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64:/home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64:/home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64_lin:/home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/mpi/intel64/lib:/home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/mpi/mic/lib:/home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/ipp/lib/intel64:/home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64:/home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64:/home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/tbb/lib/intel64/gcc4.7:/home/opt/local/cores/intel/debugger_2018/libipt/intel64/lib:/home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/daal/lib/intel64_lin:/home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/daal/../tbb/lib/intel64_lin/gcc4.7:/home/opt/local/cores/intel/compilers_and_libraries_2018.1.163/linux/daal/../compiler/lib/intel64_lin:/work/gi55/c24223/libs/grib2/lib:/work/gi55/c24223/libs/netcdf/lib:/work/gi55/c24223/libs/flex/lib

Re-run the bash_profile script:

$ . ~/.bash_profile

3. Install the WRF-Chem

Finally, install WRF-Chem as described on this previous post. If the bash_profile file has already listed netcdf libraries and impi libraries, just go straight to step #3.

The compilation of WRF-Chem's executables with KPP is somehow longer than typical model. New updates and details will be added after WPS compilation. 

Research Note #17 - WRF-Chem Simulation with ECMWF ERA-Interim Data

Running WRF-Chem with ERA-Interim data is basically similar with GFS data. The main difference is the pre-processing steps. While GFS (FNL or forecast) dataset contains everything the user needs for the pre-processing, ECMWF (ERA-Interim, Era5 etc) dataset requires more efforts from user to determine parameters included in the data before downloading and processing it. It may seems bothersome and complicated, but determination of parameters is really important to understand how WRF model works.

The key point here is WRF's Vtable file, which contains all parameter needed for simulation. Once you understand the contents of the file (in accordance with the type of input data), you can basically use WRF with many kinds of input dataset.

To make long story short, here are steps for running WRF-Chem with ERA-Interim dataset:

1. Downloading Data

• Go to ECMWF website, and log-in to it. An account is required for downloading data, thus if you don't have one, just register a new account. It's free.
• Once you logged in, browse into Forecast > Datasets > Archived Datasets > Reanalysis Datasets, and finally, select Download on ERA-Interim section. You will be directed to ERA-Interim, Daily data page.
• Now, the most important thing, WRF needs two kinds of level type for the data: surface and levels. Aside from surface level, there are at least two kinds of level type could be used for WRF: pressure levels and model levels. Which level type you use will depend on Vtable you use for WRF (wrf directory/ungrib/Variable_Tables). For this example, I use Vtable.ERA-interim.ml, which means I will use model levels type dataset. Another important thing, aside from daily data, you will need Invariant data for the data field. This data is static and not time-varied as daily data, hence you only need to download it once.
• For the surface level data, on Type of Level section, select Surface, and put a tick on the month and year section. Regardless the data period of your interest, you have to download the data for at least one whole month. Next, select the time of the data. If you want to use 6-hourly boundary condition data, put check to all time box. For the step (model cycle), choose 0. Finally, you should choose 18 parameters of the surface level data (which are listed in WRF Vtable file): 2m temperature, 10m U wind component, 10m V wind component, mean sea level pressure, sea surface temperature, sea ice area fraction, skin temperature, snow density, snow depth, surface pressure, soil temperature levels (1,2,3 and 4) and finally, volumetric soil water layers (1,2,3 and 4). Click Retrieve GRIB. The website will summarize your request and if you think it's correct, click Retrieve now, and wait until the download link appears. Depending on the size of the data you requested, this process may take up to 1 minute to hours to complete. All those parameters will be downloaded as single file. Give it suitable name e.g. era-int-201711-sfc.grib.
• For the model levels data, on Type of Level section, select Model levels. The steps are almost similar with surface level data download. This time you only need to select the month, year, time and 5 parameters on all levels: specific humidity, temperature, U component of wind, V component of wind and geopotential. Click Retrieve GRIB, and the rest will be same as surface level data download procedure. Anyway, since geopotential data is only available for one level, you could download it separately from the others. For example, the downloaded data will be named era-int-201711-geo.grib (for geopotential) and era-int-201711-mod.grib (for the other parameters). Therefore, there will be two model levels data files.  
• Finally, don't forget to download the Invariant data. On ERA Interim Fields section, select Invariant, and put check on Land-Sea Mask, then click Retrieve GRIB and so on. Name it into era-int-invariant.grib.
• In the end, based on this example, there are four files you should have from the website: era-int-201711-sfc.grib (surface levels), era-int-201711-geo.grib (model levels geopotential), era-int-201711-mod.grib (model levels other than geopotential) and era-int-invariant.grib (land-sea mask invariant).

2. WRF Pre-processing

• Make sure you have already set the model domain (by geogrid.exe) and set the dynamic link of Vtable to Vtable.ERA-interim.ml file.
• Link all ERA-Interim data downloaded by link_grib.csh script, and execute ungrib.exe. Wait until the process finished.
• The next step after ungrib is very important. ECMWF model levels data uses sigma levels, while WRF requires pressure levels, thus you need to calculate the pressure level of the corresponding sigma level using utility calc_ecmwf_p.exe (WPS/util/calc_ecmwf_p.exe). You have to do this before executing metgrid.exe. Failing to do so may result on model crashes or unrealistic simulation results (as I previously experienced). Assuming this is the first time you use ERA-Interim data, open namelist.wps and add 'PRES' to the list of prefixes in the fg_name namelist variable.

          &metgrid

          fg_name = 'FILE', 'PRES',

• Make a dynamic link of the calc_ecmwf_p.exe to the current working WPS directory and make a text file with this following template, then save and name it: 'ecmwf_coeffs':

    0         0.000000    0.00000000
    1        20.000000    0.00000000
    2        38.425343    0.00000000
    3        63.647804    0.00000000
    4        95.636963    0.00000000
    5       134.483307    0.00000000
    6       180.584351    0.00000000
    7       234.779053    0.00000000
    8       298.495789    0.00000000
    9       373.971924    0.00000000
   10       464.618134    0.00000000
   11       575.651001    0.00000000
   12       713.218079    0.00000000
   13       883.660522    0.00000000
   14      1094.834717    0.00000000
   15      1356.474609    0.00000000
   16      1680.640259    0.00000000
   17      2082.273926    0.00000000
   18      2579.888672    0.00000000
   19      3196.421631    0.00000000
   20      3960.291504    0.00000000
   21      4906.708496    0.00000000
   22      6018.019531    0.00000000
   23      7306.631348    0.00000000
   24      8765.053711    0.00007582
   25     10376.126953    0.00046139
   26     12077.446289    0.00181516
   27     13775.325195    0.00508112
   28     15379.805664    0.01114291
   29     16819.474609    0.02067788
   30     18045.183594    0.03412116
   31     19027.695313    0.05169041
   32     19755.109375    0.07353383
   33     20222.205078    0.09967469
   34     20429.863281    0.13002251
   35     20384.480469    0.16438432
   36     20097.402344    0.20247594
   37     19584.330078    0.24393314
   38     18864.750000    0.28832296
   39     17961.357422    0.33515489
   40     16899.468750    0.38389215
   41     15706.447266    0.43396294
   42     14411.124023    0.48477158
   43     13043.218750    0.53570992
   44     11632.758789    0.58616841
   45     10209.500977    0.63554746
   46      8802.356445    0.68326861
   47      7438.803223    0.72878581
   48      6144.314941    0.77159661
   49      4941.778320    0.81125343
   50      3850.913330    0.84737492
   51      2887.696533    0.87965691
   52      2063.779785    0.90788388
   53      1385.912598    0.93194032
   54       855.361755    0.95182151
   55       467.333588    0.96764523
   56       210.393890    0.97966272
   57        65.889244    0.98827010
   58         7.367743    0.99401945
   59         0.000000    0.99763012
   60         0.000000    1.00000000

• Execute the utility and wait until it completes calculating the pressure levels. This results new files with prefix 'PRES' in current directory. Once finished, you can execute metgrid.exe. This will complete the pre-processing steps.

3. Running WRF

• Open namelist.input in WRF's run directory, and pay attention to these following variables, since they are closely related to ECMWF data, e.g. number of metgrid pressure levels. Modify the values as follows:

&domains  
num_metgrid_levels = 61 [default 32]

&physics
surface_input_sources= 1 [default 3]

• Execute real.exe and finally, wrf.exe to run the model.  

That's it. This post will be updated once something new occurred in the future.

Research Note #16 - WRF Errors Showcase

I've been working with WRF model for about 4 years, which is actually not so long for someone who has been studying atmospheric science for almost 20 years. During that time, I've switched from WRF-EMS (currently known as STRC-UEMS), WRF-ARW and finally WRF-Chem. While it's quite a short period, I've experienced the mess with WRF so many times, and I think it's time to recap those issues, not just for assisting my own future research, but for others who are studying in the same field.

FYI, I've run WRF model using supercomputer of UTokyo, Oakforest PACS, with the total 8208 computation nodes and peak performance of 25 PFLOPS. Anyway, I usually only use 16 nodes with 32 MPI processes for each simulation batch jobs, because I think it's more than enough.

Without further ado, here are the most common errors I've ever experienced with WRF model. Oh yeah, I will not list errors due to typos in the namelist.input (e.g. you forgot to match the grid resolution between namelist.input and namelist.wps, forget to put comma, period etc.).

1. CFL Error

Symptoms: 

• WRF generates messages such as : "x points exceeded cfl=x in domain d0x at time ..." 
• Simulation speed degrades or simulation completely stops.

Causes:

• Model becomes unstable, mostly because the time-step used is too large for stable solution, especially while using high-res simulation grids.
• Conflicts among model physics/dynamics/domains configuration.

Solutions:

• Decrease the time-step (namelist.input > &domain > time_step).  The most common used convention is 6*DX in kilometers. That means, if the grid resolution is 10 km, then use at least 60 seconds time step. If the messages still appear, decrease the time step to 30 or 10 seconds.
• Check the parameterization/configuration used in namelist.input which could potentially cause conflicts or model crash. I usually discard some parameterization schemes, and check them individually to see if I they are the causes of the crash.        

2. Flerchinger Messages

Symptoms:

• WRF generates messages such as: "Flerchinger USEd in NEW version. Iterations= x"
• Simulation still runs but the speed degrades.
• WRF restart files are not generated.

Causes:

• It's basically not an error, but a message generates by Noah LSM (namelist.input > &physics > sf_surface_physics), when the model output soil temperature which is super low/negative soil moisture. I experienced this while running simulation over Russia, with lon and lat > 55 degree. 

Solutions:

• Change the surface physics to other options
• Change the input data. I changed from GFS FNL ds083.2 into ECMWF ERA-Interim, and I've never met such messages ever again, even if I still use Noah LSM.

3. No WRF Simulation Log (rsl.out or rsl.error) Generated

Symptoms:

• While running real.exe or wrf.exe, there are no logs of simulation. Instead of log file, the steps are shown on the screen (stdout).

Causes:

• Well ... while it's quite strange and stupid (which I experienced), but this is absolutely not an error. It happens when you compiled WRF binaries for serial computation, instead of parallel ones (dmpar or smpar).

Solutions:

• Recompile the model using dmpar or smpar or both, then you'll get your logs back.

4. Real.exe Error : Interpolation Order Error

Symptoms:

• While running real.exe, the process is stopped at certain point, indicating that there are to few data for the interpolation order with Real.exe

Causes:

• I'm not so sure about this, but I think that was caused when real.exe finds that the data for vertical interpolation is inadequate for model run. This was happened when I use GFS FNL ds083.2 for simulation over Russia, and I got so many warnings about missing level data, before this error occurred.

Solutions:

• Change the data. I used ECMWF Era-Interim and the errors were gone.

5. WRF Simulation Sudden Death

Symptoms:

• Model crashed. Real.exe and Wrf.exe are abruptly stopped, without any error messages in log files. It just stop.

Causes:

• I hate this error because it might caused by many factors, but mostly because of the conflicts within the model configuration. For example, I was using WSM-3 MP parameterization schemes, with RRTM schemes for LW and SW, with domain over high latitude and complex terrain, 10 km resolution, using several computation nodes, then many strange things happened: the model crashed many times, could only stable while running on single node, etc.  
• On several cases, it's also caused by too large time-step similar with CFL error.
• Sometimes, it also occurs if the domain is too large, in particular when grid size < 10 km with complex terrain.

 Solutions:

• Change the model configuration. For my case, I used Lin MP scheme with new RRTM schemes, and the error was gone.
• Reduce the time-step in the factor of 2 (half of time-step first, if still not works, try 0.25 of the original time-step, and so on).
• Reduce the domain size. 

6. Mismatch Landmask ivgtyp

Symptoms:

• Real.exe crashed, giving error message: 

-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE: LINE: 2963
mismatch_landmask_ivgtyp
-------------------------------------------

Causes:

• By far, only occurred while using ECMWF Era-Interim dataset. The reason is unknown. It's documented on WRF-ARW website.  

Solutions:

• Change the value of 'surface_input_source' on &physics parameter of namelist.input from '3' to '1'

7. Ungrib.exe Segmentation Fault (End Date)

Symptoms:

• Ungrib.exe crashes at the ungribbing process at the end date, giving error message: "Segmentation fault ..."

Causes:

• Probably, it has something with computer memory, because when I set ulimit to unlimited, the problem was solved.

Solution:

• ulimit -s unlimited 

8. Metgrid.exe error in ext_pkg_write_field

Symptoms:

• Metgrid.exe crashes at the beginning of the process with messages: 'ERROR: Error in ext_pkg_write_field'.

Causes:

• This will happen when new NCEP GFS data (Version 15.1 or higher) was process using old version of ungrib.exe (< Ver. 4).

Solution:

• Install Ungrib from WPS Ver.4. (the old geogrid/metgrid still could be used).