Meteo #26 - Saving Multi-variable Data into GrADS-gridded Binary File with GrADS

The title might be a little bit confusing, but what I want to share in this post is about how to save data, say, from a certain data format into another one which has GrADS format. For example, you open a NetCDF (nc) data with 5 variables in it with GrADS, and you want to save them into GrADS binary file for further analysis with the tool. 

Again, why use GrADS to do such task when you can do it by programming for example FORTRAN, C or others? Well, that's because it's efficient to do the task with GrADS only. You don't need to write a program, compile it or debug it over and over again, hence saving much of your precious time. Anyway, before doing so, one should understand the grids order in the GrADS-gridded binary file.

GrADS-gridded binary file is actually an ordinary binary file with a certain order. It doesn't have header/metadata to describe its dimensions or variables to the tool or user who wants to read it. That's why you need a descriptor or control file (CTL) in order to open the data with GrADS. In other words, you'll need to understand how the data is ordered in the file to differentiate dimensions and variables before trying to read it.

Imagine a 1-D data of a time series, for example, hourly air temperature (T) for 9 hours. The data will have 9 records which represent the time (hour), with their own values which represent the T at each time.

29, 30, 30, 31, 30, 29, 30, 31, 30

By looking at the data, we knew that the T for the 1st hour is 29, then 30 for the 2nd, and so on. That's exactly how the data stored in a binary file. It doesn't contain any information about the exact time (in real world), but we knew that the first record is the value of T at the 1st hour because it's already explained in previous paragraph that the data is a 1-D time series of T for 9 hours. If the data is just shown 'raw' as it is without any description, nobody will know what kind of information it contains since 29 or 30 could mean anything other than temperature (e.g. age, or number of apples on the tree, etc.). 

So, the key word here is 'description' about the data, which gives information to the user to interpret its contents. 

Then, how about if the description says that it's not a 1-D data, but rather a 2-D (spatial) data of air temperature at a time, for example 12 AM? Let say, the records are interpreted with gridded or matrix structure for real world coordinates like this (remember, it's still the same record as before):

30 31 30

31 30 29

29 30 30

Then we knew that the 1st three records in the data (29, 30 and 30) is located at the lowest row, while the first members of each three records (29, 31 and 30) are located at the leftmost column. If we give each record an x-y coordinate, the grid should be like this:

29 (y1,x1), 30 (y1,x2), 30 (y1,x3), ... , 30 (y3,x1), 31 (y3,x2), 30 (y3,x3)

It's clear from those two examples that a binary data file is merely a sequential blocks of data. 1-D, 2-D or even 5-D data will always be treated with sequential order by computer. What makes them different to each other is the 'description' which explain the 'rules' of the order sequence of data in the file. From previous example, we knew that, even the records is the same, it will have different interpretation based on the description about its contents. For 1-D data, all records is interpreted as values with 9 time stamps, while for 2-D data, the records follow matrix structure to give each values x-y coordinates with only 1 time stamp. To make it simple, the order is like this:

1-D data ---> Time, Value

2-D data ---> Time, y-value, x-value

Back to GrADS, the tool also has certain rules to treat gridded-binary data, and a user needs to follow such rules in order to make GrADS save or read data into its binary format. GrADS can save/read up to 5-D gridded data with the following order:

Ensemble, Time, Variable, z, y, x

If you want, for example, saving 2-D (e.g. 2x2 grid) data with 2 different variables for 2 hours to GrADS-gridded binary format. 

Variable 1 at hour 1:  A (y1,x1), B (y1,x2), C (y2,x1), D (y2,x2)

Variable 1 at hour 2:  E (y1,x1), F (y1,x2), G (y2,x1), H (y2,x2)

Variable 2 at hour 1: I (y1,x1), J (y1,x2), K (y2,x1), L (y2,x2)

Variable 2 at hour 2: M (y1,x1), N (y1,x2), O (y2,x1), P (y2,x2)

then you should save the data with this order: 

[Hour 1: Variable 1 : y1,x1,y1,x2,y2,x,1,y2,x2], [Hour 1: Variable 2 : y1,x1,y1,x2,y2,x,1,y2,x2], [Hour 2: Variable 1 : y1,x1,y1,x2,y2,x,1,y2,x2], [Hour 2: Variable 2 : y1,x1,y1,x2,y2,x,1,y2,x2]

As the result, the binary file contents will have order like this (with values):

A, B, C, D, I, J, K, L, E, F, G, H, M, N, O, P

It might be confusing at the beginning, but once you understand the pattern, everything will make sense and pretty easy to follow.

Here's an example GrADS script for saving time series (1-D, 744 hours) data of 5 variables into a binary file:

'reinit'
'open aerosol.ctl'
'set fwrite aerosol_ts.dat'
'set gxout fwrite'
'set x 1'
'set y 1'
'set z 1'
timer=1
while(timer<=744)
 say 'writing fields to file on t: ' timer
 'set t 'timer
 'd tloop(aave(dustload5,lon=76.2,lon=78.2,lat=27.6,lat=29.6))'
 'd tloop(aave(msa,lon=76.2,lon=78.2,lat=27.6,lat=29.6))'
 'd tloop(aave(dms,lon=76.2,lon=78.2,lat=27.6,lat=29.6))'
 'd tloop(aave(pm25,lon=76.2,lon=78.2,lat=27.6,lat=29.6))'
 'd tloop(aave(so2,lon=76.2,lon=78.2,lat=27.6,lat=29.6))'
 timer=timer+1
endwhile
'disable fwrite'

Notice that all variable (fields) needs to be written into the file before moving to the next time stamp.

In order to open the binary file created from the script, you should follow the variable order again in the CTL file to make GrADS understand it. If you mess with the order, you still can read it by GrADS, but the results might be strange and confusing (e.g. dms may be interpreted as msa by GrADS). Here's the example CTL file to open the previously made binary file:

DSET ^aerosol_ts.dat

TITLE This is experimental

UNDEF 99999.0

XDEF 1 LINEAR 1 1

YDEF 1 LINEAR 1 1

ZDEF 1 LINEAR 1 1

TDEF 744 LINEAR 00Z01MAY2017 1hr

VARS 5

dustload5 0 99 Total dustload

msa 0 99 MSA

dms 0 99 DMS

pm25 0 99 PM2.5 Aerosol

so2 0 99 Sulphur Dioxide

ENDVARS

If you display the result in GrADS, it may look like this (e.g. variable dustload5, with few 'cosmetics' for display):