12-3-03

		Information on Data Preparation steps for Daymet data
		
		
Before running Daymet cases, you need to retrieve and reformat two data sources:

  1.  NCDC TD3200 Cooperative Summary of the Day data.
  2.  SNOTEL Data Network from the Natural Resources Conservation Service.

------------------
A. DATA RETRIEVALS
------------------

  1.  NCDC Data Retrievals
  ------------------------

Data was purchased from NCDC on a 3-CD set, currently in Craig's office.  The 
original set contains data from 1880's (when available, most are 1940's) through
2001.  Updates are expected annually.

This data is retrieved via the on-disk interface, one state at a time.  For this
case, the collection period was 1997-2001 (5 years), to allow one year of overlap
from previous runs of Daymet.  History files (metadata) are also retrieved off the
CDs for each state.

The state-by-state files and meta data are saved at:

	/cgd/tss/craigh/daymet/data/ncdc/ncdc97-01


  2. SNOTEL Data Retrievals
  -------------------------

SNOTEL data is available freely via the Internet.  The main Web page for information
on the SNOTEL Network is:

	http://www.wcc.nrcs.usda.gov/snotel/

Files were ftp'ed from the Web link:

	ftp://ftp.wcc.nrcs.usda.gov/data/snow/snotel/snothist/

Data files are named snotxxyy.txt, where xx is the state abbreviation and yy is the year.
Metadata files are named listxx.txt.

The SNOTEL raw files (data and metadata) for 1997-2001 are saved at:

	/cgd/tss/craigh/daymet/data/snotel


--------------------
B. DATA REFORMATTING
--------------------

The steps involved in reformatting the data for Daymet runs include:

  1.  snotel_merge - to collect state-by-state SNOTEL data into single files
  2.  ncdc_cdextract - to generate .avsm2 tile-by-tile files
  3.  ncdc_snotel_append - to merge the two data sets.
  
  1. Running snotel_merge
  -----------------------
  
The executable is generated from running 'make' on the C program in:

	/fs/cgd/data0/craigh/daymet/src/data_prep.d/src.d/
	
The executable is copied to the snotel directory (/cgd/tss/craigh/daymet/data/snotel)
and run by the line command 'snotel_merge'.  This results in new files of the form

	snotout.zzzz,  zzzz = prec, tmax, tmix, tavg, pill
	snotout_merge_stats.txt  :  summary statistics file

The new files are left in the run directory, /cgd/tss/craigh/daymet/data/snotel.

  2. Running ncdc_cdextract_cmd
  -----------------------------
  
The executable is generated from running 'make' on the C program in:

	/fs/cgd/data0/craigh/daymet/src/data_prep.d/src.d/

The executable is then copied to the bin directory,

	/cgd/tss/craigh/daymet/scripts/data_prep/bin/

Runs are performed from a Perl script, 

	/cgd/tss/craigh/daymet/scripts/data_prep/ncdc_cmd.pl,

which uses data infiles in the same directory, ncdcX.dat, where X = 1-4.  These are the
tile lists and states within each tile.  The task is further automated by using a batch
command to execute the Perl script for each dat input list.  The batch scripts are:

	/cgd/tss/craigh/daymet/scripts/data_prep/ncdc_cmdX/ncdc_cmdX.script, where X = 1-4.
	
Output is saved based on the parameters in the Perl script, in this case in the directory

	/cgd/tss/craigh/daymet/data/ncdc/avsm2_nosnotel/
	(NOTE: This is a copy of the original output in .../avsm2/)

Below this directory is another directory structure with naming convention:

	./XXN-YYYE/XXN-YYYE_ZZZZ_yyyy.avsm, where XX and YYY are lat and lon of the tile
						  ZZZZ are the parameters TMAX/TMIN/PREC/WTEQ
						  yyyy is the year

  3. Running ncdc_snotel_append
  -----------------------------

Once the NCDC data has been reformatted to avsm files and the SNOTEL data merged, it is
necessary to append the SNOTEL data to the NCDC files.  This is done using ncdc_snotel_append.
The executable is generated from running 'make' on the C program in:

	/fs/cgd/data0/craigh/daymet/src/data_prep.d/src.d/

The executable is then copied to the bin directory,

	/cgd/tss/craigh/daymet/scripts/data_prep/bin/

Runs are performed from a Perl script, 

	/cgd/tss/craigh/daymet/scripts/data_prep/snotel_append.pl

The task is further automated by using a batch command to execute the Perl script, which is
located in the SNOTEL data directory:

	/cgd/tss/craigh/daymet/data/snotel/snotel_append.script

This script uses a data file for the tile list located in

	/cgd/tss/craigh/daymet/scripts/data_prep/infiles.d/snotel_tiles.dat
	
Output is saved in the original file names for the NCDC avsm2 data in

	/cgd/tss/craigh/daymet/data/ncdc/avsm2/

with the same subdirectory structure as before.