GeoCLIM/ToDo

From CHG-Wiki
Revision as of 21:28, 30 October 2013 by Diego (talk | contribs)
Jump to navigationJump to search

Team Discussion/Tracking on GeoCLIM development

Latest GeoCLIM download

  • GeoCLIM - updated 21 Oct 2013 [1] Tamuka100 08:28, 21 October 2013 (PDT)
  • GeoCLIM - updated 18 Oct 2013 [2]
  • GeoCLIM - updated 16 Oct 2013 [3] Tamuka100 08:11, 16 October 2013 (PDT)
  • GeoCLIM - updated 14 Oct 2013 [4] Tamuka100 04:13, 14 October 2013 (PDT)
  • GeoCLIM - updated 9 Oct 2013 [5] Tamuka100 15:06, 9 October 2013 (PDT)
  • GeoCLIM - updated 5 Oct 2013 [6] Tamuka100 12:26, 5 October 2013 (PDT)
  • GeoCLIM - updated 3 Oct 2013 [7] Tamuka100 13:39, 3 October 2013 (PDT)
  • GeoCLIM - updated 30 Sep 2013[8] Tamuka100 03:02, 30 September 2013 (PDT)
  • GeoCLIM - updated 25 Sep 2013[9] Tamuka100 10:19, 25 September 2013 (PDT)
  • GeoCLIM - updated 20 Sep 2013[10]


Update request Oct-28-2013

This is the new 'master T' prioritized request list, blending input from ICPAC, Diego, Chris and Gideon.

  • on the climatological analysis tool, add field to specify file name

Users would like to save results to different files, corresponding, for example, to different time periods. For example, it would be nice to create mean 1981-1998 and 1999-2012 rainfall files. So, in the lower right part of the dialog, please add an editable text box displaying the default filename, which can be edited by users if desired.

  • rationalization of missing values

Can this be cleaned up? Users need to spend too much energy thinking about missing values. Pete is going to make -9999 the CHIRPS missing value. From your end, can you replace all the missing value definitions as 'Missing values <= XXXX' where XXXX is set to -900 as a default. Missing values should be presented as <= -900, this will cover cases where there are different missing values. the user will be able to change the maximum missing value. This changes should take place in the BASIICS form, for both the grids and the stations, and on the contour form.

  • Sounds good. I will make all values default to -9999. Tamuka100 06:44, 30 October 2013 (PDT)
  • add 'Frequency' of events

The idea of this addition to the climatological analysis dialog would be to make it easy to explore the frequency of different events, such as the number of seasons receiving more than 500 mm of rain, or the number of months with minT less than 10C. This could be used to identify climatological risk (i.e. drought prone areas), and also used to explore changes over time. you could look at the number of months with Tmax greater 24C, if you were interested in negative warming impacts on coffee, or the number of months with P less than 75 mm to explore dry spells.

The basic display should be a map with the number of 'hits' per decade (so you can compare different time periods, say 1981-1998 and 1999-2012). A corresponding frequency map (the inverse of the frequency times ten) would provide the number of months or season between events .... So a dry season might have been a 1-in-5 year event, but has now become a 1 in 2 year event.... i.e. expected every other year. Similar, but different, from ranking, which we can worry about in 2014 :+)

  • There are two levels to this, one very easy, and one rather difficult, and will be time-consuming. The first one is to answer the question: how many seasons (e.g. MAM, March-only) have we had rainfall of less than 500mm. This will be very easy to program because it is very similar to existing functionality within the program. The second aspect, which is not as easy is how many months during MAM did we have rainfall less than e.g. 100. The reason for this difference is that there are currently no functions or code built into the climatological analysis for handling this type of intra-seasonal analysis. It can be done, but it will take time. Tamuka100 06:44, 30 October 2013 (PDT)


  • tool crashes too often

can we make it more bullet proof?

  • the only way (I can think of) of making it more bullet proof is a rigorous testing phase, and each crash unfortunately has to be dealt with on a case by case basis. This will take time, as well as depend on how rigorously the testing is done. I get very few crashes because I know exactly how to enter everything correctly (some crashes come about because of unhandled errors after incorrect input), and also I am working on a computer with lots of RAM, so I will hardly experience memory problems, which may be another cause of some crashes. If Diego (and other testers) can keep testing, and noting down the exact cause and/or circumstance of each crash, that would go a long way towards bullet-proofing. Perhaps we can get a team of beta testers (volunteers) who need to extensively document the cause and/or circumstances of each crash, the detailed error report generated by the software, and the specs of their system. Tamuka100 06:44, 30 October 2013 (PDT)


Latest features/updates

  • Added IGAD, Sudan and South Sudan regions Tamuka100 08:28, 21 October 2013 (PDT)
  • Fixed error which caused a crash when saving region with a new name Tamuka100 08:28, 21 October 2013 (PDT)
  • Fixed a big which caused a crash when re-installing and user decides to keep current settings Tamuka100 08:28, 21 October 2013 (PDT)


  • modified the change-in-average tool so that outputs are expressed in units per decade, and are comparable to the trend tool. Tamuka100 07:50, 18 October 2013 (PDT)
  • removed incomplete tools in the GeoCLIM, including the graphing tool and the delete button in the spatial data viewer Tamuka100 07:50, 18 October 2013 (PDT)
  • updated the about box Tamuka100 07:50, 18 October 2013 (PDT)


  • removed upload options menu item Tamuka100 08:11, 16 October 2013 (PDT)
  • removed several irrelevant menu options Tamuka100 08:11, 16 October 2013 (PDT)
  • fixed a bug in the contouring routine to allow contouring of 32-bit raster files Tamuka100 08:11, 16 October 2013 (PDT)
  • Added an option to allow showing of trends in mm/decade and degrees/decade Tamuka100 08:11, 16 October 2013 (PDT)


  • Changed climatological analysis so that if the user is analyzing rainfall, it says "Add up seasonal totals", but if analyzing temperature, it says "Calculate seasonal averages" Tamuka100 04:13, 14 October 2013 (PDT)
  • Took out range calculations Tamuka100 04:13, 14 October 2013 (PDT)
  • Modified program to show both slope (trend) and r-squared when doing a climatological trend analysis. Programmed the software so that if the user analyzes rainfall, the program shows a legend title "mm/season", and for temperature, it shows "degrees/season". For rainfall analysis the units in the grid (mm/season) are actual, but for temperature, (degrees/season) the grid is multiplied by 100 (though the legend shows actual degrees) . Tamuka100 04:13, 14 October 2013 (PDT)
  • Got rid of max slope in the smart trends analysis Tamuka100 04:13, 14 October 2013 (PDT)
  • changed the "Run BASIICS intrepolation" title to “blend grids/rasters with stations' Tamuka100 04:13, 14 October 2013 (PDT)
  • changed 'IDW interpolation' to 'insterpolate just stations' Tamuka100 04:13, 14 October 2013 (PDT)
  • changed 'for previous' functionality in the BASIICS to "for next", and fixed the units Tamuka100 04:13, 14 October 2013 (PDT)
  • put an error handler that tells the user to specify a header row. Tamuka100 04:13, 14 October 2013 (PDT)
  • took out all references to “unbias before running IRE” Tamuka100 04:13, 14 October 2013 (PDT)
  • removed user options for cointerp methods to leave only the default options Tamuka100 04:13, 14 October 2013 (PDT)


  • fixed view rainfall summaries module Tamuka100 15:06, 9 October 2013 (PDT)
  • added Africa region for easy creation of chirps-compatible averages Tamuka100 15:06, 9 October 2013 (PDT)
  • added an error-handling routine to the contouring module to help identify incorrect missing values which may cause crashes when running the module Tamuka100 15:06, 9 October 2013 (PDT)
  • fixed the GeoCLIM data specifications for the BASIICS routine Tamuka100 15:06, 9 October 2013 (PDT)


  • Fixed bugs in the startup and importing process Tamuka100 12:20, 5 October 2013 (PDT)
  • Fixed the "Update GeoCLIM averages" option in the climatological analysis Tamuka100 12:20, 5 October 2013 (PDT)
  • Fixed 'list missing data' in the available rainfall data window Tamuka100 12:20, 5 October 2013 (PDT)


  • Fixed a bug that made the program show an error before displaying a map Tamuka100 13:39, 3 October 2013 (PDT)
  • Incorporated missing values into the basic analyses, which also fixed the contouring problem and high values. Tamuka100 13:39, 3 October 2013 (PDT)
  • Added splash screen for better user interaction while the program is loading up (still buggy) Tamuka100 13:39, 3 October 2013 (PDT)


  • incorporated GeoCLIM regions for EAC, GHA and SADC Tamuka100 03:02, 30 September 2013 (PDT)
  • changed the download module to make it default to monthly downloads and automatically select rainfall Tamuka100 03:02, 30 September 2013 (PDT)
  • A prompt allowing users to delete all previous GeoCLIM settings when re-installing. Tamuka100 03:02, 30 September 2013 (PDT)
  • Fixed a bug in the setup menu which caused premature closing of the menu when setting up the data Tamuka100 03:02, 30 September 2013 (PDT)


  • added module to download CHIRPS monthly, dekadal and pentadal files Tamuka100 10:36, 25 September 2013 (PDT)
  • added module to prompt the user options to download or import if no datasets are found. Especially useful in a brand new installation. Tamuka100 10:36, 25 September 2013 (PDT)
  • GeoCLIM can now take in datasets from several different file-naming conventions and any folder on the computer. This should mainly be done under the setup tool, but is also accessible from the File menu
  • GeoCLIM can now handle rainfall, temperature (avg, max and min) and PET analysis in daily, pentadal, dekadal and monthly formats. I have only tested rainfall dekadal and monthlies
  • All references to unbiasing have been removed
  • All references to RFE have been made either (a) more generic for climate surfaces in general (b) refer explicitly to CHIRP or
  • IRE has been renamed to BASIICS: - Background-Assisted Station Interpolation for Improved Climate Surfaces: to reflect usage with other datasets such as temperature, and to reflect an improvement over the traditional distance-only-based interpolation techniques
  • File-new-dataset option is now linked to an appropriate function


Data for GeoCLIM

  • CHIRPS monthly archive - 0.2 degrees; GeoCLIM-compatible [11]


Decisions

  • Windows XP compatibility will be maintained


To-dos

Action: Diego

  • COMPLETED (2013/09/26) send Tamuka some (any) max and/or min temperature station data to allow testing of the BASIICS routine for temperature. Alternatively you can test the routine from there and provide feedback Tamuka100 03:19, 25 September 2013 (PDT)
  • COMPLETED (2013/09/25): compile, zip up and send to Tamuka shapefiles of CILLS, GHA, and the five countries in the EAC: Kenya, Uganda, Burundi, Rwanda and Tanzania. shapefiles shold be in 1st, 2nd and 3rd level admin boundaries where possible
  • COMPLETED (2013/09/25): compile, zip up, and send to Tamuka tmax and tmin climatologies.

Action: Tamuka

  • COMPLETED (2013/10/02) Fix error that pops up and requests user to continue when using the climatological analysis or the change in average. Tamuka100 04:15, 2 October 2013 (PDT)
  • COMPLETED (2013/10/03) Improve/incorporate handling of missing data in all the components (climatological analysis, change in average etc), and dynamically in the color table. Tamuka100 04:15, 2 October 2013 (PDT)
  • COMPLETED (2013/10/03) Investigate incredibly high averages occurring when making averages using 1981-1990 for Kenya MAM (probably related to missing values (9999 in CHIRPS). Tamuka100 04:15, 2 October 2013 (PDT)
  • incorporate temperature analysis more fully into GeoCLIM. Tamuka100 03:13, 30 September 2013 (PDT)
  • Incorporate month option into View Rainfall Summaries module. Tamuka100 04:15, 2 October 2013 (PDT)
  • COMPLETED (2013/09/30) create GeoCLIM regions for GHA, EAC and CILLS (regional and national) and incorporate into installation.
  • COMPLETED (2013/09/25) incorporate CHIRPS ftp folder for direct download: ftp://chg-ftpout.geog.ucsb.edu/pub/org/chg/products/CHIRPS/bils/

Action: Chris

  • COMPLETED (2013/09/25) provide guidance on bulletproofing and on download CHIRPS archive button. Tamuka100 03:19, 25 September 2013 (PDT)


Questions/clarifications/comments/suggestions

  1. I think we should also generate a sample folder with various small datasets as part of a training kit for participants. Tamuka100 03:19, 25 September 2013 (PDT)


GeoCLIM Tips and Tricks

Running GeoCLIM on a computer where you don't have administrative privileges.

  • Don't. This can lead to several errors/crashes while running the software.

Uninstalling and Re-installing GeoCLIM

  • If you want to reinstall GeoCLIM, first uninstall the existing version. If you dont have data that you want to keep, it is better to delete the GeoCLIM folder in My Documents so that you can get a fresh install.

How to make a GeoCLIM archive

If you do want to create CHIRPS archive files (which will be a good idea for when you are distributing the program), the archive is created as follows:

  • the CHIRPS (or other) data should be put in a single directory with a descriptive name (e.g. "CHIRPS_Africa_Monthly"),
  • a text file with the filename "_dataset" (no extension) should be created in the same directory. The text in the "_dataset" file should be as follows:
BEGIN DATASET
 DATASETNAME CHIRPS_Monthly
 DATATYPE ppt
 DATAPREFIX chirps
 DATADATEFORMAT YYYYMM
 DATASUFFIX .bil
 DATAMISSINGFROM -99
 DATAMISSINGTO -99
END DATASET
  • Make the appropriate adjustments to the _dataset file so that it matches the actual situation in terms of names etc
  • The directory with data should then be zipped up, and the .zip extension should be changed to .climdata
  • In the above example, the files would be called e.g. chirps201304.bil, which would be the chirps file for April 2013(2013-04).
  • There are 5 data types with different codes (in brackets below), namely:
    • precipitation (ppt), average temperature (tav), maximum temperature (tmx), minimum temperature (tmn) and potential evapotranspiration (pet)

a problem creating archives Oct 30th

  • we noticed that when creating the archives the tool ran well but the dataset directory was not red when using the 'select dataset' tool. the reason was that the _dataset file that was included in the data directory, did not loose the txt ending. even though it was deleted before saving the file, it kept it. so I had to go to DOS and copy the _datset.txt fiel into _datset. then it worked fine.
  • the only problem that we could not solve today was setting the path for averages using the new imported data from the archive. the tool does not let us make the change

DATADATEFORMAT codes for _dataset file

Below is a detailed list of codes for datadateformats which should be used for the different filenaming conventions in the _dataset file for the archives:

Days

  • YYYYMMDD: 4-digit year; 2-digit month; 2-digit day
  • YYYYMMD: 4-digit year; 2-digit month; 1-digit day
  • YYYYJJJ: 4-digit year; 3-digit Julian day (001-366)
  • YYYYJ : 4-digit year; 1-to-3-digit Julian day (1-366)
  • YYMMDD : 2-digit year; 2-digit month; 2-digit day
  • YYMMD : 2-digit year; 2-digit month; 1-digit day
  • YYJJJ : 2-digit year; 3-digit Julian day (001-366)
  • YYJ : 2-digit year; 1-to-3-digit Julian day (1-366)
  • MMDD : 2-digit month; 2-digit day [Averages]
  • MMD : 2-digit month; 1-digit day [Averages]
  • JJJ : 3-digit Julian day (001-366) [Averages]
  • J : 1-to-3-digit Julian day (1-366) [Averages]

Pentads

  • YYYYMMP: 4-digit year; 2-digit month; 1-digit pentad
  • YYYYPT : 4-digit year; 2-digit pentad (01-72)
  • YYMMP : 2-digit year; 2-digit month; 1-digit pentad
  • YYPT : 2-digit year; 2-digit pentad (01-72)
  • MMP : 2-digit month; 1-digit pentad [Averages]
  • PT : 2-digit pentad (01-72) [Averages]
  • P : 1-or-2-digit pentad (1-72) [Averages]

Dekads

  • YYYYMMK: 4-digit year; 2-digit month; 1-digit dekad
  • YYYYEK : 4-digit year; 2-digit dekad (01-36)
  • YYMMK : 2-digit year; 2-digit month; 1-digit dekad
  • YYEK : 2-digit year; 2-digit dekad (01-36)
  • MMK : 2-digit month; 1-digit dekad [Averages]
  • EK : 2-digit dekad (01-36) [Averages]
  • K : 1-or-2-digit dekad (1-36) [Averages]

Months

  • YYYYMM: 4-digit year; 2-digit month
  • YYYYM : 4-digit year; 1-digit month
  • YYMM : 2-digit year; 2-digit month
  • YYM : 2-digit year; 1-digit dekad
  • MM : 2-digit month [Averages]
  • M : 1-digit month [Averages]




Historical discussions below:

  • Error installing GeoCLIM

== Tamuka, I tried installing the program but it gave me an error, see below. I uninstalled, deleted all the files and tried it again but got the same error, Diego == Diego 08:45, 25 September 2013 (PDT)

  • Hi Diego, I have just re-uploaded a newer version with some bugs removed. That may solve your problem. However, I advise you to use GeoCLIM on a computer where you can log in with full admin settings. This will solve a large number of future problems... Tamuka100 10:19, 25 September 2013 (PDT)

I am moving this discussion to the comments section. Tamuka100 10:19, 25 September 2013 (PDT)

  • The following discussion was moved from the comments/questions section... Tamuka100 10:33, 25 September 2013 (PDT)
  1. Chris, what do you mean by "we should try to bullet proof GHA-monthly rainfall time series and analysis"
  • Chris explained bulletproofing as (a) making sure that all the functions that we really want to work work very well with no bugs and (b) removing any unnecessary functions which would distract from the teaching process.
  1. Now that I have had a look at the data in the ftp folder, I dont think there is a need to add a button to download CHIRPS archives, since all the data is already available for download one at a time, which is actually easier esp for users with slow data connections (incl some of those in Africa). Alternatively, we can have the "download archives" button download the archive to some folder specified by the user, then he/she can copy the arcgives to other computers, or ingest it to GeoCLIM at a later date? What do you think Chris?
  • Chris agreed with the idea of not adding a "download archive" button


  • Initial comments by Diego...

Tamuka Here are a few comments - I like the logo for the new GeoCLIM - In the file-new-dataset definition, does not go anywhere - Take out unbiased RFE when dowload - Please take the unbiasing rainfall estimates - Take out the install rainfall unbiasing files - Overall the program freezes

Thanks Diego. Noted. Please see a few more changes as already discussed by email:

- GeoCLIM can now take in datasets from several different file-naming conventions and any folder on the computer. This should mainly be done under the setup tool, but is also accessible from the File menu - GeoCLIM can now handle rainfall, temperature (avg, max and min) and PET analysis in daily, pentadal, dekadal and monthly formats. I have only tested rainfall dekadal and monthlies - I have taken out all references to unbiasing - I have made any RFE references more generic - to either refer explicitly to CHIRP or to climate surfaces in general. I was a bit stumped with the IRE though - should I rename IRE to ICE (improved CHIRPS interpolation)? I finally decided that since we would want to implement this procedure with any climate dataset, a complete name overhaul was in order. So I propose: BAIISICS (pronounced basics) - Background-Assisted IDW Interpolation of Stations for Improved Climate Surfaces. I think this new name also emphasizes the point that we are not just improving the rainfall grids, but we are also saying this is a better product than you would get by just doing a plain-old distance-based interpolation. What do you think, can we adopt this new name?


A couple of questions regarding the downloading: - please send me an operational ftp site and folder and naming convention where the CHIRPS is operationally updated to so that I can hardcode this. Once I have this, I will change the default download from RFE to CHIRPS. - The CHIRPS archives (which you want us to have a download utility for) will need to be in a specific format so that they can be ingested into the GeoCLIM (a zipped file with a subfolder where all the data are, and a text file called "_dataset" which contains information on the type of file in the archive. The zip file should be renamed to give it a .climdata extension. I would have to work with Pete or whoever is in charge to tell them the exact details of this file. In the meantime, I have put some dummy forms. When a user does a clean install (uninstall GeoCLIM, then delete the My Documents\GeoCLIM folder), the first time the user runs GeoCLIM, they will be asked if they want to (a) download CHIRPS, (b) import some existing data archive or (c) Cancel.

One comment regarding countries: - GeoCLIM is more likely to be implemented successfully in a country where the files for for that country are already available by default in the country. I say this from experience with GeoWRSI - making a region from scratch is a laborious process, and it's much better if the user can just click and choose. As such, please send me a list of countries (and if possible, public-domain province-and-boundary-level administrative units shapefiles for those countries) where you plan on providing training


One comment regarding the BAIISICS: I think it would be great if we could put a utility in the program which would allow users to create an archive of the BAIISICS outputs, which archive can be easily transferred to and imported by another computer. To do this however, the best would be to use a function in the .Net Framework 4.5 which would allow us to easily zip up files. Unfortunately, .Net Framework 4.5 is not supported by Windows XP, which means if we did that, GeoCLIM would not run on Windows XP. Unfortunately, I have recently come across many computers in some govt departments still running windows XP. So how should we proceed with this?

While you think about this, I leave you to enjoy the latest greatest GeoCLIM (it would be the greatest so far I guess, since, as Diego observed, the previous one was freezing a whole lot - let's hope for slightly sunnier weather for this version!)

Link to latest GeoCLIM:

https://docs.google.com/file/d/0B_H-a39WNYAUUjhwRXYwMkJIZ0U/edit?usp=sharing

Link to GeoCLIM-formatted CHIRPS data at 0.2 degrees: https://docs.google.com/file/d/0B_H-a39WNYAUaGFuSXp0QlNUUmM/edit?usp=sharing

Rename the file to .zip and unzip it if you want to see what the _dataset file looks like and use it to make your own GeoCLIM archives from there.


From Chris - 20-09-2013:

Hi Master T - this is awesome. I am also including Pete, since he'll be handling your data requests (he knows this is really important stuff!). Pete - can you read T's message and help us assemble dekadal and monthly archives of the CHIRPs for africa?

Here are some suggested thoughts/responses:

a) BASIICS - I love it!

b) CHIRPS ftp - ftp://chg-ftpout.geog.ucsb.edu/pub/org/chg/products/CHIRPS/bils/

In the monthly directory you'll find and africa sub-directory ...


c) Regions and countries ... I'd suggest we start with SADC, CILLS, GHA, and the five countries in the EAC: Kenya, Uganda, Burundi, Rwanda and Tanzania. Diego can help you find admin units as necessary.


d) XP ... I think for now we should support xp. The performance hit for switching to windows 7 is pretty major, so I think it'll stick around for a long time.


One final thought is that we should try to bullet proof GHA-monthly rainfall time series and analysis - since that will be the primary focus of our training in October. The 2nd objeective would be the same, but with temperature data - Diego, can you help T find our Tmax and Tmin climatologies?


From Diego: 20-09-2013

I am exited to use the GeoCLIM, it sounds very good. I think the IRE name is simpler than the BAIISICS. but not - CHIRPS is not operational yet, it is updated up to July 2013 and there is not a clear time step for update. here is the link where the new data will be. ftp://chg-dub.geog.ucsb.edu/pub/org/chg/products/CHIRPS/bils/ - in term of countries, if the program makes the data available by default, the list is going to be long but I think it would be great to provide the user with a region. - yes, window XP is used in many places - OK, we'll be waiting patiently. Diego

Retrieved from "https://wiki.chc.ucsb.edu/index.php?title=GeoCLIM/ToDo&oldid=1298"