EWX

Warning: This article is for the web application EWX (not the VM of the same name). EWX is currently hosted on chg-ewx, a CentOS VM. Configuration notes can be found here. Configuration notes for the old Windows installation of EWX can be found here.

Adding a data layer to the EWX GeoServer

Copy data files

Put GeoTiff files in /home/ftp_out/products/EWX/data/ on chg2.

Add dataset tag

Add, or copy and paste, a <dataset> tag in the C:\EWX\geoserver-2.1.1\data_dir\updater\datasets.xml file for the new dataset. Note: It is convenient to use 'Programmer's Notepad' with this file - control-click the first <datasets> tag closed and then click it open (without the control key) to open it back up with all other tags closed.

Set variables in dataset tag

Set the variables in the dataset tag:

• reloadAll: "true" or "false" depending if you want to reload the workspace and recalculate all the statistics.
• id: This is used as a referenced used with the window tag id attribute. See <window> tag below. This is also used as a prepend for the database spatial statistics table name.
• Important: This name can not contain the special characters "_" or "-".
• directoryPath: The location of the data files.
• name: The name used in the dataset listing in the left column of EWX.
• shortName: Not sure where this is used for.
• dataType: "RASTER" or "VECTOR"
• fileType: "GeoTIFF" or "Shapefile"
• workspace: Name of the workspace folder in C:\EWX\geoserver-2.1.1\data_dir\workspaces that will be created.
  • This name also defines the start of the name of the GeoTIFF file that is used to calculate geo-statistics.
  • Important This name can not contain the special characters "_" or "-". EWX will truncate the name at the first special character and use that to find the GeoTIFF file for spatial statistics.
  • This word also is used to define the start of the database table.
• fileNameSuffix: "tiff" or "tif"
• showInTree: "true" or "false" to show dataset in left column of EWX. Default is true.

Add a region tag

If this is a new region, add a <region> tag in datasets.xml Copy and past from existing region tag and change id, name and coordinates.

id: attribute to be referenced in window tag

name: label to use in EWX left hand pane

gridBounds: defines the min_lon, min_lat, max_lon, max_lat, respectively.

<region id="whem" name="Western Hemisphere">

<gridBounds>

<coords>

<double>-130.0</double>

<double>-50.0</double>

<double>-30.0</double>

<double>50.0</double>

</coords>

</gridBounds>

</region>

Add a window tag

Add a <window> tag for the new dataset

id: This attribute should be a combination of the region and dataset-ref attributes (see below) separated by a hyphen.

region: The region id tag to group this window under in EWX in the left hand pane.

dataset-ref: A reference to the dataset tag's id attribute to use.

label: The label to use in the EWX left hand pane.

related-layers: tag defining layers to display in and EWX data window "fly-out pane" activated by map's upper left corner "layers" icon.

<window id="global-chirpsclim" region-ref="global" dataset-ref="chirpsclim" label="CHIRPSclim">

<related-layers>

<related-layer dataset-ref="global_g2008" file-ref="global_g2008_0" name="Countries" initially-visible="true" />

Copy statistics TIFF files

Add a tif file and associated files to the GAUL data directory. /home/ftp_out/products/EWX/data/GAUL Associate files are: tfw, tif.aux.xml, tif.vat.dbf, xml files.

If this data layer has the same resolution, boundaries and target (cropzones, g2008_0, admin0) as another data layer, then the files can be copied and renamed for the new data layer. The new name is a combination of the dataset id, region and shapefile defining the sub-regions.

cp chirps_global_cropzones.tfw         chirpsclim_global_cropzones.tfw
cp chirps_global_cropzones.tif         chirpsclim_global_cropzones.tif
cp chirps_global_cropzones.tif.aux.xml chirpsclim_global_cropzones.tif.aux.xml
cp chirps_global_cropzones.tif.vat.dbf chirpsclim_global_cropzones.tif.vat.dbf
cp chirps_global_cropzones.tif.xml     chirpsclim_global_cropzones.tif.xml

Create TIFF for statistics in ArcMap

If the previous method can not be employed then this is the procedure to follow... good luck!

Copy the shape files and a raster file to your ArcMap workspace:

copy C:\EWX\data\GAUL\Africa\g_2008_1.*  C:\Users\marty\Documents\ArcGIS\CHIRPS_EAC
copy C:\EWX\data\CHIRP_EAC\monthly\data.1982.01.tiff

In ArcMap, add one of the dataset's raster files and the corresponding vector shape file:

File>Add Data, CHIRPS_EAC\g_2008_1.shp
File>Add Data, CHIRPS_EAC\data.1982.01.tiff

Open Catalog>Toolboxes>System toolboxes>Conversion Tools>ToRaster>Feature to Raster

Input Features
  g2008_0
Value field
  FEW_ID
Output Raster Dataset
  C:\EWX\data\GAUL\Global\chirpseac_g2008_1_int.tif

Note: Add _int to the name as this is the default data type. You can copy out as a float later.

Cell assignment type
  CELL_CENTER
Priority field
  NONE
Cellsize
  0.050000001, Note: this should be autofilled when you select the file in Environments/Processing Extent (see below) but it  isn't. 
  It is when you select Feature to Raster so a workaround is to open that and copy the numbers.

Open Environments.../

Output Coordinate System
  Same as layer "data.1982.01.tiff"

Processing Extent and chose raster file.
  Extent
   Same as layer data.1982.01.tiff
  Snap Raster
   data.1982.01.tiff

Raster Storage
  unselect Build pyramids
  unselect Calculate Statistics
Compression
  NONE
OK
OK

This should be enough but the tif is not parsable by EWX so it hast to be copied out as a 32 bit float image. Open ArcToolbox>Data Management Tools>Raster>Raster Dataset>Copy Raster.

Input Raster
  chirpseac_g2008_1_int
Output Raster Dataset
  C:\EWX\data\GAUL\Global\chirpseac_g2008_1.tif
Pixel Type
  32_BIT_FLOAT

Then remove the integer tif and place a copy of the tif files into the GAUL directory.

Validate datasets.xml

Run the datasets.xml file through an online xml validator (http://www.w3schools.com/xml/xml_validator.asp) to be sure you didn't mess something up.

Clear GeoWebCache

Make sure the GeoWebCache directory is cleared for the dataset before restarting. There may be some cruft in there from previous attempts.

C:\EWX\geoserver-2.1.1\data_dir\gwc

Starting and Stopping EWX

Under Windows:

cd /cygdrive/c/EWX/apache-tomcat-6.0.33/bin; ./catalina.bat stop
cd /cygdrive/c/EWX/apache-tomcat-6.0.33/bin; ./catalina.bat start

Under Linux:

sudo /opt/tomcat/bin/shutdown.sh
sudo /opt/tomcat/bin/startup.sh

Note: Do not start Tomcat as root, as this will cause known problems with X11. Instead, start Tomcat as a regular user using sudo as shown above.

Sometimes Tomcat takes an inordinately long time to shut down. You can see whether Tomcat is running by issuing a:

ps aux | grep java

If Tomcat does not shut down after a few minutes, you can kill it manually with:

sudo kill -9 <process_id>

Where <process_id> is the process id displayed by ps, above.

Adding a "style" to the EWX GeoServer

Styles modify the maps being displayed by the EWX server. The GeoServer v.2.1.1, which actually serves the maps, uses the Styled Layer Descriptor (SLD) markup language to do this. An intro to SLD can be found at:

http://docs.geoserver.org/2.1.1/user/styling/sld-introduction.html

The styles are locate is this directory:

C:\EWX\geoserver-2.1.1\data_dir\styles

They consist of a .sld and a .xml file. The xml file is generated by the GeoServer upon submission of the sld file. A new SLD style file has to be submitted through the GeoServer admin interface located at:

http://chg2.geog.ucsb.edu:8080/geoserver/web

Log in as admin after you have received the password from Marty. Then click the "styles" link in the left hand pane and click "Add a new style". You can then copy and paste your SLD content into the editing window and use the "Validate" button to validate the markup code. You have to click the "Submit" button for the GeoServer to create the xml and sld files in the styles directory. You can then use the name you entered in the "Name" box in the datasets.xml file to connect a dataset to the new style. The datasets.xml file is located at:

C:\EWX\geoserver-2.1.1\data_dir\updater

The name is entered as the style attribute to the statistic tag of the dataset tag.

If the style is being applied an existing dataset:

• set the dataset reloadAll tag to true
• remove the datasets workspace located at: C:\EWX\geoserver-2.1.1\data_dir\workspaces\…
• stop and restart EWX in a cygwin terminal window on CHG2 with the following commands:
  • cd /cygdrive/c/EWX/apache-tomcat-6.0.33/bin
  • ./catalina.bat stop
  • cd /cygdrive/c/EWX/apache-tomcat-6.0.33/bin
  • ./catalina.bat start

Tips

Use the GeoServer admin interface to edit and modify your SLD file as it will color code your markup and you can submit and apply the modified version in the interface. You can then examine the results by going to and EWX window (http://chg2.geog.ucsb.edu:8080/EWX/index.html) and opening the dataset. If you are iterating through multiple changes, you will need to either remove the GeoWebCache files at:

C:\EWX\geoserver-2.1.1\data_dir\gwc

or avoid viewing the same dataset and time period by progressing through the time series without selecting previously cached dates.

---

You can look for errors in the EWX log console (it appears when you start EWX) up or look in the log files at:

C:\EWX\apache-tomcat-6.0.33\logs\catalina.yyyy-mm-dd.log

Updating EWX with EROS Python Requirements

Per ederer's request, Libby performed the following updates on chg2 to bring the EWX installation in line with the EROS setup:

• Installed python27 (python27 was already installed as part of ArcGIS 10.1, but because that particular version may be needed by ArcGIS, python-2.7.8.amd64.msi was installed along side it).
• Added a PYTHONPATH environment variable and added %PYTHONPATH% to path environment variable.
• Tested python installation by
  • Opened DOS window
  • Entered 'python and enter'
    • Success!
• Installed numpy using the installation provided in the 'EWX_package_for_Greg' folder in chg2's ftproot (it is an exe).
• MS Visual C++ 2008 Redistributable 64x already installed (this is required by GDAL).
• Installed GDAL core (used gdal-111-1600-x64-core.msi, 'package for Greg' version is gdal-110-1500-x64-core.msi)
  • Added GDAL (C:\Program Files\GDAL) to 'path' in environment variable.
  • Installed GDAL python binding ('package for Greg' version is GDAL-1.10.0.win-amd64-py2.7, installed GDAL-1.11.0.win-amd64-py2.7.msi)
  • Created environment variables (are we sure these aren't reversed?)
    • GDAL_DATA - C:\Program Files\GDAL\gdalplugins
    • GDAL_DRIVER_PATH - C:\Program Files\GDAL\gdal-data
  • Tested GDAL installation by
    • Opened DOS window
    • Entered 'gdalinfo' --version
    • Success!
• Extracted geos-3.2.2-1.tar.bz2 (downloaded from www.filewatcher.com/m/geos-3.2.2-1.tar.bz2.427238-0.html before Libby was aware of the 'package for Greg' folder).
• Added geos_c.dll to C:\Python27\Lib\site-packages\shapely\DLLs
  • Added GEOS_LIBRARY_PATH - C:\Python27\Lib\site-packages\shapely\DLLsgeos_c.dll environment variable.
• Installed pip
  • Copied get-pip.py to python27 folder from EWX_package_for_Greg\PythonInstallation\pip.
  • In a command prompt, navigated to python27 folder and entered 'python get-pip.py'.
    • (to upgrade) python -m pip install -U pip
• Installed Shapely
  • Used executable found in 'EWX_package_for_Greg\PythonInstallation\Shapely' (Shapely-1.3.0.win-amd64-py2.7.exe).
    • (to upgrade) python -m pip install Shapely --upgrade
• Installed rasterstats module
  • Enter into command prompt:

python -m pip install https://pypi.python.org/packages/source/r/rasterstats/rasterstats-0.3.3.tar.gz 

(Downloads and install only when new version is needed.)

python -m pip install rasterstats 

(Downloads and installs.)

• Replaced the rasterStats module with EROS's updated source code (found in 'package for Greg' folder) by placing main.py in C:\Python27\Lib\site-packages\rasterstats (old main.py saved as main-org.py)
• Tested installation by running 'rasterstat_test.py' script to ensure it runs without errors ('rasterstat_test.py' ran with some errors, but they seem to be due to filepath issues originating from the fact that all EROS scripts are filepath specific for the EROS server and do not match chg2's.)
  • 'rasterstat_test.py is in 'EWX_package_for_Greg\PythonInstallation\scripts'.