Installing

As PyGNOME is under active development, it is usually best to install it from the source code. This also provides access to the code for examples, and to examine it in order to better understand it and potentially customize it.

The source can be found on gitHub here:

https://github.com/NOAA-ORR-ERD/PyGnome

You can either “clone” the repository with git (recommended), or download a zip file of the source code.

Installing from Source:

PyGNOME consists of compiled C++ code (libgnome), compiled Cython code (*.pyx files), and compiled Python extensions.

Currently, the only option is to install from the source code, which requires an appropriate compiler. At this time, NOAA is not maintaining current binaries.

PyGNOME depends on a number of third party packages – the complete list can be found in the various conda_requirements.txt files.

There are many dependencies that can be hard to build, so the easiest way is to use the conda package manager, but you can do it by hand as well – see below.

Building / Installing PyGNOME with the conda package manager

TL;DR

If you are already set up to use conda, then it’s as simple as:

Add the conda-forge channel:

> conda config --add channels conda-forge

Set the channel priority to “strict”:

> conda config --set channel_priority strict

Create an environment for PyGNOME with all requirements:

If you only need to run PyGNOME:

> conda create -n gnome --file python=3.10 conda_requirements.txt

If you need to build, etc PyGNOME:

> conda create -n gnome python=3.10 --file conda_requirements.txt --file conda_requirements_build.txt --file conda_requirements_test.txt

Activate the gnome environment:

> conda activate gnome

Build the gnome package:

> cd py_gnome
> python setup.py develop

Warning

when you run this command, you will get a warning from setuptools that running setup.py directly is deprecated. That is expected, and the build system will be updated before it stops working altogether.

You now should be good to go, but to make sure:

Run the tests:

> cd tests/unit_tests
> pytest --runslow

NOTE: the “runslow” tests requiring downloading data for the tests – you can eliminate that flag to get most of the tests to run without that delay.

All the details

Anaconda is a Python distribution that has most of the difficult-to-build packages that PyGNOME needs already built in. Thus it’s a nice target for running GNOME on your own system. “conda” is the packaging manager used to manage the system.

PyGNOME CAN be used with any Python distribution, but you will need to find or build a number of packages that are not easy to manage. If you are familiar with complex python packaging, then you can probably make it work. But conda makes it much easier, and that’s what we use ourselves, and support that use.

Anaconda vs miniconda vs miniforge:

Anaconda provides a fairly complete python system for computational programming – it is a large install, but comes with a lot of nice stuff pre-packaged that all works together.

miniconda is a much smaller install – it provides only Python and the conda package management system. You can install miniconda, and then install only the packages you need to run PyGNOME.

miniforge is similar to miniconda, but designed to work primarily with the conda-forge packages. As everything required for PyGNOME is available via conda-forge, and many are not available anywhere else, miniforge is the easiest way to get PyGNOME up and running. You can install miniforge, and then install only the packages you need to run PyGNOME.

Either of these should work fine with PyGNOME, as long as you create an independent environment for it.

NOTES:

PyGNOME requires Python version 3.9 or greater (currently 3.10 is currently used operationally – 3.11 should work, but it not well tested)

Anaconda (and miniconda) can be installed in either single-user or multi-user mode:

https://docs.continuum.io/anaconda/install

We (and Anaconda) recommend single-user mode (Select an install for “Just Me”) – that way, administrator privileges are not required for either initial installation or maintaining the system.

Windows:

You need the Windows 64 bit Python 3 version. Installing with the defaults works fine. You probably want to let it set the PATH for you – that’s a pain to do by hand.

OS-X:

Anaconda provides a 64 bit version – this should work well with PyGNOME. Either the graphical installer or the command line one is fine – use the graphical one if you are not all that comfortable with the *nix command line.

Linux:

The Linux 64bit-python3.10 is the one to use.

We do not support 32 bit on any platform.

conda

conda is the package manager that Anaconda is built on. So when working with Anaconda, you use the conda package manager for installing conda packages. pip can also be used within conda, but it’s best to use use conda for as much as you can.

As a rule, if you need a new package, you should try to conda install it, and then, if there is not conda package available, you can pip install it.

We have made sure that every package you need for PyGNOME is available for conda.

conda-forge

Conda-Forge (https://conda-forge.org/) is a community project that supplies a huge number of packages for the conda package manager. We have tried to assure that everything you need to run PyGNOME is available via the conda-forge channel.

Setting up

Install: Anaconda

or alternatively: Miniconda or Miniforge

Once you have Anaconda, Miniconda, or Miniforge installed, the rest of the instructions should be the same.

Update your (new) system

Once you have a conda system installed, you should start by getting everything up to date, as sometimes packages have been updated since the installer was built.

First, update the conda package manager itself:

Enter the following on the command-line:

> conda update conda
Setting up anaconda.org channels

anaconda.org is a web service for hosting conda packages for download. The way this is done is through anaconda “channels”, which can be thought of simply as places on anaconda.org where collections of packages are bundled together by the people hosting them.

Many of the dependencies that PyGNOME requires come out of the box with Anaconda (or the conda “defaults” channel), but a few important ones don’t.

The “conda-forge” project:

https://conda-forge.github.io/

Is a community project to build a wide variety of packages for conda – it should support everything that PyGNOME needs.

Adding another channel to conda:

To make it easy for your install to find conda-forge packages, it should be added to your conda configuration:

Add the conda-forge channel:

> conda config --add channels conda-forge

When you add a channel to conda, it puts it at the top of the list. So now when you install a package, conda will first look in conda-forge, and then in the default channel. This order should work well for PyGNOME.

You can see what channels you have with:

> conda config --get channels

It should return something like this:

--add channels 'defaults'   # lowest priority
--add channels 'conda-forge'   # highest priority

In that order – the order is important

You need to set the channel prioroty to “strict”:

> conda config --set channel_priority strict

This will assure that you will get packages from conda-forge, even if there are newer ones available in the defaults channel.

conda environments

The conda system supports isolated “environments” that can be used to maintain different versions of various packages for different projects. For more information see:

http://conda.pydata.org/docs/using/envs.html

NOTE: We highly recommend that you use a conda environment for GNOME.

If you are only going to use Python / conda for PyGNOME, then you could use the base environment. However, PyGNOME needs a number of specific package versions, so it is best to keep it separate from any other work you are doing.

(NOTE: you can do these steps with the Anaconda Navigator GUI if you have that installed)

Create an environment for PyGNOME:

> conda create -n gnome python=3.10 --file conda_requirements.txt --file conda_requirements_build.txt --file conda_requirements_test.txt

This will create an environment called “gnome” with Python itself and everything that it needs to be built, run, and tested – it will be quite a bit, so it may take a while.

To use that environment, you activate it with:

> conda activate gnome

and when you are done, you can deactivate it with:

> conda deactivate

After activating the environment, you can proceed with these instructions, and all the packages PyGNOME needs will be installed into that environment and kept separate from your main Anaconda install.

You will need to activate the environment any time you want to work with PyGNOME in the future

Download the PyGNOME Code

PyGNOME is not currently available as a conda package, as it is under active development, and many users will need access to the source code.

Once you have a conda environment set up, you can compile and install PyGNOME.

You will need the files from the PyGNOME sources. If you have not downloaded it yet, it is available here:

https://github.com/NOAA-ORR-ERD/PyGNOME

You can either download a zip file of all the sources and unpack it, or you can “clone” the git repository. If you clone the repository, you will be able to update the code with the latest version with a simple command, rather than having to re-download the whole package.

Downloading a single release

zip and tar archives of the PyGNOME source code can be found here:

https://github.com/NOAA-ORR-ERD/PyGNOME/releases

This will get you the entire source archive of a given release, which is a fine way to work with PyGNOME. However, if you want to be able to quickly include changes as we update the code, you may want to work with a git “clone” of the source code instead.

Cloning the PyGNOME git repository
git

You will need a git client:

Linux:

it should be available from your package manager:

> apt_get install git
or
> yum install git
OS-X:

git comes with the XCode command line tools:

http://osxdaily.com/2014/02/12/install-command-line-tools-mac-os-x/

Windows:

The “official” git for Windows installer is a good bet:

https://git-for-windows.github.io/

Once you have the client, it’s as easy as:

> git clone https://github.com/NOAA-ORR-ERD/PyGNOME.git

This will create a PyGNOME directory with all the code in it.

git branches:

git supports a number of different “branches” or versions of the code. You will most likley want to use the “main” branch (the default) unless you specifically want to experiment with a new feature.

Setting up conda

If you have not already created an environment in which to run PyGNOME, follow the instructions above.

To use the gnome environment you created, it needs to be activated with:

> conda activate gnome

and when you are done, you can deactivate it with:

> conda deactivate

If you don’t want to create an environment (or already have one), you can install what PyGNOME needs into an existing environment:

> cd PyGNOME  # or wherever you put the PyGNOME project
> conda install --file conda_requirements.txt --file conda_requirements_build.txt --file conda_requirements_test.txt

NOTE: PyGNOME has a lot of specific dependencies – it can be very hard for conda to resolve them with an large installed package base. If you have trouble, it’s easiest to make a new environment just for PyGNOME.

This should install all the packages required by PyGNOME.

(make sure you are in the correct conda environment, and you have the conda-forge channel enabled)

If installing the conda_requirements.txt fails:

If you get an error about a particular package not being able to be installed, then conda will not install ANY of the packages in the file. We try hard to make sure everything is available on conda-forge. If however, a package of that particular version is missing, you can try:

Edit the conda_requirements.txt file and comment out the offending package by putting a “#” at the start of the line:

...
scipy>=0.17
py_gd>=0.1.5
# libgd>=2.2.2
gsw>=3.0.3
...

That will disable that particular package, and hopefully everything else will install.

You can then try installing the offending package without a version specification:

> conda install libgd

And it may work for you.

The ADIOS Oil Database

If you want to use PyGNOME with “real oil”, rather than inert particles, you will need NOAA’s adios_db package from the ADIOS Oil Database Project:

https://github.com/NOAA-ORR-ERD/adios_oil_database

This will allow you to use the JSON oil data format downloadable from NOAA’s ADIOS Oil Database web app:

https://adios.orr.noaa.gov/

The adios_db package is available on conda-forge, and should have been installed by the process above. If not, it can be installed with

> conda install adios_db

However, the adios_db package is also under active development along with PyGNOME, so if you are working with the develop branch of PyGNOME, you may need the latest version of adios_db as well. In which case, you are best off downloading the sources from GitHub and installing it from source – similar to PyGNOME.

The latest releases (of the same branch) of each should be compatible.

cloning the repository

> git clone https://github.com/NOAA-ORR-ERD/adios_oil_database.git

Installing its dependencies:

> cd adios_db
> conda install --file conda_requirements.txt

Installing the package:

> pip install ./

(or pip install -e ./ to get an “editable” version)

Testing the adios_db install.

If you run the PyGNOME tests after having installed adios_db, it will run a few additional tests that require the adios_db. It should not need independent testing.

But if you want to test it directly, you will need additional requirements:

> conda install --file conda_requirements_test.txt

And then you can run the tests:

> pytest –pyargs adios_db

Compilers

To build PyGNOME, you will need a C/C++ compiler. The procedure for getting the compiler tools varies with the platform you are on.

OS-X

The system compiler for OS-X is XCode. It can be installed from the App Store.

Apple has changed the XCode install process a number of times over the years.

Rather than providing out of date information:

You need the “Xcode Command Line Tools” – look for Apple’s documentation for how to install those.

Once the command line tools are installed, you should be able to build PyGNOME as described below.

Windows

For compiling python extensions on Windows with python3 it is best to use the Microsoft the Visual Studio 2019 (or later) Build Tools. They should be available here:

https://visualstudio.microsoft.com/downloads/

The free “Community” version should be fine.

Once installed, you will want to use one of the “Visual Studio Developer Command Prompts” provided to actually build PyGNOME – it sets up the compiler for you.

Warning

On some locked down systems, such as those at NOAA, the standard way to use the MS compiler will not work for a user that does not have administration privileges. If you get errors about not being able to run the vcvarsall.bat script, then the compiler must be run as an administrator. If you have access to the NOAA/ORR GitLab server, a work around is supplied here: Building Python extensions on Windows. If you have this issue and are not from NOAA, ask for help on Python fora or as an issue in the PyGNOME gitHub project.

Linux

Linux uses the GNU gcc compiler. If it is not already installed on your system, use your system package manager to get it.

  • apt for Ubuntu and Linux Mint

  • rpm for Red Hat

  • dpkg for Debian

  • yum for CentOS

  • ??? for other distros

Building PyGNOME

At this point you should have all the necessary third-party tools in place.

And it is probably best to build the “develop” target for your PyGNOME package if you plan on developing or debugging the PyGNOME source code (or updating the source code from GitHub).

Building the “develop” target allows changes in the python code to be immediately available in your python environment without re-installing.

Of course if you plan on simply using the package, you may certainly build with the “install” target. Just keep in mind that any updates to the project will need to be rebuilt and re-installed in order for changes to take effect.

There are a number of options for building:

> python setup.py develop

builds and installs the gnome package in “development” (editable) mode.

> python setup.py install

builds and installs the gnome package into your Python install.

> python setup.py cleanall

cleans files generated by the build as well as files auto-generated by cython. It is a good idea to run cleanall after updating from the gitHub repo – particularly if strange errors are occurring.

You will need to re-run develop or install after running cleanall

NOTE: PyGNOME is not currently configured to be built with pip – you need to call setup.py directly.

Testing PyGNOME

We have an extensive set of unit and functional tests to make sure that PyGNOME is working properly.

To run the tests:

> cd py_gnome/tests/unit_tests
> pytest

and if those pass, you can run:

> pytest --runslow

which will run some more tests, some of which take a while to run.

Note that the tests will try to auto-download some data files. If you are not on the internet, this will fail. And of course if you have a slow connection, these files could take a while to download. Once the tests are run once, the downloaded files are cached for future test runs.

What if some tests fail?

We do our best to keep all tests passing on release versions of the package. But sometimes tests will fail due to the setup of the machine they are being run on – package versions, etc. So the first thing to do is to make sure you have installed the dependencies as specified.

But gnome is large package – hardly anyone is going to use all of it. So while we’d like all tests to pass, a given test failure may not be an issue for any given use case. It’s a bit hard to know whether a given test failure will affect your use case, but if you look at the name of the tests that fail, you might get a hint. For example, if any of the tests fail under test_weathering, and you are not doing any oil weathering modeling, you don’t need to worry about it.

In any case, you can try to run your use case, and see what happens.

Please report any unresolved test failures as an Issue on the gitHub project.

Running scripts

There are a number of scripts in the scripts directory.

In example_scripts you will find examples of using the gnome package for various tasks.

In testing_scripts you will find scripts that have been developed to test various features of the model. There are many more of these, so do look to see if they have what you need. But they are generally written in a less compact way as they are designed to exercise particular features.

You should be able to run these scripts in the same way as any Python script (with an IDE such as Spyder or PyCharm, or at the command line).

To run a script on the command line:

> cd py_gnome/scripts/example_scripts

If you are using a conda environment:

> conda activate gnome

Run the script:

> python example_script.py

Each of the scripts exercises different features of PyGNOME – they are hopefully well commented to see how they work.

In the testing_scripts dir, there is a run_all.py script that will run all the testing scripts – primarily to make sure they all can still run as we update the model.

For further documentation of PyGNOME, see:

https://gnome.orr.noaa.gov/doc/PyGNOME/index.html

Installation without conda / Anaconda

WARNING: These notes are out of date and not tested. All our development work is done using conda and conda-forge – there are a few too many dependencies not well supported by “pure” pip / PyPi packages at this point.

Overview:

PyGNOME has a lot of dependencies – you can find the full list by looking at the conda_requirements_*.txt file.

Most of them are well maintained 3rd party packages, that can be installed via pip, or by following the package author’s instructions.

But a few are maintained by the GNOME team at NOAA. These are all best installed by downloading the source code from the NOAA-ORR-ERD gitHub organization, and building and installing them from that source. These are:

pynucos: https://github.com/NOAA-ORR-ERD/PyNUCOS

gridded: https://github.com/NOAA-ORR-ERD/gridded

cell_tree2d (required by gridded): https://github.com/NOAA-ORR-ERD/cell_tree2d

py_gd: https://github.com/NOAA-ORR-ERD/py_gd

Building everything by hand / with pip

It is always good practice to create and run a complex system like this in a virtual environment of some sort: virtualenv, pipenv, etc.

A virtual environment is not required to run PyGNOME. But you may be working in an environment (on a corporate network, for example) that restricts your access to the system files on your computer. In such a case, you may require a virtualenv in order to freely install python packages in python’s site-packages dir. (site-packages is the standard place where python installers will put packages after building them)

You may also want to consider using conda environments – see above.

There is C++/Cython code that must be built - you will need the correct C/C++ compiler and recent setuptools, etc. See “Installing With Anaconda” for more detail (or online for docs on “building C extensions to Python”)

Python

Most people use Python itself from Python.org:

https://www.python.org/downloads/

Linux (Tested in 64-bit, CentOS)

For Linux use appropriate package manager (yum on CentOS, apt-get on Ubuntu) to download/install binary dependencies.

Binary Dependencies
  1. setuptools is required.

    > sudo apt-get install python-setuptools ` `

  2. To compile Python extensions, you need the development libs for Python:

    > sudo apt-get install python-dev

  3. netCDF4 python module requires NetCDF libraries:

    libhdf5-serial-dev

    libnetcdf-dev

  4. The following python packages, documented in PyGNOME’s conda_requirements.txt, may need to be manually installed.

Binaries for

Numpy and

Cython can be installed using apt-get.

Current binaries for these are sufficiently new: (Numpy >=1.20) and (Cython >= 3.0).

If you use virtualenv and apt-get to install these system site packages. Remember to use the --system-site-packages option when creating a new virtualenv so the system packages are available in the virtualenv.

Alternatively, pip install should also work. The following builds the latest packages in your virtualenv once the above dependencies are met:

> pip install numpy
> pip install cython
> pip install netCDF4

The remaining dependencies are python packages and can be installed using:

pip install -r requirements.txt

(NOTE: we do not regularly test the requirements.txt file – it may be incomplete – PR’s accepted)

See Build PyGNOME section below.

Windows

For compiling python extensions on Windows with python3 it is best to use the Microsoft the Visual Studio 2019 (or later) Build Tools. They should be available here:

https://visualstudio.microsoft.com/downloads/

The free “Community” version should be fine.

Once installed, you will want to use one of the “Visual Studio Developer Command Prompts” provided to actually build PyGNOME – it sets up the compiler for you.

Only 64 bit Windows is supported by PyGNOME

Binary Dependencies

Download and install the newest Windows executable distribution of

Python 3.10

(3.9 – 3.11 shoule work)

A number of the packages that GNOME depends on have very complex and brittle build processes, and depend on third-party libraries that can be a challenge to build.

Fortunately, many of them are available as binary wheels on PyPi

Another option is to use a Python scientific distribution, such as Anaconda – in which case, stop and read the Installing.rst file.

Dependencies can be installed using the command:

> pip install -r requirements.txt

See Build PyGNOME section below.

Build PyGNOME

  1. Clone the PyGNOME repository:

    > git clone https://github.com/NOAA-ORR-ERD/PyGNOME.git

  2. pip install all of GNOME’s python package dependencies:

    > cd PyGNOME/py_gnome
> pip install -r requirements.txt

  3. Install the adios_db pacakge – it is under active development along with py_gnome, so it’s best to install that from source as well:

    https://github.com/NOAA-ORR-ERD/adios_oil_database/tree/production/adios_db

  4. build the py_gnome module in develop or install mode:

    > python setup.py build

    or

    > python setup.py install

    or

    > python setup.py develop

The other option you may need is cleanall, which should clean the development environment – good to do after puling new code from git:

> python setup.py cleanall

  1. If this successfully completes, then run the unit tests:

    > pytest --runslow tests/unit_tests

Once all of the py_gnome unit tests pass, PyGNOME is now built and ready to be put to use. You can use the gnome module inside your python scripts to set up a variety of modeling scenarios.

There are example full scripts in the py_gnome/scripts directory.

Documentation of PyGNOME can be found at:

https://gnome.orr.noaa.gov/doc/PyGNOME/index.html