Virtual Environments

Virtual Environments - venv

Python has inbuilt support for creating isolated virtual environments through venv.

$ python3 -m venv rse-venv
$ source rse-venv/bin/activate
(rse-venv) $ pip install <packagename>
(rse-venv) $ deactivate
$

PS> python -m venv rse-venv
PS> rse-venv\Scripts\Activate.ps1
(rse-venv) PS> pip install <packagename>
(rse-venv) PS> deactivate
PS>

C:\> python -m venv rse-venv
C:\> rse-venv\Scripts\activate.bat
(rse-venv) C:\> pip install <packagename>
(rse-venv) C:\> deactivate
C:\>

You will see that a directory rse-venv/ has been created.
Pip will install dependencies into this directory.
To remove the venv we delete this directory with rm -r rse-venv on Unix, or rmdir /s rse-venv on Windows.

Other Languages

There are various other tools available to manage dependencies:

• Python and more - conda
• C, C++, Fortran:
  • Module environments
  • Spack
• Rust - cargo
• Julia - Pkg environments
• R - renv

Exercise 1

Scenario: you have just finished some simulations with a climate model that should improve precipitation modelling and have the output data as a netCDF file.

You know that your colleague has produced relevant figures and analysis before, so ask them for a copy of their code (yay, reuse :+1:).

Go to exercise 1 and:

• Examine the code in precipitation_climatology.py
• Create and load a virtual environment
• Install the necessary dependencies
• Run the code - does it do what you thought?
• Deactivate the environment

Basic packaging concepts

Code/software will often have several dependencies required to run.

Provide a record of these to users to save time and errors when installing.

netcdf4
xarray
scipy==1.13.1
numpy<2.0
cartopy

(rse-venv) $ pip install -r requirements.txt

Exercise 1 revisited

Scenario: you have just finished some simulations with a climate model that should improve precipitation modelling and have the output data as a netCDF file.

You know that your colleague has produced relevant figures and analysis before, so ask them for a copy of their code (yay, reuse :+1:).

Go to exercise 1 and:

• Examine the code in precipitation_climatology.py
• Create and load a virtual environment
• Install the requirements from the supplied requirements.txt
• Run the code - does it do what you thought?
• Deactivate the environment

Python PEPs

Python Enhancement Proposals

• Technical documentation for the python community
• Guidelines, standards, and best-practice

Relevant to us today are:

• PEP8 - Python Style Guide (Rossum, Warsaw, and Coghlan 2001, 2013)
• PEP257 - Docstring Conventions (Goodger and Rossum 2001)
• PEP621 - Packaging (Cannon et al. 2020)

PEP8 & Formatting

“Readability counts”
    - Tim Peters in the Zen of Python

By ensuring code aligns with PEP8 we:

• standardise style,
• conform to best-practices, and
• improve code readability to
• make code easier to share, and
• reduce misinterpretation.

PEP8 & Formatting - Ruff

Ruff (Astral 2025) - docs.astral.sh/ruff

• a PEP 8 compliant formatter
  • Strict subset of PEP8
  • “Opinionated so you don’t have to be.”
• For full details see style guide
• Try online

(rse-venv) $ pip install ruff
(rse-venv) $ ruff format myfile.py
(rse-venv) $ ruff format mydirectory/

(rse-venv) PS> pip install ruff
(rse-venv) PS> ruff format myfile.py
(rse-venv) PS> ruff format mydirectory/

PEP8 & Formatting - Example

def long_func(x, param_one, param_two=[], param_three=24, param_four=None,
        param_five="Empty Report", param_six=123456):


    val = 12*16 +(24) -10*param_one +  param_six

    if x > 5:
        
        print("x is greater than 5")


    else:
        print("x is less than or equal to 5")


    if param_four:
        print(param_five)



    print('You have called long_func.')
    print("This function has several params.")

    param_2.append(x*val)
    return param_2

def long_func(
    x,
    param_one,
    param_two=[],
    param_three=24,
    param_four=None,
    param_five="Empty Report",
    param_six=123456,
):
    val = 12 * 16 + (24) - 10 * param_one + param_six

    if x > 5:
        print("x is greater than 5")

    else:
        print("x is less than or equal to 5")

    if param_four:
        print(param_five)

    print("You have called long_func.")
    print("This function has several params.")

    param_2.append(x * val)
    return param_2

PEP8 & Formatting - Ruff

• Also runs on jupyter notebooks.

• Highly configurable via configuration files.

• I suggest incorporating into your projects now

  • Widely-used standard¹
  • Plugins/lsp available for many editors.
  • Well suited to incorporation into continuous integration through workflows and git hooks.

Other languages

Similar formatting tools exist for other languages:

• Python - ruff or black
• C and C++ - clang-format
• Fortran - fortitude (in development)
• Rust - rustfmt
• Julia - JuliaFormatter.jl
• R - styler or Air

Exercise 2

Go to exercise 2 and:

• install ruff
• run ruff format on precipitation_climatology.py
• examine the output
  • Is it more readable?
  • Is there any aspect of the formatting style you find unintuitive?
• See exercises/02_formatting/README.md for more detailed instructions.

Static Analysis

We will be using ruff check for static analysis, which we already installed as part of ruff in a previous exercise.

Code Quality - ruff check

def long_func(
    x,
    param_one,
    param_two=[],
    param_three=24,
    param_four=None,
    param_five="Empty Report",
    param_six=123456,
):
    val = 12 * 16 + (24) - 10 * param_one + param_six

    if x > 5:
        print("x is greater than 5")

    else:
        print("x is less than or equal to 5")

    if param_four:
        print(param_five)

    print("You have called long_func.")
    print("This function has several params.")

    param_2.append(x * val)
    return param_2

Code Quality - ruff check

def long_func(
    x,
    param_one,
    param_two=[],
    param_three=24,
    param_four=None,
    param_five="Empty Report",
    param_six=123456,
):
    val = 12 * 16 + (24) - 10 * param_one + param_six

    if x > 5:
        print("x is greater than 5")

    else:
        print("x is less than or equal to 5")

    if param_four:
        print(param_five)

    print("You have called long_func.")
    print("This function has several params.")

    param_2.append(x * val)
    return param_2

(rse-venv) $ ruff check long_func.py
long_func.py:4:5: ARG001 Unused function argument: `param_two`
long_func.py:4:15: B006 Do not use mutable data structures for argument defaults
long_func.py:5:5: ARG001 Unused function argument: `param_three`
long_func.py:24:5: F821 Undefined name `param_2`
long_func.py:25:12: F821 Undefined name `param_2`
Found 5 errors.

(rse-venv) $

Code Quality - ruff check

def long_func(
    x,
    param_one,
    param_two=[],
    param_four=None,
    param_five="Empty Report",
    param_six=123456,
):
    val = 12 * 16 + (24) - 10 * param_one + param_six

    if x > 5:
        print("x is greater than 5")

    else:
        print("x is less than or equal to 5")

    if param_four:
        print(param_five)

    print("You have called long_func.")
    print("This function has several params.")

    param_two.append(x * val)
    return param_two

(rse-venv) $ ruff check long_func.py
long_func.py:4:15: B006 Do not use mutable data structures for argument defaults
Found 1 errors.

(rse-venv) $

ruff rule B006

IDE Integration

• Catch issues before running ruff
• Gradually coerces you to become a better programmer
• See ruff editor integration docs for instructions on setup for:
  • Vim
  • pycharm
  • Sublime
  • VS Code
  • Emacs

Other languages

Similar tools for linting and static analysis exist for other languages:

• C and C++ - clang-tidy and cppcheck
• Fortran - fortitude
• Rust - clippy
• R - lintr

More generally see this list of static analysis tools.

Exercise 3

Go to exercise 3 and:

• run ruff check on precipitation_climatology.py
• examine the report and try and address some of the issues.
  • Try and deal with: F401 unused imports, I001 unsorted imports, B006 dangerous default, and D202 Blank lines
  • For a challenge try and fix: B904 try exceptions

Extensions:

• explore autofixes using the --fix flag
• explore rules in development using the --preview flag
• try and add linting to your preferred text editor or IDE

Configuration

The default set of linting rules for ruff is quite simple.

The ruleset to be applied can be configured in a ruff.toml file, as we do in this project, or pyproject.toml for packaged code.
For full details see the ruff configuration documentation.

Details of the different rules and rulesets that can be selected can be found in the ruff rules documentation.

Let’s take a look and make some changes in preparation for the next sections:

• We will add the entire "D" (pydocstyle) ruleset.
• We will stop ignoring:
  • "PLR2004" (magic number comparisons) and
  • "E501" (line length).

Functions

• Avoid code duplication
  • Bad style
  • You will forget to update all the copies at one point when you make changes!
• Readability
  • A clearly named function replaces a chunk of code.
• Functions as building blocks
  • Create generic interface with several exchangeable options (e.g. “possion_solver”, “gauss_seidel_solver”, etc.)
  • Separation of concerns/single responsibility principles

Functions for readability and maintainability

"""Module implementing pendulum equations."""
import numpy as np

def max_speed(l, theta):
    """..."""
    return np.sqrt(2.0 * 9.81 *  l * np.cos(theta))

def energy(m, l, theta):
    """..."""
    return m * 9.81 * l * np.cos(theta)

def check_small_angle(theta):
    """..."""
    if theta <= np.pi / 1800.0:
        return True
    return False

def bpm(l):
    """..."""
    return 60.0 / 2.0 * np.pi * np.sqrt(l / 9.81)

"""Module implementing pendulum equations."""
import numpy as np

GRAV = 9.81

def get_period(l):
    """..."""
    return 2.0 * np.pi * np.sqrt(l / GRAV)

def max_height(l, theta):
    """..."""
    return l * np.cos(theta)

def max_speed(l, theta):
    """..."""
    return np.sqrt(2.0 * GRAV * max_height(l, theta))

def energy(m, l, theta):
    """..."""
    return m * GRAV * max_height(l, theta)

def check_small_angle(theta, small_ang=np.pi/1800.0):
    """..."""
    if theta <= small_ang:
        return True
    return False

def bpm(l):
    """..."""
    # Divide 60 seconds by period [s] for beats per minute
    return 60.0 / get_period(l)

Further structuring

• Breaking code into modules and files instead of having everything in one file
• Improves readability and manageability (-> scalability, extendability, coupling, testing)

Exercise 4

• Go to Exercise 4 and try to think of how you can structure the code in the plotting.py file to avoid code duplication, and improve readability and reusability.
• What enables the improved code you to do?
• Now look at plotting_solution.py . Can you further adapt the code to allow, for example, for customised axis labels?

That’s it for today!

See you all tomorrow afternoon!

Welcome back!

Let’s dive straight into naming, comments, and docstrings!

Naming for clarity

It may seem inconsequential, but carefully naming variables and methods can greatly improve the readability of code.

Since “code is read more than it is run” this is important for future you, but also for anyone you collaborate with or who might use your code in future.

It helps to make code self-documenting, reducing future bugs due to misunderstandings.

Here we cover some key considerations when writing code.

Naming for clarity

• Show the intention – how will someone else (future you) read it?

• Use readable, pronounceable, memorable, and searchable names:

  ms    --> mass
  chclt --> chocolate
  stm   --> stem

  avoid abbreviations and single letters unless commonly used

• Employ concept consistency
  e.g. only one of get_, retrive_, fetch_ in the code base

• Describe content rather than storage type
  Use plurals to indicate groups
  Name booleans using prefixes like is_, has_, can_ and avoid negations like not_:

  array       --> dogs                    float_or_int --> returns_int
  age_int     --> age                     not_plant    --> is_plant
  country_set --> countries               sidekick     --> has_sidekick

Explaining Variables

Without explaining variable:

def calculate_fare(age):
    if (age < 14):
        return 3
        ...

With explaining variable:

def calculate_fare(age):
    is_child = age < 14
    if (is_child):
        return 3
    ...

Explaining Variables

Without an explaining variable, it is hard to see what this code is doing:

import re

re.search("^\\+?[1-9][0-9]{7,14}$", "Sophie: CV56 9PQ, +12223334444")

import re

phone_number_regex = "^\\+?[1-9][0-9]{7,14}$"
re.search(phone_number_regex, "Sophie: CV56 9PQ, +12223334444")

Magic Numbers

Numbers in code that are not immediately obvious.

• Hard to read
• Hard to maintain
• Hard to adapt

Instead:

• Name a variable conveying meaning
• Set to a constant
• Use a comment to explain

Handling Magic Numbers

"""Module implementing pendulum equations."""
import numpy as np

def get_period(l):
    """..."""
    return 2.0 * np.pi * np.sqrt(l / 9.81)

def max_height(l, theta):
    """..."""
    return l * np.cos(theta)

def max_speed(l, theta):
    """..."""
    return np.sqrt(2.0 * 9.81 * max_height(l, theta))

def energy(m, l, theta):
    """..."""
    return m * 9.81 * max_height(l, theta)

def check_small_angle(theta):
    """..."""
    if theta <= np.pi / 1800.0:
        return True
    return False

def beats_per_minute(l):
    """..."""
    return 60.0 / get_period(l)

"""Module implementing pendulum equations."""
import numpy as np

GRAV = 9.81

def get_period(l):
    """..."""
    return 2.0 * np.pi * np.sqrt(l / GRAV)

def max_height(l, theta):
    """..."""
    return l * np.cos(theta)

def max_speed(l, theta):
    """..."""
    return np.sqrt(2.0 * GRAV * max_height(l, theta))

def energy(m, l, theta):
    """..."""
    return m * GRAV * max_height(l, theta)

def check_small_angle(theta, small_ang=np.pi/1800.0):
    """..."""
    if theta <= small_ang:
        return True
    return False

def beats_per_minute(l):
    """..."""
    # Divide 60 seconds by period [s] for beats per minute
    return 60.0 / get_period(l)

Exercise 5

Look through the code for method or variable names that could be improved or clarified and update them.¹

Look through the code and identify any magic numbers.
Implement what you feel to be the best approach in each case.

Does this make the code easier to follow?

Comments

Comments to avoid

• Dead code e.g.

  # plt.plot(time, velocity, "r0")
  plt.plot(time, velocity, "kx")
  # plt.plot(time, acceleration, "kx")
  # plt.ylabel("acceleration")
  plt.ylabel("velocity")

• Variable definitions e.g.

  # Set Force
  f = m * a

• Redundant comments e.g. i += 1 # Increment i

Comments - some thoughts¹

• Comments should not duplicate the code.
• Good comments do not excuse unclear code.
  • Comments should dispel confusion, not cause it.
  • If you can’t write a clear comment, there may be a problem with the code.
• Explain unidiomatic code in comments.
• Provide links to:
  • the original source of copied code.
  • external references where they will be most helpful.
• Use comments to mark incomplete implementations.
• Comments are not documentation.
  • Read by developers, documentation is for…

Docstrings

These are what make your code reusable (by you and others).

• In python docstrings are designated at the start of ‘things’ using triple quotes: """...""".
• PEP257 (Goodger and Rossum 2001) tells us what docstrings should say.
  Specific conventions tell us how they should say it.
• Where comments describe how it works, docstrings describe how to use it.
  Unlike comments, docstrings follow a set format.

Various formatting options exist: numpy, Google, reST, etc.
We will follow numpydoc as it is readable and widely used in scientific code.
Full guidance for numpydoc is available.

Docstrings

"""
Short one-line description.

Parameters
----------
name : type
    description of parameter

Returns
-------
name : type
    description of return value
"""

Docstrings

def calculate_gyroradius(mass, v_perp, charge, B, gamma=None):
    """
    Calculates the gyroradius of a charged particle in a magnetic field

    Parameters
    ----------
    mass : float
        The mass of the particle [kg]
    v_perp : float
        velocity perpendicular to magnetic field [m/s]
    charge : float
        particle charge [coulombs]
    B : float
        Magnetic field strength [teslas]
    gamma : float, optional
        Lorentz factor for relativistic case. default=None for non-relativistic case.

    Returns
    -------
    r_g : float
        Gyroradius of particle [m]

    Notes
    -----
    .. [1]  Walt, M, "Introduction to Geomagnetically Trapped Radiation,"
       Cambridge Atmospheric and Space Science Series, equation (2.4), 2005.
    """

    r_g = mass * v_perp / (abs(charge) * B)

    if gamma:
        r_g = r_g * gamma

    return r_g

Docstrings - pydocstyle

(rse-venv) $ ruff check gyroradius.py
gyroradius.py:3:5:
        D417 Missing argument description in the docstring for `calculate_gyroradius`: `B`
gyroradius.py:4:5 in public function `calculate_gyroradius`:
        D202: No blank lines allowed after function docstring
gyroradius.py:4:5 in public function `calculate_gyroradius`:
        D400: First line should end with a period
gyroradius.py:4:5 in public function `calculate_gyroradius`:
        D401: First line should be in imperative mood
(rse-venv) $

def calculate_gyroradius(mass, v_perp, charge, B, gamma=None):
    """
    Calculates the gyroradius of a charged particle in a magnetic field

    Parameters
    ----------
    mass : float
        The mass of the particle [kg]
    v_perp : float
        velocity perpendicular to magnetic field [m/s]
    charge : float
        particle charge [coulombs]
    gamma : float, optional
        Lorentz factor for relativistic case. default=None for non-relativistic case.

    Returns
    -------
    r_g : float
        Gyroradius of particle [m]

    Notes
    -----
    .. [1]  Walt, M, "Introduction to Geomagnetically Trapped Radiation,"
       Cambridge Atmospheric and Space Science Series, equation (2.4), 2005.
    """

    r_g = mass * v_perp / (abs(charge) * B)

    if gamma:
        r_g = r_g * gamma

    return r_g

Other languages

subroutine add(a, b, sum)
    !! Add two integers.

    integer, intent(in) :: a     !! First number, a
    integer, intent(in) :: b     !! Second number, b
    integer, intent(out) :: sum  !! Sum of a and b

    sum = a + b
end subroutine add

"""
    add(a,b)

Add two numbers `a` and `b`.

# Arguments
- `a::Number`: The first number.
- `b::Number`: The second number.

# Returns
- `Number`: The sum of `a` and `b`.
"""
function add(a::Number, b::Number)::Number
    return a + b
end

/**
 * Add two integers.
 * @param a, First integer.
 * @param b, Second integer.
 * @return Sum of a and b.
 */
int add(int a, int b) {
    return a + b;
}

#' Add two numbers.
#'
#' @param a First number.
#' @param b Second number.
#' @return Sum of `a` and `b`.
add <- function(a, b) {
    return(a + b)
}

Exercise 6

Go to exercise 6 and examine the comments:

• Is there any dead code?
  • How is it best to handle it?
• Are comments used sensibly?
  • Are any redundant and better off being removed?
  • Is there anywhere that would benefit from a comment?

Now turn your attention to the docstrings:

• Work through the file adding docstrings where they are missing.
• Can you resolve the issues raised by ruff with the pydocstuyle ruleset enabled?
• If you are unsure about variable types or meanings at any point you can sneak a look ahead to the code in exercise 7.

READMEs

• First point of contact for a new user/contributor with the code repository
• Should give essential information on
  • what this software is
  • what it’s for
  • how to get started
• In the best case it also tells you
  • who built/is building this
  • how to contribute
  • how to reuse
• Usually written in Markdown as README.md
• See makeareadme.com and readme.so for detailed information, examples, and tools

Recommended README Content

License

All public codes should have a license attached!

• LICENSE file in the main directory
• Protect ownership
• Limit liability
• Clarify what can be done with the code

The right selection may depend on your organisation and/or funder.

See choosealicense.com for more information.

GitHub and GitLab contain helpers to create popular licenses.

Types of Licenses

• Public Domain, Permissive, Copyleft

How to choose a license

Example: MIT License

Copyright <YEAR> <COPYRIGHT HOLDER>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Add a license in Github

Add a license in Gitlab

Other potential files in your repository

• CONTRIBUTING.md
• CITATION.cff
• CODE_OF_CONDUCT.md
• CHANGES.md

Other things

Beyond the scope of today are a few other honourable mentions:

• Functions and modules
• Packaging
  • Breaking projects into modules and __init__.py
  • Distributing projects with pyproject.toml
• Documentation
  • Auto-generation from docstrings with sphinx or mkdocs
• Type hinting
  • Adding type hinting to python code - how and why?
  • Type checking with mypy

These lessons are beyond the scope of today.

Where can I get help?

The ICCS RSE team are always keen to support researchers with developing and applying the principles discussed today.

If you would like to discuss applying this to your own codebase consider signing up for an ICCS Climate Code Clinic:

• 1hr slot
• RSEs will review code in advance and provide feedback and guidance.
• Online booking form

Where can I learn more?