(in Python)
2024-07-11
To access links or follow on your own device these slides can be found at:
jatkinson1000.github.io/rse-skills-workshop
All materials are available at:
Except where otherwise noted, these presentation materials are licensed under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) License.
Today:
Tomorrow:
Major Computational Programs
Data processing
Experiment support
Bathymetry by NOAA under public domain CTD Bottles by WHOI under public domain
Keeling Curve by Scripps under public domain
Dawn HPC by Joe Bishop with permission
Climate simulation by NSF under public domain
More widely than publishing papers, code is used in control and decision making:
Your code (or its derivatives) may well move from research to operational one day.
Margaret Hamilton and the Apollo XI by NASA under public domain
def calc_p(n,t):
return n*1.380649e-23*t
data = np.genfromtxt("mydata.csv")
p = calc_p(data[0,:],data[1,:]+273.15)
print(np.sum(p)/len(p))
What does this code do?
# Boltzmann Constant and 0 Kelvin
Kb = 1.380649e-23
T0 = 273.15
def calc_pres(n, t):
"""
Calculate pressure using ideal gas law p = nkT
Parameters:
n : array of number densities of molecules [N m-3]
t : array of temperatures in [K]
Returns:
array of pressures [Pa]
"""
return n * Kb * t
# Read in data from file and convert T from [oC] to [K]
data = np.genfromtxt("mydata.csv")
n = data[0, :]
temp = data[1, :] + T0
# Calculate pressure, average, and print
pres = calc_pres(n, temp)
pres_av = np.sum(pres) / len(pres)
print(pres_av)
venv
Python has inbuilt support for creating isolated virtual environments through venv
.
You will see that a directory rse-venv/
has been created.
Pip will install dependencies into this directory.
To remove the venv we delete this directory with rm -r rse-venv
on Unix, or rmdir /s rse-venv
on Windows.
There are various other tools available to manage dependencies:
Scenario: you have just finished some simulations with a climate model that should improve precipitation modelling and have the output data as a netCDF file.
You know that your colleague has produced relevant figures and analysis before, so ask them for a copy of their code (yay, reuse :+1:).
Go to exercise 1 and:
precipitation_climatology.py
Code/software will often have several dependencies required to run.
Provide a record of these to users to save time and errors when installing.
Recorded in a requirements.txt
file:
Scenario: you have just finished some simulations with a climate model that should improve precipitation modelling and have the output data as a netCDF file.
You know that your colleague has produced relevant figures and analysis before, so ask them for a copy of their code (yay, reuse :+1:).
Go to exercise 1 and:
precipitation_climatology.py
requirements.txt
Relevant to us today are:
“Readability counts”
- Tim Peters in the Zen of Python
By ensuring code aligns with PEP8 we:
“But I don’t have time to read and memorise all of this…”
Ruff (Astral 2025) - docs.astral.sh/ruff
def long_func(x, param_one, param_two=[], param_three=24, param_four=None,
param_five="Empty Report", param_six=123456):
val = 12*16 +(24) -10*param_one + param_six
if x > 5:
print("x is greater than 5")
else:
print("x is less than or equal to 5")
if param_four:
print(param_five)
print('You have called long_func.')
print("This function has several params.")
param_2.append(x*val)
return param_2
def long_func(
x,
param_one,
param_two=[],
param_three=24,
param_four=None,
param_five="Empty Report",
param_six=123456,
):
val = 12 * 16 + (24) - 10 * param_one + param_six
if x > 5:
print("x is greater than 5")
else:
print("x is less than or equal to 5")
if param_four:
print(param_five)
print("You have called long_func.")
print("This function has several params.")
param_2.append(x * val)
return param_2
Also runs on jupyter notebooks.
Highly configurable via configuration files.
I suggest incorporating into your projects now
Similar formatting tools exist for other languages:
Go to exercise 2 and:
ruff format
on precipitation_climatology.py
exercises/02_formatting/README.md
for more detailed instructions.It may seem inconsequential, but carefully naming variables and methods can improve the readability of code massively and can help to make code self-documenting.
A few naming tips and conventions:
ms --> mass
chclt --> chocolate
stm --> stem
put
, insert
, add
in the same code basedogs
, not dog_list
array --> dogs
age_int --> age
country_set --> countries
is
, has
or can
and avoid negations like not_green
e.g.purple --> is_purple
not_plant --> is_plant
sidekick --> has_sidekick
Without explaining variable:
With explaining variable:
Without an explaining variable, it is hard to see what this code is doing:
It is easier to see the intention. The code is more self-documenting.
Look through the code for any names of methods or variables that could be improved or clarified and update them. Note if you are using an IDE like Intellij or VSCode, you can use automatic renaming. Can you find an example from each of the suggestions listed below? Does this make the code easier to follow?
Consider the following:
mass
not ms
, stem
not stm
put
, insert
, add
in the same code baseis
, has
or can
and avoid negations like not_green
dogs
, not dog_list
Static Analysis
There are various tools available:
We will be using ruff check
for static analysis, which we already installed as part of ruff
in a previous exercise.
ruff check
def long_func(
x,
param_one,
param_two=[],
param_three=24,
param_four=None,
param_five="Empty Report",
param_six=123456,
):
val = 12 * 16 + (24) - 10 * param_one + param_six
if x > 5:
print("x is greater than 5")
else:
print("x is less than or equal to 5")
if param_four:
print(param_five)
print("You have called long_func.")
print("This function has several params.")
param_2.append(x * val)
return param_2
ruff check
def long_func(
x,
param_one,
param_two=[],
param_three=24,
param_four=None,
param_five="Empty Report",
param_six=123456,
):
val = 12 * 16 + (24) - 10 * param_one + param_six
if x > 5:
print("x is greater than 5")
else:
print("x is less than or equal to 5")
if param_four:
print(param_five)
print("You have called long_func.")
print("This function has several params.")
param_2.append(x * val)
return param_2
(myvenv) $ ruff check long_func.py
long_func.py:1:1: D100 Missing docstring in public module
long_func.py:1:5: PLR0913 Too many arguments in function definition (7 > 5)
long_func.py:1:5: D103 Missing docstring in public function
long_func.py:4:5: ARG001 Unused function argument: `param_two`
long_func.py:4:15: B006 Do not use mutable data structures for argument defaults
long_func.py:5:5: ARG001 Unused function argument: `param_three`
long_func.py:12:12: PLR2004 Magic value used in comparison, consider replacing `5` with a constant variable
long_func.py:24:5: F821 Undefined name `param_2`
long_func.py:25:12: F821 Undefined name `param_2`
Found 9 errors.
(myvenv) $
Note: use the
--output-format=concise
flag for this shortened output.
ruff check
def long_func(
x,
param_one,
param_two=[],
param_four=None,
param_five="Empty Report",
param_six=123456,
):
val = 12 * 16 + (24) - 10 * param_one + param_six
if x > 5:
print("x is greater than 5")
else:
print("x is less than or equal to 5")
if param_four:
print(param_five)
print("You have called long_func.")
print("This function has several params.")
param_two.append(x * val)
return param_two
(myvenv) $ ruff check long_func.py
long_func.py:1:1: D100 Missing docstring in public module
long_func.py:1:5: PLR0913 Too many arguments in function definition (6 > 5)
long_func.py:1:5: D103 Missing docstring in public function
long_func.py:4:15: B006 Do not use mutable data structures for argument defaults
long_func.py:11:12: PLR2004 Magic value used in comparison, consider replacing `5` with a constant variable
Found 5 errors.
(myvenv) $
Use ruff rule
to understand different rules:
ruff rule B006
will display the docs for B006
Similar tools for linting and static analysis exist for other languages:
More generally see this list of static analysis tools.
The default set of linting rules for ruff is quite simple.
The ruleset to be applied can be configured in a ruff.toml
file, as we do in this project, or pyproject.toml
for packaged code. For full details see the ruff configuration documentation.
Details of the different rules and rulesets that can be selected can be found in the ruff rules documentation.
Go to exercise 4 and:
ruff check
on precipitation_climatology.py
F401
unused imports, I001
unsorted imports, B006
dangerous default, and D202
Blank linesB904
try exceptionsD100
/D103
missing docstrings and PLR2004
magic values for now - we’ll come to them later.PLR0913
Too many argumentsExtensions:
--fix
flag--preview
flagSee you all tomorrow afternoon!
Let’s dive straight into comments and docstrings!
"""Module implementing pendulum equations."""
import numpy as np
def max_speed(l, theta):
"""..."""
return np.sqrt(2.0 * 9.81 * l * np.cos(theta))
def energy(m, l, theta):
"""..."""
return m * 9.81 * l * np.cos(theta)
def check_small_angle(theta):
"""..."""
if theta <= np.pi / 1800.0:
return True
return False
def bpm(l):
"""..."""
return 60.0 / 2.0 * np.pi * np.sqrt(l / 9.81)
"""Module implementing pendulum equations."""
import numpy as np
GRAV = 9.81
def get_period(l):
"""..."""
return 2.0 * np.pi * np.sqrt(l / GRAV)
def max_height(l, theta):
"""..."""
return l * np.cos(theta)
def max_speed(l, theta):
"""..."""
return np.sqrt(2.0 * GRAV * max_height(l, theta))
def energy(m, l, theta):
"""..."""
return m * GRAV * max_height(l, theta)
def check_small_angle(theta, small_ang=np.pi/1800.0):
"""..."""
if theta <= small_ang:
return True
return False
def bpm(l):
"""..."""
# Divide 60 seconds by period [s] for beats per minute
return 60.0 / get_period(l)
A better way to format strings since Python 3.6
Not catching on because of self-teaching from old code.
Strings are prepended with an f
allowing variables to be used in-place:
name = "electron"
mass = 9.1093837015E-31
# modulo
print("The mass of an %s is %.3e kg." % (name, mass))
# format
print("The mass of an {} is {:.3e} kg.".format(name, mass))
# f-string
print(f"The mass of an {name} is {mass:.3e} kg.")
f-strings can take expressions:
See Real Python for more information. Note: pylint W1203 recommends against using f-strings in logging calls.
Numbers in code that are not immediately obvious.
Instead:
numberwang by Mitchell and Webb under fair use
"""Module implementing pendulum equations."""
import numpy as np
def get_period(l):
"""..."""
return 2.0 * np.pi * np.sqrt(l / 9.81)
def max_height(l, theta):
"""..."""
return l * np.cos(theta)
def max_speed(l, theta):
"""..."""
return np.sqrt(2.0 * 9.81 * max_height(l, theta))
def energy(m, l, theta):
"""..."""
return m * 9.81 * max_height(l, theta)
def check_small_angle(theta):
"""..."""
if theta <= np.pi / 1800.0:
return True
return False
def bpm(l):
"""..."""
return 60.0 / get_period(l)
"""Module implementing pendulum equations."""
import numpy as np
GRAV = 9.81
def get_period(l):
"""..."""
return 2.0 * np.pi * np.sqrt(l / GRAV)
def max_height(l, theta):
"""..."""
return l * np.cos(theta)
def max_speed(l, theta):
"""..."""
return np.sqrt(2.0 * GRAV * max_height(l, theta))
def energy(m, l, theta):
"""..."""
return m * GRAV * max_height(l, theta)
def check_small_angle(theta, small_ang=np.pi/1800.0):
"""..."""
if theta <= small_ang:
return True
return False
def bpm(l):
"""..."""
# Divide 60 seconds by period [s] for beats per minute
return 60.0 / get_period(l)
Instead:
{
"config_name": "June 2022 m01 n19 run",
"start_date": "2022-05-28 00:00:00",
"end_date": "2022-06-12 23:59:59",
"satellites": ["m01", "n19"],
"noise_floor": [3.0, 3.0, 3.0],
"check_SNR": true,
"L_lim": [1.5, 8.0],
"telescopes": [90],
"n_bins": 27
}
{'config_name': 'June 2022 m01 n19 run', 'start_date': '2022-05-28 00:00:00', 'end_date': '2022-06-12 23:59:59', 'satellites': ['m01', 'n19'], 'noise_floor': [3.0, 3.0, 3.0], 'check_SNR': True, 'L_lim': [1.5, 8.0], 'telescopes': [90], 'n_bins': 27}
Magic Numbers
f-strings
Configuration settings
"__main__"
.README.md
All public codes should have a license attached!
LICENSE
file in the main directoryThe right selection may depend on your organisation and/or funder.
See choosealicense.com for more information.
GitHub and GitLab contain helpers to create popular licenses.
License guide by TechTarget under fair use
GPL3 image is in the Public Domain
MIT logo is in the Public Domain
Apache License image by Apache Software Foundation under Apache License 2.0
Copyright <YEAR> <COPYRIGHT HOLDER>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
A better way to format strings since Python 3.6
Not catching on because of self-teaching from old code.
Strings are prepended with an f
allowing variables to be used in-place:
name = "electron"
mass = 9.1093837015E-31
# modulo
print("The mass of an %s is %.3e kg." % (name, mass))
# format
print("The mass of an {} is {:.3e} kg.".format(name, mass))
# f-string
print(f"The mass of an {name} is {mass:.3e} kg.")
f-strings can take expressions:
See Real Python for more information. Note: pylint W1203 recommends against using f-strings in logging calls.
Numbers in code that are not immediately obvious.
Instead:
numberwang by Mitchell and Webb under fair use
"""Module implementing pendulum equations."""
import numpy as np
def get_period(l):
"""..."""
return 2.0 * np.pi * np.sqrt(l / 9.81)
def max_height(l, theta):
"""..."""
return l * np.cos(theta)
def max_speed(l, theta):
"""..."""
return np.sqrt(2.0 * 9.81 * max_height(l, theta))
def energy(m, l, theta):
"""..."""
return m * 9.81 * max_height(l, theta)
def check_small_angle(theta):
"""..."""
if theta <= np.pi / 1800.0:
return True
return False
def bpm(l):
"""..."""
return 60.0 / get_period(l)
"""Module implementing pendulum equations."""
import numpy as np
GRAV = 9.81
def get_period(l):
"""..."""
return 2.0 * np.pi * np.sqrt(l / GRAV)
def max_height(l, theta):
"""..."""
return l * np.cos(theta)
def max_speed(l, theta):
"""..."""
return np.sqrt(2.0 * GRAV * max_height(l, theta))
def energy(m, l, theta):
"""..."""
return m * GRAV * max_height(l, theta)
def check_small_angle(theta, small_ang=np.pi/1800.0):
"""..."""
if theta <= small_ang:
return True
return False
def bpm(l):
"""..."""
# Divide 60 seconds by period [s] for beats per minute
return 60.0 / get_period(l)
Instead:
{
"config_name": "June 2022 m01 n19 run",
"start_date": "2022-05-28 00:00:00",
"end_date": "2022-06-12 23:59:59",
"satellites": ["m01", "n19"],
"noise_floor": [3.0, 3.0, 3.0],
"check_SNR": true,
"L_lim": [1.5, 8.0],
"telescopes": [90],
"n_bins": 27
}
{'config_name': 'June 2022 m01 n19 run', 'start_date': '2022-05-28 00:00:00', 'end_date': '2022-06-12 23:59:59', 'satellites': ['m01', 'n19'], 'noise_floor': [3.0, 3.0, 3.0], 'check_SNR': True, 'L_lim': [1.5, 8.0], 'telescopes': [90], 'n_bins': 27}
Magic Numbers
f-strings
Configuration settings
"__main__"
.Beyond the scope of today are a few other honourable mentions:
__init__.py
pyproject.toml
These lessons are beyond the scope of today.
The ICCS RSE team are always keen to support researchers with developing and applying the principles discussed today.
If you would like to discuss applying this to your own codebase consider signing up for an ICCS Climate Code Clinic:
Get in touch:
The code in this workshop is based on a script from (Irving 2019).