tdevelopment-workflow.rst - pism - [fork] customized build of PISM, the parallel ice sheet model (tillflux branch)
(HTM) git clone git://src.adamsgaard.dk/pism
(DIR) Log
(DIR) Files
(DIR) Refs
(DIR) LICENSE
---
tdevelopment-workflow.rst (10497B)
---
1 .. include:: ../global.txt
2
3 .. default-role:: literal
4
5 .. _sec-development-workflow:
6
7 Development workflow
8 ====================
9
10 The recommended development workflow is:
11
12 #. When starting a new project, create a topic branch starting from `dev`. If you are
13 fixing a bug in a released version of PISM, create a topic branch starting from
14 `master`.
15 #. Make changes to the code or documentation (or both).
16
17 a) Compile.
18 b) Fix any compilation errors and warnings. Repeat until your code compiles without
19 warnings.
20 c) Run `make test` and fix any test failures.
21 d) Push your code to GitHub for review or to get help with b) and c).
22
23 #. Add verification or regression tests.
24 #. Test your code and repeat 2a--2c until all tests pass.
25
26 #. Update documentation (if necessary).
27 #. Update the change log `CHANGES.rst`.\ [#]_
28 #. Merge new features into `dev` and fixes into `master` and `dev` (or submit a pull
29 request).
30
31 This document covers the tools and approaches we found useful for the steps listed above.
32
33 .. contents::
34
35 .. _sec-developmental-environment:
36
37 Setting up the environment
38 ^^^^^^^^^^^^^^^^^^^^^^^^^^
39
40 The majority of interesting PISM runs are performed on supercomputers, but we **do not**
41 recommend using supercomputers for development.
42
43 Use a desktop (or a laptop) computer running Linux or macOS.
44
45 While you can use `SSH` to connect to a remote system to write, compile, and test your
46 code, doing so will reduce your productivity when compared to using a computer you have
47 physical access to.
48
49 Any MPI implementation would work, but we prefer to use MPICH_ for PISM development. This
50 MPI implementation
51
52 - has headers that compile without warnings,
53 - provides type checking for pointer arguments in MPI calls, and
54 - does not produce "false positives" when debugging memory access with Valgrind_.
55
56 When working on a fix for a stubborn bug it *may* be helpful to use PETSc compiled with
57 debugging enabled (option `--with-debugging=1`), but in our experience this is rarely
58 needed. Optimized PETSc builds (using `--with-debugging=0`) are faster and this helps with
59 overall productivity.
60
61 Configure PISM with debugging symbols enabled.
62
63 .. code-block:: bash
64
65 export PETSC_DIR=~/local/petsc/petsc-3.11.3/
66 export PETSC_ARCH=opt
67
68 CC=mpicc CXX=mpicxx cmake \
69 -DCMAKE_BUILD_TYPE=Debug \
70 -DPism_BUILD_EXTRA_EXECS=YES \
71 -DPism_BUILD_PYTHON_BINDINGS=YES \
72 -DPism_DEBUG=YES \
73 ${pism_source_dir}
74
75 .. list-table:: PISM's configuration flags for development
76 :name: tab-cmake-development-flags
77 :header-rows: 1
78
79 * - Flag
80 - Meaning
81 * - `-DCMAKE_BUILD_TYPE=Debug`
82 - Enables pedantic compiler warnings
83 * - `-DPism_BUILD_EXTRA_EXECS=YES`
84 - Build extra testing executables (needed by some of regression test)
85 * - `-DPism_BUILD_PYTHON_BINDINGS=YES`
86 - Build PISM's Python bindings (used by many regression tests)
87 * - `-DPism_DEBUG=YES`
88 - Enables extra sanity checks in PISM
89
90 .. _sec-editor:
91
92 Editing source code
93 ^^^^^^^^^^^^^^^^^^^
94
95 Any text editor supporting C++, Python, and reStructuredText will work, but we recommend
96 Emacs_.
97
98 Your editor needs to provide the ability to jump from a compiler's error message to the
99 relevant part of the code. In Emacs, use `M-x compile` to start a compilation and `M-x
100 recompile` to re-run it.
101
102 An editor that can help you navigate the code and find function definitions, etc is also
103 helpful; try an IDE such as KDevelop_, for example.
104
105 .. _sec-compiling-pism:
106
107 Compiling PISM
108 ^^^^^^^^^^^^^^
109
110 If the computer you use for development has multiple CPU cores you should tell `make` to
111 use all of them. Run `make -j4` on a four-core laptop, for example; this will
112 significantly speed up compilation.
113
114 To further speed up re-compiling PISM, install ccache_ and configure PISM as follows:
115
116 .. code-block:: bash
117
118 CC="ccache mpicc" CXX="ccache mpicxx" cmake ...
119
120 .. _sec-debugging-pism:
121
122 Debugging
123 ^^^^^^^^^
124
125 The first step in debugging an issue is *always* this:
126
127 find the shortest simulation using the smallest possible grid that exhibits the
128 problematic behavior.
129
130 It does not have to be *the* shortest simulation, but it should complete (or stop because
131 of a failure) within seconds when running on the machine used for development.
132
133 A debugger such as GDB_ or LLDB_ can be very useful.\ [#]_ There are many online tutorials
134 for both.
135
136 You will need to know how to
137
138 - start a program,
139 - interrupt execution,
140 - set and remove a breakpoint,
141 - continue execution after stopping at a breakpoint,
142 - continue execution to the next line of the code,
143 - continue execution to the end of the current function call,
144 - step into a function call,
145 - print the value of a variable,
146 - print the stack trace.
147
148 This basic set of debugging skills is often sufficient.
149
150 Sometimes a failure happens in a loop that iterates over grid points and stepping through
151 the code in a debugger is impractical. A *conditional breakpoint* would help (i.e. stop
152 only if a condition is true), but this debugger feature is not always well supported and
153 often significantly slows down execution.
154
155 Here's a different way to stop the code when a condition is met: add `#include <cassert>`
156 to the top of the file (if it is not there), then add `assert(!condition);` to the place
157 in the code where you would like to stop if `condition` is met.
158
159 For example,
160
161 .. code-block:: c++
162
163 assert(!(i == 228 and j == 146));
164
165 will stop execution at the grid point where `i == 228` and `j == 146`.
166
167 Some of the more troublesome bugs involve memory access errors (*segmentation fault*
168 errors are often caused by these). Consider using Valgrind_ to detect them.
169
170 .. note::
171
172 Your code will run much, much slower when using Valgrind, so it is important to find
173 a small test case reproducing the error.
174
175 Issues visible in parallel runs only
176 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
177
178 Every once in a while a bug shows up in a parallel run but *not* in an equivalent serial
179 one. These bugs tend to be hard to fix and there is no definitive technique (or tool) that
180 helps with this. Here are some tips, though.
181
182 - Reduce the number of processes as much as possible. Most of the time the number of
183 processes can be reduced all the way down to 2 (the smallest truly parallel case).
184 - Run PISM with the option `-start_in_debugger`. This will produce a number of terminal
185 windows with GDB_. You will need to *continue execution* (GDB's command `c`) in all of
186 the windows. If PISM freezes, interrupting execution and printing the stack trace would
187 tell you where it got stuck.
188
189 Executing commands in all the windows with GDB is tedious and error-prone. To execute a
190 number of commands in all of them at the beginning of the run, create a file called
191 `.gdbinit` (in the current directory) and put GDB commands there (one per line).
192
193 For example,
194
195 .. code-block:: none
196
197 break pism::RuntimeError::RuntimeError()
198 continue
199
200 will set a breakpoint at `pism::RuntimeError::RuntimeError()` and continue execution.
201 - A parallel debugger such as TotalView may be helpful but requires a license. We don't
202 have experience with it and cannot give any advice.
203
204 .. _sec-continuous-integration:
205
206 Issues caught by automatic tests
207 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
208
209 Every time somebody pushes changes to PISM's repository on GitHub the continuous
210 integration system attempts to build PISM and (if it was built successfully) run a suite
211 of tests.
212
213 It is often helpful to be able to run the same tests locally. To do this, install Docker_
214 and `CircleCI CLI`_ (command-line interface), then run
215
216 .. code-block:: none
217
218 circleci local execute
219
220 in PISM's source code directory.
221
222 .. _sec-writing-tests:
223
224 Writing tests
225 ^^^^^^^^^^^^^
226
227 All contributions containing new features should contain tests for the new code.\ [#]_
228
229 A contribution fixing a bug should (ideally) contain a test that will ensure that it is
230 fixed.
231
232 Add verification tests (tests comparing results to an analytical solution) whenever
233 possible. If a verification test is not an option, consider adding a *regression* test
234 that compares computed results to a stored output from a trusted version of the code. This
235 will make it easier to detect a regression, i.e. an undesirable change in model results.
236
237 Here are some test writing tips:
238
239 - Make sure that a verification test uses a grid that is not square (different number of
240 grid points in `x` and `y` directions).
241 - If possible, write a test that performs a number of computations along a *refinement
242 path* and compare the computed convergence rate to the theoretical one.
243 - Try to include tests ensuring that `x` and `y` directions are interchangeable: in most
244 cases flow from left to right should behave the save as from bottom towards the top, etc.
245
246 Here are two ways to do this:
247
248 - Repeat the test twice, the second time using transposed inputs, then transpose results
249 and compare.
250 - Repeat the test twice, once using a refinement path along `x` and the second time
251 along `y`; make sure that you see the same convergence rate.
252 - It is good to check if the implementation preserves symmetries if the setup has any.
253
254 Python bindings make it possible to test many PISM's components in isolation from the rest
255 of the code. See tests in `test/regression` for some examples.
256
257 .. note::
258
259 This manual should cover PISM's Python bindings. If you see this, please e-mail
260 |pism-email| and remind us to document them.
261
262 .. _sec-editing-the-manual:
263
264 Editing PISM's manual
265 ^^^^^^^^^^^^^^^^^^^^^
266
267 PISM's manual is written using the reStructuredText_ markup and Sphinx_.
268
269 See :ref:`sec-install-documentation` for a list of tools needed to build PISM's
270 documentation.
271
272 When working on major edits, sphinx-autobuild_ can save you a lot of
273 time. Run
274
275 .. code-block:: bash
276
277 sphinx-autobuild -B /path/to/pism/doc/sphinx/ /tmp/pism-manual
278
279 To get a browser window containing PISM's manual that will stay up to date with your
280 edits.
281
282 Run `make manual_html` to re-build the manual and open `doc/sphinx/html/index.html`
283 inside your build directory.
284
285 Lists of configuration parameters and diagnostics reported by PISM are generated from the
286 |config-cdl| and the PISM code itself. To make sure that they are up to date, run `make`
287 in `doc/sphinx` and commit the changes.
288
289 .. rubric:: Footnotes
290
291 .. [#] See `Keep a change log <keep-a-change-log_>`_ for inspiration.
292 .. [#] In most cases a serial debugger is sufficient.
293 .. [#] Contributions of tests for existing code are also welcome.