tpetsc-options.rst - pism - [fork] customized build of PISM, the parallel ice sheet model (tillflux branch)
(HTM) git clone git://src.adamsgaard.dk/pism
(DIR) Log
(DIR) Files
(DIR) Refs
(DIR) LICENSE
---
tpetsc-options.rst (3603B)
---
1 .. include:: ../../global.txt
2
3 .. _sec-petscoptions:
4
5 PETSc options for PISM users
6 ----------------------------
7
8 All PETSc programs including PISM accept command line options which control how PETSc
9 distributes jobs among parallel processors, how it solves linear systems, what additional
10 information it provides, and so on. The PETSc manual :cite:`petsc-user-ref` is the complete
11 reference on these options. We list some here that are useful to PISM users. They can be
12 mixed in any order with PISM options.
13
14 Both for PISM and PETSc options, there are ways of avoiding the inconvenience of long
15 commands with many runtime options. Obviously, and as illustrated by examples in the
16 previous sections, shell scripts can be set up to run PISM. But PETSc also provides two
17 mechanisms to give runtime options without retyping at each run command.
18
19 First, the environment variable ``PETSC_OPTIONS`` can be set. For example, a sequence of
20 runs might need the same refined grid, and you might want to know if other options are
21 read, ignored, or misspelled. Set (in Bash):
22
23 .. code-block:: none
24
25 export PETSC_OPTIONS="-Mx 101 -My 101 -Mz 51 -options_left"
26
27 The runs
28
29 .. code-block:: none
30
31 pismv -test F -y 100
32 pismv -test G -y 100
33
34 then have the same refined grid in each run, and the runs report on which options were
35 read.
36
37 Alternatively, the file ``.petscrc`` is always read, if present, from the directory where
38 PISM (i.e. the PETSc program) is started. It can have a list of options, one per line. In
39 theory, these two PETSc mechanisms (``PETSC_OPTIONS`` and ``.petscrc``) can be used
40 together.
41
42 Now we address controls on how PETSc solves systems of linear equations, which uses the
43 PETSc "KSP" component (Krylov methods). Such linear solves are needed each time the
44 nonlinear SSA stress balance equations are used (e.g. with the option ``-stress_balance
45 ssa -ssa_method fd``).
46
47 Especially for solving the SSA equations with high resolution on multiple processors, it
48 is recommended that the option :opt:`-ssafd_ksp_rtol` be set lower than its default value
49 of :math:`10^{-5}`. For example,
50
51
52 .. code-block:: none
53
54 mpiexec -n 8 ssa_testi -Mx 3 -My 769 -ssa_method fd
55
56 may fail to converge on a certain machine, but adding "``-ssafd_ksp_rtol 1e-10``" works
57 fine.
58
59 There is also the question of solver *type*, using option :opt:`-ssafd_ksp_type`. Based on
60 one processor evidence from ``ssa_testi``, the following are possible choices in the sense
61 that they work and allow convergence at some reasonable rate: ``cg``, ``bicg``, ``gmres``,
62 ``bcgs``, ``cgs``, ``tfqmr``, ``tcqmr``, and ``cr``. It appears ``bicg``, ``gmres``,
63 ``bcgs``, and ``tfqmr``, at least, are all among the best. The default is ``gmres``.
64
65 Actually the KSP uses preconditioning. This aspect of the solve is critical for parallel
66 scalability, but it gives results which are dependent on the number of processors. The
67 preconditioner type can be chosen with :opt:`-ssafd_pc_type`. Several choices are
68 possible, but for solving the ice stream and shelf equations we recommend only
69 ``bjacobi``, ``ilu``, and ``asm``. Of these it is not currently clear which is fastest;
70 they are all about the same for ``ssa_testi`` with high tolerances (e.g. ``-ssafd_picard_rtol
71 1e-7`` ``-ssafd_ksp_rtol 1e-12``). The default (as set by PISM) is ``bjacobi``. To force
72 no preconditioning, which removes processor-number-dependence of results but may make the
73 solves fail, use ``-ssafd_pc_type none``.
74
75 For the full list of PETSc options controlling the SSAFD solver, run
76
77 .. code-block:: none
78
79 ssa_testi -ssa_method fd -help | grep ssafd_ | less