tgrid-sequencing.rst - pism - [fork] customized build of PISM, the parallel ice sheet model (tillflux branch)
(HTM) git clone git://src.adamsgaard.dk/pism
(DIR) Log
(DIR) Files
(DIR) Refs
(DIR) LICENSE
---
tgrid-sequencing.rst (9134B)
---
1 .. include:: ../../global.txt
2
3 .. _sec-gridseq:
4
5 Grid sequencing
6 ---------------
7
8 The previous sections were not very ambitious. We were just getting started! Now we
9 demonstrate a serious PISM capability, the ability to change, specifically to *refine*,
10 the grid resolution at runtime.
11
12 One can of course do the longest model runs using a coarse grid, like the 20 km grid used
13 first. It is, however, only possible to pick up detail from high quality data, for
14 instance bed elevation and high-resolution climate data, using high grid resolution.
15
16 A 20 or 10 km grid is inadequate for resolving the flow of the ice sheet through the kind
17 of fjord-like, few-kilometer-wide topographical confinement which occurs, for example, at
18 Jakobshavn Isbrae in the west Greenland ice sheet :cite:`Joughinetal08`, an important
19 outlet glacier which both flows fast and drains a large fraction of the ice sheet. One
20 possibility is to set up an even higher-resolution PISM regional model covering only one
21 outlet glacier, but this requires decisions about coupling to the whole ice sheet flow.
22 (See section :ref:`sec-jako`.) But here we will work on high resolution for the whole ice
23 sheet, and thus all outlet glaciers.
24
25 Consider the following command and compare it to the :ref:`first one <firstcommand>`:
26
27 .. code-block:: none
28
29 mpiexec -n 4 pismr -i pism_Greenland_5km_v1.1.nc -bootstrap -Mx 301 -My 561 \
30 -Mz 201 -Mbz 21 -z_spacing equal -Lz 4000 -Lbz 2000 -ys -200 -ye 0 \
31 -regrid_file g20km_10ka_hy.nc -regrid_vars litho_temp,thk,enthalpy,tillwat,bmelt ...
32
33 Instead of a 20 km grid in the horizontal (``-Mx 76 -My 141``) we ask for a 5 km grid
34 (``-Mx 301 -My 561``). Instead of vertical grid resolution of 40 m (``-Mz 101 -z_spacing
35 equal -Lz 4000``) we ask for a vertical resolution of 20 m (``-Mz 201 -z_spacing equal -Lz
36 4000``).\ [#]_ Most significantly, however, we say ``-regrid_file g20km_10ka_hy.nc`` to
37 regrid --- specifically, to bilinearly-interpolate --- fields from a model result computed on
38 the coarser 20 km grid. The regridded fields (``-regrid_vars litho_temp,...``) are the
39 evolving mass and energy state variables which are already approximately at equilibrium on
40 the coarse grid. Because we are bootstrapping (i.e. using the ``-bootstrap`` option), the
41 other variables, especially the bedrock topography ``topg`` and the climate data, are
42 brought in to PISM at "full" resolution, that is, on the original 5 km grid in the data
43 file ``pism_Greenland_5km_v1.1.nc``.
44
45 This technique could be called "grid sequencing".\ [#]_ The result of the above command
46 will be to compute the near-equilibrium result on the fine 5 km grid, taking advantage of
47 the coarse-gridded computation of approximate equilibrium, and despite a run of only 200
48 model years (``-ys -200 -ye 0``). How close to equilibrium we get depends on both
49 durations, i.e. on both the coarse and fine grid run durations, but certainly the
50 computational effort is reduced by doing a short run on the fine grid. Note that in the
51 previous subsection we also used regridding. In that application, however,
52 ``-regrid_file`` only "brings in" fields from a run on the same resolution.
53
54 Generally the fine grid run duration in grid sequencing should be at least `t =
55 \Delta x / v_{\text{min}}` where `\Delta x` is the fine grid resolution and
56 `v_{\text{min}}` is the lowest ice flow speed that we expect to be relevant to our
57 modeling purposes. That is, the duration should be such that slow ice at least has a
58 chance to cross one grid cell. In this case, if `\Delta x = 5` km and
59 `v_{\text{min}} = 25` m/year then we get `t=200` a. Though we use this as the
60 duration, it is a bit short, and the reader might compare `t=500` results (i.e.
61 using `v_{\text{min}} = 10` m/year).
62
63 Actually we will demonstrate how to go from `20\,\text{km}` to `5\,\text{km}`
64 in two steps, `20\,\text{km}\,\to\,10\,\text{km}\,\to\,5\,\text{km}`, with durations
65 of 10 ka, 2 ka, and 200 a, respectively. The 20 km coarse grid run is already done; the
66 result is in ``g20km_10ka_hy.nc``. So we run the following script which is ``gridseq.sh``
67 in ``examples/std-greenland/``. It calls ``spinup.sh`` to collect all the right PISM
68 options:
69
70 .. code-block:: bash
71
72 #!/bin/bash
73 NN=4
74 export PARAM_PPQ=0.5
75 export REGRIDFILE=g20km_10ka_hy.nc
76 export EXSTEP=100
77 ./spinup.sh $NN const 2000 10 hybrid g10km_gridseq.nc
78 export REGRIDFILE=g10km_gridseq.nc
79 export EXSTEP=10
80 ./spinup.sh $NN const 200 5 hybrid g5km_gridseq.nc
81
82 Environment variable ``EXSTEP`` specifies the time in years between writing the
83 spatially-dependent, and large-file-size-generating, frames for the ``-extra_file ...``
84 diagnostic output.
85
86 .. warning::
87
88 The 5 km run requires 8 Gb of memory at minimum!
89
90 If you try it without at least 8 Gb of memory then your machine will "bog down" and start
91 using the hard disk for swap space! The run will not complete and your hard disk will get
92 a lot of wear! (If you have less than 8 Gb memory, comment out the last three lines of the
93 above script by putting the "``#``" character at the beginning of the line so that you
94 only do the 20 km `\to` 10 km refinement.)
95
96 Run the script like this:
97
98 .. code-block:: none
99
100 ./gridseq.sh &> out.gridseq &
101
102 The 10 km run takes under two wall-clock hours (8 processor-hours) and the 5 km run takes
103 about 6 wall-clock hours (24 processor-hours).
104
105 .. figure:: figures/g40-20-10-5km-detail.png
106 :name: fig-gridseqdetail
107
108 Detail of field ``velsurf_mag`` showing the central western coast of Greenland,
109 including Jakobshavn Isbrae (lowest major flow), from runs of resolution 40, 20, 10, 5
110 km (left-to-right). Color scheme and scale, including 100 m/year contour (solid black),
111 are all identical to ``velsurf_mag`` Figures :numref:`fig-secondoutputcoarse`,
112 :numref:`fig-csurfvsobserved`, and :numref:`fig-secondoutputfiner`.
113
114 :numref:`fig-gridseqdetail`, showing only a detail of the western coast of Greenland, with
115 several outlet glaciers visible, suggests what is accomplished: the high resolution runs
116 have separated outlet glacier flows, as they are in reality. Note that all of these results
117 were generated in a few wall clock hours on a laptop! The surface speed ``velsurf_mag``
118 from files ``g10km_gridseq.nc`` and ``g5km_gridseq.nc`` is shown (two right-most
119 subfigures). In the two left-hand subfigures we show the same field from NetCDF files
120 ``g40km_10ka_hy.nc`` and ``g20km_10ka_hy.nc``; the former is an added 40 km result using
121 an obvious modification of the run in section :ref:`sec-ssarun`.
122
123 .. figure:: figures/ivol-gridseq.png
124 :name: fig-ivolgridseq
125
126 Time series of ice volume ``ice_volume_glacierized`` from the three runs in our grid
127 sequencing example: 20 km for 10 ka = ``ts_g20km_10ka_hy.nc``, 10 km for 2 ka =
128 ``ts_g10km_gridseq.nc``, and 5 km for 200 a = ``ts_g5km_gridseq.nc``.
129
130 :numref:`fig-ivolgridseq`, which shows time series of ice volume, also shows the cost of
131 high resolution, however. The short 200 a run on the 5 km grid took about 3 wall-clock
132 hours compared to the 10 minutes taken by the 10 ka run on a 20 km grid. The fact that the
133 time series for ice volume on 10 km and 5 km grids are not very "steady" also suggests
134 that these runs should actually be longer.
135
136 In this vein, if you have an available supercomputer then a good exercise is to extend our
137 grid sequencing example to 3 km or 2 km resolutions
138 :cite:`AschwandenAdalgeirsdottirKhroulev`; these grids are already supported in the script
139 ``spinup.sh``. Note that the vertical grid also generally gets refined as the horizontal
140 grid is refined.
141
142 Going to a 1km grid is possible, but you will start to see the limitations of distributed
143 file systems in writing the enormous NetCDF files in question :cite:`DickensMorey2013`.
144 Notice that a factor-of-five refinement in all three dimensions, e.g. from 5 km to 1 km in
145 the horizontal, and from 20 m to 4 m in the vertical, generates an output NetCDF file
146 which is 125 times larger. Since the already-generated 5 km result ``g5km_gridseq.nc`` is
147 over 0.5 Gb, the result is a very large file at 1 km.
148
149 On the other hand, on fine grids we observe that *memory* parallelism, i.e. spreading the
150 stored model state over the separated memory of many nodes of supercomputers, is as
151 important as the usual *computation* (CPU) parallelism.
152
153 This subsection has emphasized the "P" in PISM, the nontrivial parallelism in which the
154 solution of the conservation equations, especially the stress balance equations, is
155 distributed across processors. An easier and more common mode of parallelism is to
156 distribute distinct model runs, each with different parameter values, among the
157 processors. For scientific purposes, such parameter studies, whether parallel or not, are
158 at least as valuable as individual high-resolution runs.
159
160 .. rubric:: Footnotes
161
162 .. [#] See subsections :ref:`sec-bootstrapping`, :ref:`sec-coords`, and :ref:`sec-grid`
163 for more about determining the computation domain and grid at bootstrapping.
164
165 .. [#] It is not quite "multigrid." That would both involve refinement and coarsening
166 stages in computing the fine grid solution.