.. _neknek: ------------------------- Overlapping Overset Grids ------------------------- This tutorial will describe how to run a case with two overlapping meshes (NekNek) from scratch. We illustrate this by using the :ref:`perhill` case. If you are not familiar with *Nek5000*, we strongly recommend you begin with the periodic hill tutorial first! The three key steps to running a case with NekNek are: 1. Setting up the mesh with appropriate boundary conditions for the overlapping interface. 2. Specifying parameters to control stability and accuracy for the overlapping-Schwarz iterations. 3. Modifying ``userbc`` to read Dirichlet boundary data for the overlapping interfaces. .......................... Pre-processing .......................... The pre-processing steps for a NekNek calculation differ from a mono-domain Nek5000 calculation only in terms of ensuring that the appropriate boundary condition ``int`` is specified at the overlapping interface. This boundary condition serves as the flag to let the solver know that Dirichlet data for these boundary conditions must come from interpolation of the data in the overlapping mesh. .......................... Mesh generation .......................... In this tutorial, we generate two meshes to span the entire domain, an upper and a lower mesh. The lower mesh is generated by ``genbox`` with the following input file: .. literalinclude:: neknek/lower.box and the upper mesh is generated by ``genbox`` with the following input file: .. literalinclude:: neknek/upper.box Note: the lower domain spans :math:`y \in [0,2.5]` and the upper domain spans :math:`y \in [2.25,3]`. We also use ``int`` for the boundary surfaces that overlap the other domain. Now the meshes can be generated by running ``genbox`` for each of these cases to obtain ``upper.re2`` and ``lower.re2``. .. code-block:: console $ genbox <<< upper.box $ mv box.re2 upper.re2 $ genbox <<< lower.box $ mv box.re2 upper.re2 :Reminder: ``genbox`` produces a file named ``box.re2``, so you will need to rename the files between generating the separate parts. Once we have both upper and lower mesh files, we need to run ``genmap`` for both meshes. .. code-block:: console $ genmap On input specify ``lower`` as your casename and press enter to use the default tolerance. This step will produce ``lower.ma2`` which needs to be generated only once. Next do the same for the upper mesh to generate ``upper.ma2``. .......................... usr file .......................... As for the mono-domain periodic hill case, we start by copying the template to our case directory: .. code-block:: console $ cp $HOME/Nek5000/core/zero.usr hillnn.usr _________________________________________ Modify mesh and add forcing to the flow _________________________________________ As for the mono-domain periodic hill case, we modify the mesh in ``usrdat2`` as: .. literalinclude:: neknek/hillnn.usr :language: fortran :lines: 143-166 .. _fig:hillnn_mesh: .. figure:: neknek/hillnn_mesh.png :align: center :figclass: align-center :alt: neknek_mesh Modified box mesh graded Currently, applying a constant mass flux with ``param(54)`` and ``param(55)`` is **not** supported with overlapping overset grids. For this case, we drive the flow using a constant acceleration term in ``userf`` as: .. literalinclude:: neknek/hillnn.usr :language: fortran :lines: 33-52 :emphasize-lines: 15 _________________________________________ Specify NekNek parameters _________________________________________ In ``usrdat``, we specify the number of Schwarz-like iterations and extrapolation order for the boundary conditions of the overlapping interface. These parameters impact the stability and accuracy of the calculation. This is done as: .. literalinclude:: neknek/hillnn.usr :language: fortran :lines: 132-141 :emphasize-lines: 6,7 Here we do 4 sub-iterations, ``ngeom=5``, at each time-step between the two meshes, and the temporal extrapolation order, ``ninter``, is 2. _______________________________ Boundary and initial conditions _______________________________ The next step is to specify the initial conditions. This can be done in the subroutine ``useric`` as follows: .. literalinclude:: neknek/hillnn.usr :language: fortran :lines: 100-114 Next we modify userbc to read the boundary conditions, interpolated from overlapping domains, as follows: .. literalinclude:: neknek/hillnn.usr :language: fortran :lines: 69-98 :emphasize-lines: 23-26 .......................... Control parameters .......................... The control parameters for this case are the same as that for the mono-domain periodic hill case. Create two files called ``lower.par`` and ``upper.par``, and type in each the following: .. literalinclude:: neknek/lower.par .. _neknek_size: .......................... SIZE file .......................... The static memory layout of Nek5000 requires the user to set some solver parameters through a so called ``SIZE`` file. Typically it's a good idea to start from our template. Copy the ``SIZE.template`` file from the core directory and rename it ``SIZE`` in the working directory: .. code-block:: console $ cp $HOME/Nek5000/core/SIZE.template SIZE For NekNek, the parameter choices are not as straightforward as for a regular case. Because the two cases are being run from a single executable, the ``SIZE`` file must be able to accommodate both cases. This can be accomplished by "brute force" by picking ``lelg`` for the larger of the two cases, ``lpmin`` for the smaller of the two and leaving ``lelt`` alone. However, for many practical cases, the two domains may not be of similar size and we may run out of memory, resulting in truncation errors at compile time (see :ref:`here`). This can be overcome by setting ``lelt`` manually and preserving the ratio of number of elements, :math:`E`, to the number of MPI-ranks, :math:`L`. If we have :math:`L_{tot}` MPI-ranks (CPU-cores) available, we might choose .. math:: L_{lower} & =\mathrm{floor}\left[\frac{E_{lower}}{E_{lower}+E_{upper}} L_{tot}\right]\\ L_{upper} & =L_{tot}-L_{lower}\\ \mathrm{lelt}&=\mathrm{max}\left[\frac{E_{upper}}{L_{upper}},\frac{E_{lower}}{L_{lower}}\right]+3 We then pick the largest mesh size for ``lelg``, and the smallest rank count for ``lpmin`` .. math:: \mathrm{lelg} &= \mathrm{max}\left[E_{lower},E_{upper}\right]\\ \mathrm{lpmin} &= \mathrm{min}\left[L_{lower},L_{upper}\right] Assuming we have 8 MPI-ranks available for this case, that gives ``lelg=110``, ``lpmin = 3``, and ``lelt=25``. Then, adjust the parameters in the BASIC section .. literalinclude:: neknek/SIZE :language: fortran :lines: 10 - 21 :emphasize-lines: 7-9 and in the OPTIONAL section, set the maximum number of sessions, ``nsessmax`` to 2 .. literalinclude:: neknek/SIZE :language: fortran :lines: 21 - 36 :emphasize-lines: 7 For this tutorial we have set our polynomial order to be :math:`N=7` - this is defined in the ``SIZE`` file above as ``lx1=8`` which indices that there are 8 points in each spatial dimension of every element. Additional details on the parameters in the ``SIZE`` file are given :ref:`here `. .......................... Compilation .......................... With the ``hillnn.usr``, and ``SIZE`` files created, you should now be all set to compile and run your case! As a final check, you should have the following files: * :download:`hillnn.usr ` * :download:`lower.par ` * :download:`lower.re2 ` * :download:`lower.ma2 ` * :download:`upper.par ` * :download:`upper.re2 ` * :download:`upper.ma2 ` * :download:`SIZE ` If for some reason you encountered an insurmountable error and were unable to generate any of the required files, you may use the provided links to download them. After confirming that you have all eight, you are now ready to compile .. code-block:: console $ makenek hillnn If all works properly, upon compilation the executable ``nek5000`` will be generated. ......................... Running the case ......................... Now you are all set, just run .. code-block:: console $ neknekb lower upper 5 3 to launch an MPI job on your local machine using 8 ranks. ........................... Post-processing the results ........................... As the case runs, it will produce restart files for both sessions, upper and lower, independently. You will need to use ``visnek`` for both sessions to generate two metadata files. They can be visualized separately or together by loading both sets into ParaView or VisIt to post-process the results .. _fig:hillnn_flow: .. figure:: neknek/hillnn_result.png :align: center :figclass: align-center :alt: hillnn_flow Steady-State flow field visualized in Visit/Paraview. Vectors represent velocity. Colors represent velocity magnitude. Note, velocity vectors are equal size and not scaled by magnitude.