Chapter 3
User files

3.1 Case set-up .usr

Each simulation is defined by three files, the .rea file, the .usr file, and the SIZE file. In addition, there is a derived .map file that is generated from the .rea file by running genmap which will determine how the elements will be split across processors in the case of a parallel run. SIZE controls (at compile time) the polynomial degree used in the simulation, as well as the space dimension d = 2  or 3.

The SESSION.NAME file contains the current run, it must provide the name of the .rea file and the path to it. It does not however need to correspond to an .usr file of an identical name. This allows for different test cases (.usr files) that use the same geometry and boundary conditions (.rea files).

This chapter provides an introduction to the basic files required to set up a Nek5000 simulation.

3.1.1 Contents of .usr file

The most important interface to Nek5000 is the set of Fortran subroutines that are contained in the .usr file. This file allows direct access to all runtime variables. Here, the user may specify spatially varying properties (e.g., viscosity), volumetric heating sources, body forces, and so forth. One can also specify arbitrary initial and boundary conditions through the routines useric() and userbc(). The routine userchk() allows the user to interrogate the solution at the end of each timestep for diagnostic purposes. The .usr files provided in the /examples/... directories illustrate several of the more common analysis tools. For instance, there are utilities for computing the time average of u  , u2   , etc. so that one can analyze mean and rms distributions with the postprocessor. There are routines for computing the vorticity or the scalar λ2   for vortex identification, and so forth.

Routines in .usr file

The routine uservp specifies the variable properties of the governing equations. This routine is called once per processor, and once per discrete point therein.





Equation utrans udiff ifield








Momentum Eq.1.1 ρ  μ  1
Energy Eq.1.4 ρcp  k  2
Passive scalar Eq.1.6 (ρcp)i  ki  i-1




      subroutine uservp (ix,iy,iz,eg) 
      include SIZE 
      include TOTAL 
      include NEKUSE 
 
      integer iel 
      iel = gllel(eg) 
 
      udiff =0. 
      utrans=0. 
 
      return 
      end

The routine userdat is called right after the geometry is loaded into NEK5000 and prior to the distribution of the GLL points. This routine is called once per processor but for all the data on that processor. At this stage the elements can be modified as long as the topology is preserved. It is also possible to alter the type of boundary condition that is initially attributed in the .rea file, as illustrated below (the array cbc(face,iel,field) contains the boundary conditions per face and field of each element). Note the spacing allocated to each BC string is of three units.

      subroutine usrdat 
      include SIZE 
      include TOTAL 
      include NEKUSE 
      integer iel,f 
 
      do iel=1,nelt  !  Force flux BCs 
      do f=1,2*ndim 
         if (cbc(f,iel,1).eq.’W  ’) cbc(f,iel,2) = f   ! flux BC for temperature 
      enddo 
      enddo 
 
      return 
      end

The routine usrdat2 is called after the GLL points were distributed and allows at this point only for affine transformations of the geometry.

      subroutine usrdat2 
      include SIZE 
      include TOTAL 
 
      return 
      end

The routine userf is called once for each point and provides the force term in Eq.1.1. Not that according to the dimensionalization in Eq.1.1 the force term f  is in fact multiplied by the density ρ  .

      subroutine userf  (ix,iy,iz,eg) 
      include SIZE 
      include TOTAL 
      include NEKUSE 
 
      ffx = 0.0 
      ffy = 0.0 
      ffz = 0.0 
 
      return 
      end

Similarly to userf the routine userq provides the force term in Eq.1.4 and the subsequent passive scalar equations according to Eq.1.6.

      subroutine userq  (ix,iy,iz,eg) 
      include SIZE 
      include TOTAL 
      include NEKUSE 
 
      qvol   = 0. 
 
      return 
      end

The boundary conditions are assigned in userbc for both the fluid, temperature and all other scalars. An extensive list of such possible boundary conditions is available in Section. 4.4.1.

      subroutine userbc (ix,iy,iz,iside,ieg) 
      include SIZE 
      include TOTAL 
      include NEKUSE 
 
      ux=0.0 
      uy=0.0 
      uz=0.0 
      temp=0.0 
      flux = 1.0 
 
      return 
      end

Initial conditions are attributed in useric similarly to the boundary conditions

      subroutine useric (ix,iy,iz,ieg) 
      include SIZE 
      include TOTAL 
      include NEKUSE 
 
      uy=0.0 
      ux=0.0 
      uz=1.0 
 
      return 
      end

The routine userchk is called once per processor after each timestep (and once after the initialization is finished). This is the section where the solution can be interrogated and subsequent changes can be made.

      subroutine userchk 
      include SIZE 
      include TOTAL 
      include NEKUSE 
 
      call outpost(vx,vy,vz,pr,t,’ext’) 
 
      return 
      end

The routine usrdat3 is not widely used, however it shares the same properties with usrdat2.

      subroutine usrdat3 
      include SIZE 
      include TOTAL 
c 
      return 
      end

Nek5000 can solve the dimensional or non-dimensional equations by setting the following parameters




Dimensional parameters Non-dimensional parameters




p1=ρ  p1=1
p2=ν  p2=1/Re (-Re)
p7=ρCp  p7=1
p8=k  p8=1/Pe (-Pe)



alternatively the variable properties can be set in the USERVP routine.

What is a SESSION file?

To run NEK5000, each simulation must have a SESSION.NAME file. This file is read in by the code and gives the path to the relevant files describing the structure and parameters of the simulation. The SESSION.NAME file is a file that contains the name of the simulation and the full path to supporting files. For example, to run the eddy example from the repository, the SESSION.NAME file would look like:

eddy_uv\\  
/homes/user\_ name/nek5\_ svn/examples/eddy/

3.2 Problem size file SIZE

The SIZE file defines the problem size, i.e. spatial points at which the solution is to be evaluated within each element, number of elements per processor etc. The SIZE file governs the memory allocation for most of the arrays in Nek5000, with the exception of those required by the C utilities. The primary parameters of interest in SIZE are:

ldim = 2 or 3. This must be set to 2 for two-dimensional or axisymmetric simulations (the latter only partially supported) or to 3 for three-dimensional simulations.
lx1 controls the polynomial order of the approximation, N  =lx1-1.
lxd controls the polynomial order of the integration for convective terms. Generally, lxd=3 * lx1/2. On some platforms, however, it is important for memory access performance that lx1 and lxd be even.
lx2 = lx1 or lx1-2. This determines the formulation for the Navier-Stokes solver (i.e., the choice between the lP N  lP N  or lP N  lPN -2   methods) and the approximation order for the pressure, lx2-1.
lelt determines the maximum number of elements per processor.

The total size of the problem is lx1*ly1*lz1*lelt.

3.2.1 Memory Requirements

Per-processor memory requirements for Nek5000 scale roughly as 400 8-byte words per allocated gridpoint. The number of allocated gridpoints per processor is nmax   =lx1*ly1*lz1*lelt. (For 3D, lz1=ly1=lx1; for 2D, lz1=1, ly1=lx1.) If required for a particular simulation, more memory may be made available by using additional processors. For example, suppose one needed to run a simulation with 6000 elements of order N  = 9  . To leading order, the total memory requirements would be            3
≈  E(N  + 1)    points × 400 (wds/pt) × 8 bytes/wd = 6000     3
×10  × 400 × 8   = 19.2 GB. Assuming there is 400 MB of memory per core available to the user (after accounting for OS requirements), then one could run this simulation with P ≥  19,200   MB ∕(400   MB/proc) = 48   processors. To do so, it would be necessary to set lelt ≥ 6000/48 = 125.

We note two other parameters of interest in the parallel context:

lp, the maximum number of processors that can be used.
lelg, an upper bound on the number of elements in the simulation.

There is a slight memory penalty associated with these variables, so one generally does not want to have them excessively large. It is common, however, to have lp be as large as anticipated for a given case so that the executable can be run without recompiling on any admissible number of processors (Pmem  ≤  P ≤ E  , where Pmem  is the value computed above).

3.3 Geometry and Parameters file .rea

The .rea file consists of several sections. The mesh specifications with geometry, curvature and boundary conditions are in the second section.

Parameters and logical switches
parameters These control the runtime parameters such as viscosity, conductivity, number of steps, timestep size, order of the timestepping, frequency of output, iteration tolerances, flow rate, filter strength, etc. There are also a number of free parameters that the user can use as handles to be passed into the user defined routines in the .usr file.
passive scalar data This information can be specified also in the .uservp routine in the .usr file. If specified in the .rea file then the coefficients for the conductivity term are listed in ascending order for passive scalars ranging 1..9 followed by the values for the ρcp  coefficients.
4  Lines of passive scalar data follows 2 CONDUCT; 2 RHOCP  
   1.00000       1.00000       1.00000       1.00000       1.00000  
   1.00000       1.00000       1.00000       1.00000  
   1.00000       1.00000       1.00000       1.00000       1.00000  
   1.00000       1.00000       1.00000       1.00000

logicals These determine whether one is computing a steady or unsteady solution, whether advection is turned on, etc.

Next we have the logical switches as follow, a detailed explanation to be found in Sec:7.1.4

           13  LOGICAL SWITCHES FOLLOW  
  T     IFFLOW  
  T     IFHEAT  
  T     IFTRAN  
  T T F F F F F F F F F IFNAV & IFADVC (convection in P.S. fields)  
  F F T T T T T T T T T T IFTMSH (IF mesh for this field is T mesh)  
  F     IFAXIS  
  F     IFSTRS  
  F     IFSPLIT  
  F     IFMGRID  
  F     IFMODEL  
  F     IFKEPS  
  F     IFMVBD  
  F     IFCHAR

Mesh and boundary condition info
geometry The geometry is specified in an arcane format specifying the xyz  locations of each of the eight points for each element, or the xy  locations of each of the four points for each element in 2D. A line of the following type may be encountered at the beginning of the mesh section of the area file.
3.33333       3.33333     -0.833333      -1.16667     XFAC,YFAC,XZERO,YZERO

This part is to be read by Prenek and provides the origin of the system of coordinates XZERO;YZERO as well as the size of the cartesian units XFAC;YFAC. This one line has no impact on the mesh as being read in Nek5000.

The header of the mesh data may have the following representation

 **MESH DATA** 6 lines are X,Y,Z;X,Y,Z. Columns corners 1-4;5-8  
      226  3         192           NEL,NDIM,NELV  

The header states first how many elements are available in total (226  ), what dimension is the the problem (here three dimensional), and how many elements are in the fluid mesh (192  ). Following the header, all elements are listed. The fluid elements are listed first, followed by all solid elements if present. In this case there are (34  ) solid elements.

The data following the header is formatted as shown in Table. 3.1. This provides all the coordinates of an element for top and bottom faces. The numbering of the vertices is shown in Fig. 3.1. The header for each element as in Table. 3.1, i.e. [1A] GROUP is reminiscent of older Nek5000 format and does not impact the mesh generation at this stage.1


ELEMENT 1 [ 1A] GROUP 0

Face {1,2,3,4 }
x1,...,4 =  0.000000E+00 0.171820E+00 0.146403E+00 0.000000E+00
y1,...,4 =  0.190000E+00 0.168202E+00 0.343640E+00 0.380000E+00
z1,...,4 =  0.000000E+00 0.000000E+00 0.000000E+00 0.000000E+00
Face  {5,6,7,8}
x5,...,8 =  0.000000E+00 0.171820E+00 0.146403E+00 0.000000E+00
y     =
 5,...,8  0.190000E+00 0.168202E+00 0.343640E+00 0.380000E+00
z     =
 5,...,8  0.250000E+00 0.250000E+00 0.250000E+00 0.250000E+00

Table 3.1: Geometry description in .rea file


PIC

Figure 3.1: Geometry description in .rea file (sketch of one element ordering - Preprocessor corner notation)



PIC

Figure 3.2: Edge numbering in .rea file, the edge number is in between parenthesis. The other numbers represent vertices.


curvature This section describes the curvature of the elements. It is expressed as deformation of the linear elements. Therefore, if no elements are curved (if only linear elements are present) the section remains empty.

The section header may look like this:

640 Curved sides follow IEDGE,IEL,CURVE(I),I=1,5, CCURVE

Curvature information is provided by edge and element. Therefore up to 12 curvature entries can be present for each element. Only non-trivial curvature data needs to be provided, i.e., edges that correspond to linear elements, since they have no curvature, will have no entry. The formatting for the curvature data is provided in Table. 3.2.










IEDGE IEL CURVE(1) CURVE(2) CURVE(3) CURVE(4) CURVE(5) CCURVE
















9 2 0.125713 -0.992067 0.00000 0.00000 0.00000 m
10 38 0.125713 -0.992067 3.00000 0.00000 0.00000 m
1 40 1.00000 0.000000 0.00000 0.00000 0.00000 C









Table 3.2: Curvature information specification

There are several types of possible curvature information represented by CCURVE. This include:

Both ’C’ and ’s’ types allow for a surface of as high order as the polynomial used in the spectral method, since they have an underlying analytical description, any circle arc can be fully determined by the radius and end points. However for the ’m’ curved element descriptor the surface can be reconstructed only up to second order. This can be later updated to match the high-order polynomial after the GLL points have been distributed across the boundaries. This is the only general mean to describe curvature currrently in Nek5000 and corresponds to a HEX20 representation.

For better understanding let us focus on what the data in Table. 3.2 signifies. Edge 9 of element 2 has a edge midpoint at (0.125713,-0.992067,0.00000) and so on. For edge numbering the reader is advised to check Fig. 3.2, which illustrates the relationship between vertex numbering and edge numbering.

To maninpulate the geometry in Nek5000 at runtime, it is possible to use usrdat2. In this subroutine the user can deform the geometry to match the intended surface, followed by a call to the subroutine ’fixgeom’ which can realign the point distribution in the interior of the element.

We also note, that, unlike the geometry data, each curvature entry (as shown in Table. 3.2) is formatted and the format is dependent on the total number of elements. Three cases exist as shown in the code below:

                     if (nelgt.lt.1000) then  
                        write(10,’(i3,i3,5g14.6,1x,a1)’) i,eg,  
     $                  (vcurve(k,i,kb),k=1,5),cc  
                     elseif (nelgt.lt.1000000) then  
                        write(10,’(i2,i6,5g14.6,1x,a1)’) i,eg,  
     $                  (vcurve(k,i,kb),k=1,5),cc  
                     else  
                        write(10,’(i2,i12,5g14.6,1x,a1)’) i,eg,  
     $                  (vcurve(k,i,kb),k=1,5),cc

The fortran format is as follows:


PIC

Figure 3.3: Example mesh - without curvature. Square dots represent example vertices.



PIC

Figure 3.4: Example mesh - with curvature. Circular dots represent example midsize points.


To further illustrate the usage of curvature data, let us examine an example of .rea file with and wiuthout curvature information and the corresponding mesh representation. Figure 3.3 represents a 12 element box mesh (2x2x3, with periodic conditions in z) without curvature, while Fig. 3.4 presents the same mesh with a sinusoidal deformation in direction y. Only two edges per element are curved. The input for the mesh without curvature is:

          12  3          12 NEL,NDIM,NELV  
      ELEMENT             1 [    1a]    GROUP     0  
   0.000000       1.000000       1.000000       0.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
   0.000000       0.000000       0.000000       0.000000  
   0.000000       1.000000       1.000000       0.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.5000000E-01  0.5000000E-01  0.5000000E-01  0.5000000E-01  
      ELEMENT             2 [    1a]    GROUP     0  
   1.000000       2.000000       2.000000       1.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
   0.000000       0.000000       0.000000       0.000000  
   1.000000       2.000000       2.000000       1.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.5000000E-01  0.5000000E-01  0.5000000E-01  0.5000000E-01  
      ELEMENT             3 [    1a]    GROUP     0  
   0.000000       1.000000       1.000000       0.000000  
   0.000000       0.000000      0.5000000      0.5000000  
   0.000000       0.000000       0.000000       0.000000  
   0.000000       1.000000       1.000000       0.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.5000000E-01  0.5000000E-01  0.5000000E-01  0.5000000E-01  
      ELEMENT             4 [    1a]    GROUP     0  
   1.000000       2.000000       2.000000       1.000000  
   0.000000       0.000000      0.5000000      0.5000000  
   0.000000       0.000000       0.000000       0.000000  
   1.000000       2.000000       2.000000       1.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.5000000E-01  0.5000000E-01  0.5000000E-01  0.5000000E-01  
      ELEMENT             5 [    1a]    GROUP     0  
   0.000000       1.000000       1.000000       0.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.5000000E-01  0.5000000E-01  0.5000000E-01  0.5000000E-01  
   0.000000       1.000000       1.000000       0.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.1000000      0.1000000      0.1000000      0.1000000  
      ELEMENT             6 [    1a]    GROUP     0  
   1.000000       2.000000       2.000000       1.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.5000000E-01  0.5000000E-01  0.5000000E-01  0.5000000E-01  
   1.000000       2.000000       2.000000       1.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.1000000      0.1000000      0.1000000      0.1000000  
      ELEMENT             7 [    1a]    GROUP     0  
   0.000000       1.000000       1.000000       0.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.5000000E-01  0.5000000E-01  0.5000000E-01  0.5000000E-01  
   0.000000       1.000000       1.000000       0.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.1000000      0.1000000      0.1000000      0.1000000  
      ELEMENT             8 [    1a]    GROUP     0  
   1.000000       2.000000       2.000000       1.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.5000000E-01  0.5000000E-01  0.5000000E-01  0.5000000E-01  
   1.000000       2.000000       2.000000       1.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.1000000      0.1000000      0.1000000      0.1000000  
      ELEMENT             9 [    1a]    GROUP     0  
   0.000000       1.000000       1.000000       0.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.1000000      0.1000000      0.1000000      0.1000000  
   0.000000       1.000000       1.000000       0.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.1500000      0.1500000      0.1500000      0.1500000  
      ELEMENT            10 [    1a]    GROUP     0  
   1.000000       2.000000       2.000000       1.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.1000000      0.1000000      0.1000000      0.1000000  
   1.000000       2.000000       2.000000       1.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.1500000      0.1500000      0.1500000      0.1500000  
      ELEMENT            11 [    1a]    GROUP     0  
   0.000000       1.000000       1.000000       0.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.1000000      0.1000000      0.1000000      0.1000000  
   0.000000       1.000000       1.000000       0.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.1500000      0.1500000      0.1500000      0.1500000  
      ELEMENT            12 [    1a]    GROUP     0  
   1.000000       2.000000       2.000000       1.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.1000000      0.1000000      0.1000000      0.1000000  
   1.000000       2.000000       2.000000       1.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.1500000      0.1500000      0.1500000      0.1500000  
  ***** CURVED SIDE DATA *****  
         0 Curved sides follow IEDGE,IEL,CURVE(I),I=1,5, CCURVE  
  ***** BOUNDARY CONDITIONS *****  
  *****    FLUID   BOUNDARY CONDITIONS *****  
 W    1  1   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    1  2   2.00000       4.00000       0.00000       0.00000       0.00000  
 E    1  3   3.00000       1.00000       0.00000       0.00000       0.00000  
 v    1  4   0.00000       0.00000       0.00000       0.00000       0.00000  
 P    1  5   9.00000       6.00000       0.00000       0.00000       0.00000  
 E    1  6   5.00000       5.00000       0.00000       0.00000       0.00000  
 W    2  1   0.00000       0.00000       0.00000       0.00000       0.00000  
 O    2  2   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    2  3   4.00000       1.00000       0.00000       0.00000       0.00000  
 E    2  4   1.00000       2.00000       0.00000       0.00000       0.00000  
 P    2  5   10.0000       6.00000       0.00000       0.00000       0.00000  
 E    2  6   6.00000       5.00000       0.00000       0.00000       0.00000  
 E    3  1   1.00000       3.00000       0.00000       0.00000       0.00000  
 E    3  2   4.00000       4.00000       0.00000       0.00000       0.00000  
 W    3  3   0.00000       0.00000       0.00000       0.00000       0.00000  
 v    3  4   0.00000       0.00000       0.00000       0.00000       0.00000  
 P    3  5   11.0000       6.00000       0.00000       0.00000       0.00000  
 E    3  6   7.00000       5.00000       0.00000       0.00000       0.00000  
 E    4  1   2.00000       3.00000       0.00000       0.00000       0.00000  
 O    4  2   0.00000       0.00000       0.00000       0.00000       0.00000  
 W    4  3   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    4  4   3.00000       2.00000       0.00000       0.00000       0.00000  
 P    4  5   12.0000       6.00000       0.00000       0.00000       0.00000  
 E    4  6   8.00000       5.00000       0.00000       0.00000       0.00000  
 W    5  1   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    5  2   6.00000       4.00000       0.00000       0.00000       0.00000  
 E    5  3   7.00000       1.00000       0.00000       0.00000       0.00000  
 v    5  4   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    5  5   1.00000       6.00000       0.00000       0.00000       0.00000  
 E    5  6   9.00000       5.00000       0.00000       0.00000       0.00000  
 W    6  1   0.00000       0.00000       0.00000       0.00000       0.00000  
 O    6  2   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    6  3   8.00000       1.00000       0.00000       0.00000       0.00000  
 E    6  4   5.00000       2.00000       0.00000       0.00000       0.00000  
 E    6  5   2.00000       6.00000       0.00000       0.00000       0.00000  
 E    6  6   10.0000       5.00000       0.00000       0.00000       0.00000  
 E    7  1   5.00000       3.00000       0.00000       0.00000       0.00000  
 E    7  2   8.00000       4.00000       0.00000       0.00000       0.00000  
 W    7  3   0.00000       0.00000       0.00000       0.00000       0.00000  
 v    7  4   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    7  5   3.00000       6.00000       0.00000       0.00000       0.00000  
 E    7  6   11.0000       5.00000       0.00000       0.00000       0.00000  
 E    8  1   6.00000       3.00000       0.00000       0.00000       0.00000  
 O    8  2   0.00000       0.00000       0.00000       0.00000       0.00000  
 W    8  3   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    8  4   7.00000       2.00000       0.00000       0.00000       0.00000  
 E    8  5   4.00000       6.00000       0.00000       0.00000       0.00000  
 E    8  6   12.0000       5.00000       0.00000       0.00000       0.00000  
 W    9  1   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    9  2   10.0000       4.00000       0.00000       0.00000       0.00000  
 E    9  3   11.0000       1.00000       0.00000       0.00000       0.00000  
 v    9  4   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    9  5   5.00000       6.00000       0.00000       0.00000       0.00000  
 P    9  6   1.00000       5.00000       0.00000       0.00000       0.00000  
 W   10  1   0.00000       0.00000       0.00000       0.00000       0.00000  
 O   10  2   0.00000       0.00000       0.00000       0.00000       0.00000  
 E   10  3   12.0000       1.00000       0.00000       0.00000       0.00000  
 E   10  4   9.00000       2.00000       0.00000       0.00000       0.00000  
 E   10  5   6.00000       6.00000       0.00000       0.00000       0.00000  
 P   10  6   2.00000       5.00000       0.00000       0.00000       0.00000  
 E   11  1   9.00000       3.00000       0.00000       0.00000       0.00000  
 E   11  2   12.0000       4.00000       0.00000       0.00000       0.00000  
 W   11  3   0.00000       0.00000       0.00000       0.00000       0.00000  
 v   11  4   0.00000       0.00000       0.00000       0.00000       0.00000  
 E   11  5   7.00000       6.00000       0.00000       0.00000       0.00000  
 P   11  6   3.00000       5.00000       0.00000       0.00000       0.00000  
 E   12  1   10.0000       3.00000       0.00000       0.00000       0.00000  
 O   12  2   0.00000       0.00000       0.00000       0.00000       0.00000  
 W   12  3   0.00000       0.00000       0.00000       0.00000       0.00000  
 E   12  4   11.0000       2.00000       0.00000       0.00000       0.00000  
 E   12  5   8.00000       6.00000       0.00000       0.00000       0.00000  
 P   12  6   4.00000       5.00000       0.00000       0.00000       0.00000

The input for the mesh with curvature is:

          12  3          12 NEL,NDIM,NELV  
      ELEMENT             1 [    1a]    GROUP     0  
   0.000000       1.000000       1.000000       0.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
   0.000000       0.000000       0.000000       0.000000  
   0.000000       1.000000       1.000000       0.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.5000000E-01  0.5000000E-01  0.5000000E-01  0.5000000E-01  
      ELEMENT             2 [    1a]    GROUP     0  
   1.000000       2.000000       2.000000       1.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
   0.000000       0.000000       0.000000       0.000000  
   1.000000       2.000000       2.000000       1.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.5000000E-01  0.5000000E-01  0.5000000E-01  0.5000000E-01  
      ELEMENT             3 [    1a]    GROUP     0  
   0.000000       1.000000       1.000000       0.000000  
   0.000000       0.000000      0.5000000      0.5000000  
   0.000000       0.000000       0.000000       0.000000  
   0.000000       1.000000       1.000000       0.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.5000000E-01  0.5000000E-01  0.5000000E-01  0.5000000E-01  
      ELEMENT             4 [    1a]    GROUP     0  
   1.000000       2.000000       2.000000       1.000000  
   0.000000       0.000000      0.5000000      0.5000000  
   0.000000       0.000000       0.000000       0.000000  
   1.000000       2.000000       2.000000       1.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.5000000E-01  0.5000000E-01  0.5000000E-01  0.5000000E-01  
      ELEMENT             5 [    1a]    GROUP     0  
   0.000000       1.000000       1.000000       0.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.5000000E-01  0.5000000E-01  0.5000000E-01  0.5000000E-01  
   0.000000       1.000000       1.000000       0.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.1000000      0.1000000      0.1000000      0.1000000  
      ELEMENT             6 [    1a]    GROUP     0  
   1.000000       2.000000       2.000000       1.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.5000000E-01  0.5000000E-01  0.5000000E-01  0.5000000E-01  
   1.000000       2.000000       2.000000       1.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.1000000      0.1000000      0.1000000      0.1000000  
      ELEMENT             7 [    1a]    GROUP     0  
   0.000000       1.000000       1.000000       0.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.5000000E-01  0.5000000E-01  0.5000000E-01  0.5000000E-01  
   0.000000       1.000000       1.000000       0.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.1000000      0.1000000      0.1000000      0.1000000  
      ELEMENT             8 [    1a]    GROUP     0  
   1.000000       2.000000       2.000000       1.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.5000000E-01  0.5000000E-01  0.5000000E-01  0.5000000E-01  
   1.000000       2.000000       2.000000       1.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.1000000      0.1000000      0.1000000      0.1000000  
      ELEMENT             9 [    1a]    GROUP     0  
   0.000000       1.000000       1.000000       0.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.1000000      0.1000000      0.1000000      0.1000000  
   0.000000       1.000000       1.000000       0.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.1500000      0.1500000      0.1500000      0.1500000  
      ELEMENT            10 [    1a]    GROUP     0  
   1.000000       2.000000       2.000000       1.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.1000000      0.1000000      0.1000000      0.1000000  
   1.000000       2.000000       2.000000       1.000000  
 -0.5000000     -0.5000000       0.000000       0.000000  
  0.1500000      0.1500000      0.1500000      0.1500000  
      ELEMENT            11 [    1a]    GROUP     0  
   0.000000       1.000000       1.000000       0.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.1000000      0.1000000      0.1000000      0.1000000  
   0.000000       1.000000       1.000000       0.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.1500000      0.1500000      0.1500000      0.1500000  
      ELEMENT            12 [    1a]    GROUP     0  
   1.000000       2.000000       2.000000       1.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.1000000      0.1000000      0.1000000      0.1000000  
   1.000000       2.000000       2.000000       1.000000  
   0.000000       0.000000      0.5000000      0.5000000  
  0.1500000      0.1500000      0.1500000      0.1500000  
  ***** CURVED SIDE DATA *****  
        24 Curved sides follow IEDGE,IEL,CURVE(I),I=1,5, CCURVE  
  1  1  0.500000     -0.550000       0.00000       0.00000       0.00000     m  
  5  1  0.500000     -0.550000      0.500000E-01   0.00000       0.00000     m  
  1  2   1.50000     -0.450000       0.00000       0.00000       0.00000     m  
  5  2   1.50000     -0.450000      0.500000E-01   0.00000       0.00000     m  
  3  3  0.500000      0.550000       0.00000       0.00000       0.00000     m  
  7  3  0.500000      0.550000      0.500000E-01   0.00000       0.00000     m  
  3  4   1.50000      0.450000       0.00000       0.00000       0.00000     m  
  7  4   1.50000      0.450000      0.500000E-01   0.00000       0.00000     m  
  1  5  0.500000     -0.550000      0.500000E-01   0.00000       0.00000     m  
  5  5  0.500000     -0.550000      0.100000       0.00000       0.00000     m  
  1  6   1.50000     -0.450000      0.500000E-01   0.00000       0.00000     m  
  5  6   1.50000     -0.450000      0.100000       0.00000       0.00000     m  
  3  7  0.500000      0.550000      0.500000E-01   0.00000       0.00000     m  
  7  7  0.500000      0.550000      0.100000       0.00000       0.00000     m  
  3  8   1.50000      0.450000      0.500000E-01   0.00000       0.00000     m  
  7  8   1.50000      0.450000      0.100000       0.00000       0.00000     m  
  1  9  0.500000     -0.550000      0.100000       0.00000       0.00000     m  
  5  9  0.500000     -0.550000      0.150000       0.00000       0.00000     m  
  1 10   1.50000     -0.450000      0.100000       0.00000       0.00000     m  
  5 10   1.50000     -0.450000      0.150000       0.00000       0.00000     m  
  3 11  0.500000      0.550000      0.100000       0.00000       0.00000     m  
  7 11  0.500000      0.550000      0.150000       0.00000       0.00000     m  
  3 12   1.50000      0.450000      0.100000       0.00000       0.00000     m  
  7 12   1.50000      0.450000      0.150000       0.00000       0.00000     m  
  ***** BOUNDARY CONDITIONS *****  
  *****    FLUID   BOUNDARY CONDITIONS *****  
 W    1  1   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    1  2   2.00000       4.00000       0.00000       0.00000       0.00000  
 E    1  3   3.00000       1.00000       0.00000       0.00000       0.00000  
 v    1  4   0.00000       0.00000       0.00000       0.00000       0.00000  
 P    1  5   9.00000       6.00000       0.00000       0.00000       0.00000  
 E    1  6   5.00000       5.00000       0.00000       0.00000       0.00000  
 W    2  1   0.00000       0.00000       0.00000       0.00000       0.00000  
 O    2  2   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    2  3   4.00000       1.00000       0.00000       0.00000       0.00000  
 E    2  4   1.00000       2.00000       0.00000       0.00000       0.00000  
 P    2  5   10.0000       6.00000       0.00000       0.00000       0.00000  
 E    2  6   6.00000       5.00000       0.00000       0.00000       0.00000  
 E    3  1   1.00000       3.00000       0.00000       0.00000       0.00000  
 E    3  2   4.00000       4.00000       0.00000       0.00000       0.00000  
 W    3  3   0.00000       0.00000       0.00000       0.00000       0.00000  
 v    3  4   0.00000       0.00000       0.00000       0.00000       0.00000  
 P    3  5   11.0000       6.00000       0.00000       0.00000       0.00000  
 E    3  6   7.00000       5.00000       0.00000       0.00000       0.00000  
 E    4  1   2.00000       3.00000       0.00000       0.00000       0.00000  
 O    4  2   0.00000       0.00000       0.00000       0.00000       0.00000  
 W    4  3   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    4  4   3.00000       2.00000       0.00000       0.00000       0.00000  
 P    4  5   12.0000       6.00000       0.00000       0.00000       0.00000  
 E    4  6   8.00000       5.00000       0.00000       0.00000       0.00000  
 W    5  1   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    5  2   6.00000       4.00000       0.00000       0.00000       0.00000  
 E    5  3   7.00000       1.00000       0.00000       0.00000       0.00000  
 v    5  4   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    5  5   1.00000       6.00000       0.00000       0.00000       0.00000  
 E    5  6   9.00000       5.00000       0.00000       0.00000       0.00000  
 W    6  1   0.00000       0.00000       0.00000       0.00000       0.00000  
 O    6  2   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    6  3   8.00000       1.00000       0.00000       0.00000       0.00000  
 E    6  4   5.00000       2.00000       0.00000       0.00000       0.00000  
 E    6  5   2.00000       6.00000       0.00000       0.00000       0.00000  
 E    6  6   10.0000       5.00000       0.00000       0.00000       0.00000  
 E    7  1   5.00000       3.00000       0.00000       0.00000       0.00000  
 E    7  2   8.00000       4.00000       0.00000       0.00000       0.00000  
 W    7  3   0.00000       0.00000       0.00000       0.00000       0.00000  
 v    7  4   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    7  5   3.00000       6.00000       0.00000       0.00000       0.00000  
 E    7  6   11.0000       5.00000       0.00000       0.00000       0.00000  
 E    8  1   6.00000       3.00000       0.00000       0.00000       0.00000  
 O    8  2   0.00000       0.00000       0.00000       0.00000       0.00000  
 W    8  3   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    8  4   7.00000       2.00000       0.00000       0.00000       0.00000  
 E    8  5   4.00000       6.00000       0.00000       0.00000       0.00000  
 E    8  6   12.0000       5.00000       0.00000       0.00000       0.00000  
 W    9  1   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    9  2   10.0000       4.00000       0.00000       0.00000       0.00000  
 E    9  3   11.0000       1.00000       0.00000       0.00000       0.00000  
 v    9  4   0.00000       0.00000       0.00000       0.00000       0.00000  
 E    9  5   5.00000       6.00000       0.00000       0.00000       0.00000  
 P    9  6   1.00000       5.00000       0.00000       0.00000       0.00000  
 W   10  1   0.00000       0.00000       0.00000       0.00000       0.00000  
 O   10  2   0.00000       0.00000       0.00000       0.00000       0.00000  
 E   10  3   12.0000       1.00000       0.00000       0.00000       0.00000  
 E   10  4   9.00000       2.00000       0.00000       0.00000       0.00000  
 E   10  5   6.00000       6.00000       0.00000       0.00000       0.00000  
 P   10  6   2.00000       5.00000       0.00000       0.00000       0.00000  
 E   11  1   9.00000       3.00000       0.00000       0.00000       0.00000  
 E   11  2   12.0000       4.00000       0.00000       0.00000       0.00000  
 W   11  3   0.00000       0.00000       0.00000       0.00000       0.00000  
 v   11  4   0.00000       0.00000       0.00000       0.00000       0.00000  
 E   11  5   7.00000       6.00000       0.00000       0.00000       0.00000  
 P   11  6   3.00000       5.00000       0.00000       0.00000       0.00000  
 E   12  1   10.0000       3.00000       0.00000       0.00000       0.00000  
 O   12  2   0.00000       0.00000       0.00000       0.00000       0.00000  
 W   12  3   0.00000       0.00000       0.00000       0.00000       0.00000  
 E   12  4   11.0000       2.00000       0.00000       0.00000       0.00000  
 E   12  5   8.00000       6.00000       0.00000       0.00000       0.00000  
 P   12  6   4.00000       5.00000       0.00000       0.00000       0.00000

Note that element and boundary condition information are identical between the two cases.

boundary conditions Boundary conditions (BCs) are specified for each field in sequence: velocity, temperature and passive scalars. The section header for each field will be as follows (example for the velocity):

***** FLUID BOUNDARY CONDITIONS *****

and the data is stored as illustarted in Table. 3.3. For each field boundary conditions are listed for each face of each element.

Boundary conditions are given in order per each element, see Table. 3.3 column IEL, and faces listed in ascending order 1 - 6 in column IFACE. Note that the header in Table. 3.3 does not appear in the actual .rea. The ordering for faces each element is shown in Fig. 3.5. A total equivalent to 6Nfield boundary conditions are listed for each field, where Nfield is the number of elements for the specific field. Nfield is equal to the total number of fluid elements for the velocity and equal to the total number of elements (including solid elements) for temperature. For the passive scalars it will depend on the specific choice, but typically scalars are solved on the temeprature mesh (solid+fluid).


PIC

Figure 3.5: Face ordering for each element.


Each BC letter condition is formed by three characters. Common BCs include:

[1]
E - internal boundary condition. No additional information needs to be provided.
[2]
SYM - symmetry boundary condition. No additional information needs to be provided.
[3]
P - periodic boundary conditions, which indicates that an element face is connected to another element to establish a periodic BC. The connecting element and face need be to specified in CONN-IEL and CONN-IFACE.
[4]
v - imposed velocity boundary conditions (inlet). The value is specified in the user subroutines. No additional information needs to be provided in the .rea file.
[5]
W - wall boundary condition (no-slip) for the velocity. No additional information needs to be provided.
[6]
O - outlet boundary condition (velocity). No additional information needs to be provided.
[7]
t - imposed temperature boundary conditions (inlet). The value is specified in the user subroutines. No additional information needs to be provided in the .rea file.
[8]
f - imposed heat flux boundary conditions (temperature). The value is specified in the user subroutines. No additional information needs to be provided in the .rea file.
[9]
I - adiabatic boundary conditions (temeperature). No additional information needs to be provided.

Many of the BCs support either a constant specification or a user defined specification which may be an arbitrary function. For example, a constant Dirichlet BC for velocity is specified by V, while a user defined BC is specified by v. This upper/lower-case distinction is used for all cases. There are about 70 different types of boundary conditions in all, including free-surface, moving boundary, heat flux, convective cooling, etc. The above cases are just the most used types.










CBC IEL IFACE CONN-IEL CONN-IFACE
















E 1 1 4.00000 3.00000 0.00000 0.00000 0.00000
.. .. .. .. .. .. .. ..
W 5 3 0.00000 0.00000 0.00000 0.00000 0.00000
.. .. .. .. .. .. .. ..
P 23 5 149.000 6.00000 0.00000 0.00000 0.00000









Table 3.3: Formatting of boundary conditions input.

As in the case of the curvature entries, the boundary conditions entries are formatted and the format is dependent on the total number of elements. The code below shows an example of writing statement for boundary conditions:

                  if (nlg.lt.1000) then  
                     write(10,’(a1,a3,2i3,5g14.6)’)  
     $               chtemp,s3,eg,i,(vbc(ii,i,kb),ii=1,5)  
                  elseif (nlg.lt.100000) then  
                     write(10,’(a1,a3,i5,i1,5g14.6)’)  
     $               chtemp,s3,eg,i,(vbc(ii,i,kb),ii=1,5)  
                  elseif (nlg.lt.1000000) then  
                     write(10,’(a1,a3,i6,5g14.6)’)  
     $               chtemp,s3,eg,(vbc(ii,i,kb),ii=1,5)  
                  else  
                     write(10,’(a1,a3,i12,5g18.11)’)  
     $               chtemp,s3,eg,(vbc(ii,i,kb),ii=1,5)  
                  endif

The fortran format is as follows:

We note that:

Output info
restart conditions

Here, one can specify a file to use as an initial condition. The initial condition need not be of the same polynomial order as the current simulation. One can also specify that, for example, the velocity is to come from one file and the temperature from another. The initial time is taken from the last specified restart file, but this can be overridden.

history points

The following section defines history points in the .rea file, see example vortex/r1854a.rea, or shear4/shear4.rea

0 PACKETS OF DATA FOLLOW  
***** HISTORY AND INTEGRAL DATA *****  
    56 POINTS. H code, I,J,H,IEL  
UVWP    H     31     31   1   6  
UVWP    H     31     31   31  6  
UVWP    H     31     31   31  54  
 "      "      "      "    "   "

The "56 POINTS" line needs to be followed by 56 lines of the type shown. However, in each of the following lines, which have the UVWP etc., location is CRUCIAL, it must be layed out exactly as indicated above2, it is therefore advisable to refer to the examples vortex, shear4. If you want to pick points close to the center of element 1 and are running with lx1=10, say, you might choose UVWP H 5 5 5 1. 3

The UVWP tells the code to write the 3 velocity components and pressure to the .sch file at each timestep (or, more precisely, whenever mod(istep,iohis)=0, where iohis=param(52)). Note that if you have more than one history point then they are written sequentially at each timestep. Thus 10 steps in the first example with param(52)=2 would write (10/2)*56 = 280 lines to the .sch file, with 4 entries per line. The "H" indicates that the entry corresponds to a requested history point. A note of caution: if the ijk values (5 5 5 in the preceding example line) exceed lx1,ly1,lz1 of your SIZE file, then they are truncated to that value. For example, if lx1=10 for the data at the top (31 31 31) then the code will use ijk of (10 10 10), plus the given element number, in identifying the history point. It is often useful to set ijk to large values (i.e., > lx1) because the endpoints of the spectral element mesh are invariant when lx1 is changed.

output specifications Outputs are discussed in a separate section of the manual, available online.

It is important to note that Nek5000 currently supports two input file formats, ASCII and binary. The .rea file format described above is ASCII. For the binary format, all sections of the .rea file having storage requirements that scale with number of elements (i.e., geometry, curvature, and boundary conditions) are moved to a second, .re2, file and written in binary. The remaining sections continue to reside in the .rea file. The distinction between the ASCII and binary formats is indicated in the .rea file by having a negative number of elements. There are converters, reatore2 and re2torea, in the Nek5000 tools directory to change between formats. The binary file format is the default and important for I/O performance when the number of elements is large ( >  100000, say).

3.3.1 Parameters

In the solution of the governing equations together with the boundary and initial conditions, Nek5000 treats the above parameters as pure numerical values; their physical significance depends on the user’s choice of units. The system of units used is arbitrary (MKS, English, CGS, etc.). However, the system chosen must be used consistently throughout. For instance, if the equations and geometry have been non-dimensionalized, the μ∕ρ in the fluid momentum equation is in fact the inverse Reynolds number, whereas if the equations are dimensional, μ∕ρ represents the kinematic viscosity with dimensions of length + 2∕time.

3.4 Data Layout

Nek5000 was designed with two principal performance criteria in mind, namely, single-node performance and parallel performance.

A key precept in obtaining good single node performance was to use, wherever possible, unit-stride memory addressing, which is realized by using contiguously declared arrays and then accessing the data in the correct order. Data locality is thus central to good serial performance. To ensure that this performance is not compromised in parallel, the parallel message-passing data model is used, in which each processor has its own local (private) address space. Parallel data, therefore, is laid out just as in the serial case, save that there are multiple copies of the arrays—one per processor, each containing different data. Unlike the shared memory model, this distributed memory model makes data locality transparent and thus simplifies the task of analyzing and optimizing parallel performance.

Some fundamentals of Nek5000’s internal data layout are given below.

[1]
Data is laid out as ue  = u(i,j,k,e)
 ijk

i=1,...,nx1 (nx1 = lx1)
j=1,...,ny1 (ny1 = lx1)
k=1,...,nz1 (nz1 = lx1 or 1, according to ndim=3 or 2)

e=1,...,nelv, where nelv ≤ lelv, and lelv is the upper bound on number of elements, per processor.

[2]
Fortran data is stored in column major order (opposite of C).
[3]
All data arrays are thus contiguous, even when nelv <  lelv
[4]
Data accesses are thus primarily unit-stride (see chap.8 of DFM for importance of this point), and in particular, all data on a given processor can be accessed as, e.g.,
   do i=1,nx1*ny1*nz1*nelv  
      u(i,1,1,1) = vx(i,1,1,1)  
   enddo

which is equivalent but superior (WHY?) to:

   do e=1,nelv  
   do k=1,nz1  
   do j=1,ny1  
   do i=1,nx1  
      u(i,j,k,e) = vx(i,j,k,e)  
   enddo  
   enddo  
   enddo  
   enddo

which is equivalent but vastly superior (WHY?) to:

   do i=1,nx1  
   do j=1,ny1  
   do k=1,nz1  
   do e=1,nelv  
      u(i,j,k,e) = vx(i,j,k,e)  
   enddo  
   enddo  
   enddo  
   enddo

[5]
All data arrays are stored according to the SPMD programming model, in which address spaces that are local to each processor are private — not accessible to other processors except through interprocessor data-transfer (i.e., message passing). Thus
   do i=1,nx1*ny1*nz1*nelv  
      u(i,1,1,1) = vx(i,1,1,1)  
   enddo

means different things on different processors and nelv may differ from one processor to the next.

[6]
For the most part, low-level loops such as above are expressed in higher level routines only through subroutine calls, e.g.,:
   call copy(u,vx,n)

where n:=nx1*ny1*nz1*nelv. Notable exceptions are in places where performance is critical, e.g., in the middle of certain iterative solvers.

1we are inquiring whether other groups still use it

2these lines contain character strings, they use formatted reads

3the indicated point would really be at the middle of the element only if lx1=9