Chapter 2
Quick start

This chapter provides a quick overview to using Nek5000 for some basic flow problems provided in the .../examples directory.

Nek5000 runs under Linux or any Linux-like OS such as MAC, AIX, BG, Cray etc. The source is maintained in a legacy SVN repository (last updated June 2, 2016) and up-to-date Git repositories. Both repositories are available for download, but we recommended that new users work with the Git repositories.

2.1 The Git Repositories

2.1.1 Downloading the source code

The Nek5000 source code is hosted on GitHub at https://github.com/Nek5000/Nek5000. You can download the source code as a .zip file1 or a .tar.gz archive2. You can also download (or “clone”) the repository itself. The following commands will clone the repository in a directory named $HOME/Nek5000 (where $HOME denotes your home directory):

cd && git clone https://github.com/Nek5000/Nek5000.git -b master

You can also clone the repository in any directory of your choice:

cd $HOME/my-repos/ && git clone https://github.com/Nek5000/Nek5000.git -b master

2.1.2 Downloading the example problems

The Nek5000 example problems are hosted in a second repository on GitHub, https://github.com/Nek5000/NekExamples. Likewise, you can download the examples as a .zip archive3, a .tar.gz archive4, or you may clone the repository to your home directory:

cd && git clone https://github.com/Nek5000/NekExamples.git -b master

or a directory of your choice:

cd $HOME/my-repos/ && git clone https://github.com/Nek5000/Nek5000.git -b master

2.1.3 Tools and scripts

The Nek5000/tools directory contains programs for pre- and post-processing tasks, such as generating meshes from geometry descriptions. The following commands will build the tools in $HOME/bin:

cd Nek5000/tools && ./maketools all

By default, the maketools script will use gfortran and gcc as the Fortran and C compilers; you can specify a different set of compilers by editing maketools. You can also build the tools in another directory of your choice by providing a second argument to maketools; for example:

cd $HOME/my-repos/Nek5000/tools && ./maketools all $HOME/my-tools-bin/

Besides the compiled tools, there are several convenience scripts in Nek5000/bin that allow you to easily set up and execute runs of Nek5000. These scripts are executable as-is and do not need to be compiled.

We recommend that you add the paths to the tools and scripts to the $PATH variable in your shell’s environment. This will allow you execute the tools and scripts without typing the full path to the executables. In the bash shell, if you have cloned Nek5000 in $HOME/Nek5000 and compiled tools in $HOME/bin, you can edit your executable path with:

export PATH="$HOME/bin:$HOME/Nek5000/bin:$PATH"

or if the Nek5000 repository and tools are in custom locations, use:

export PATH="$HOME/my-tools-bin:$HOME/my-repos/Nek5000/bin:$PATH"

In the following examples, we will assume that the tools and scripts have been added to your $PATH.

2.2 The SVN Repository

2.2.1 Downloading the source code and examples

The SVN repository can be downloaded with the following commands:

cd && svn co https://svn.mcs.anl.gov/repos/nek5 nek5_svn

This will create a directoy named nek5_svn in your home directory. The example problems are included in the same SVN repo and do not need to be downloaded separately.

2.2.2 Tools and scripts

The nek5_svn/trunk/tools directory contains programs for pre- and post-processing tasks, such as generating meshes from geometry descriptions. The following commands will build the tools in $HOME/bin:

cd $HOME/nek5_svn/trunk/tools && ./maketools all

By default, the maketools script will use gfortran and gcc as the Fortran and C compilers; you can specify a different set of compilers by editing maketools.

Besides the compiled tools, there are several convenience scripts in nek5_svn/trunk/tools/scripts that allow you to easily set up and execute runs of Nek5000. These scripts are executable as-is and do not need to be compiled.

We recommend that you append the paths to the tools and scripts to the $PATH variable in your shell’s environment. This will allow you execute the tools and scripts without typing the full path to the executables. In the bash shell, you can edit your executable path with:

export PATH="$HOME/bin:$HOME/nek5_svn/trunk/tools/scripts:$PATH"

In the following examples, we will assume that the tools and scripts have been added to your $PATH.

2.3 A Worked Example

As a first example, we consider the eddy problem due to Walsh 5. To get started, execute the following commands for the Git repositories:

cd  
mkdir eddy  
cd eddy  
cp $HOME/NekExamples/eddy/* .  
cp $HOME/Nek5000/core/makenek .

or the equivalent commands for the SVN repository:

cd  
mkdir eddy  
cd eddy  
cp $HOME/nek5_svn/examples/eddy/* .  
cp $HOME/nek5_svn/trunk/nek/makenek .

Modify makenek.

If you do not have mpi installed on your system, edit makenek, uncomment the IFMPI="false" flag, and change the Fortran and C compilers according to what is available on your machine. (Most any Fortran compiler save g77 or g95 will work.)

Nek5000 is written in F77 which has implicit typesetting as default. This means in practice that if the user defines a new variable in the user file and forgets to define its type explicitly then variable beginning with a character from I to N, its type is INTEGER. Otherwise, it is REAL.

This common type of mistake for a beginner can be avoided using a warning flag -Wimplicit. This flag warns whenever a variable, array, or function is implicitly declared. Has an effect similar to using the IMPLICIT NONE statement in every program unit.

Another useful flag may -mcmodel which allows for arrays of size larger than 2GB. This option tells the compiler to use a specific memory model to generate code and store data. It can affect code size and performance. If your program has global and static data with a total size smaller than 2GB, -mcmodel=small is sufficient. Global and static data larger than 2GB requires -mcmodel=medium or -mcmodel=large.

If you have mpi installed on your system or have made the prescribed changes to makenek, the eddy problem can be compiled as follows

Compiling nek. makenek eddy_uv

If all works properly, upon comilation the executable nek5000 will be generated using eddy_uv.usr to provide user-supplied initial conditions and analysis. Note that if you encountered a problem during a prior attempt to build the code you should type

makenek clean;

makenek eddy_uv

Once compilation is successful, start the simulation by typing

Running a case: nekb eddy_uv

which runs the executable in the background (nekb, as opposed to nek, which will run in the foreground). If you are running on a multi-processor machine that supports MPI, you can also run this case via

A parallel run: nekbmpi eddy_uv 4

which would run on 4 processors. If you are running on a system that supports queuing for batch jobs (e.g., pbs), then the following would be a typical job submission command

nekpbs eddy_uv 4

In most cases, however, the details of the nekpbs script would need to be modified to accommodate an individual’s user account, the desired runtime and perhaps the particular queue. Note that the scripts nek, nekb, nekmpi, nekbmpi, etc. perform some essential file manipulations prior to executing nek5000, so it is important to use them rather than invoking nek5000 directly.

To check the error for this case, type

grep -i err eddy_uv.log | tail

or equivalently

grep -i err logfile | tail

where, because of the nekb script, logfile is linked to the .log file of the given simulation. If the run has completed, the above grep command should yield lines like

 1000  1.000000E-01  6.759103E-05  2.764445E+00  2.764444E+00  1.000000E+00  X err  
 1000  1.000000E-01  7.842019E-05  1.818632E+00  1.818628E+00  3.000000E-01  Y err

which gives for the x- and y-velocity components the step number, the physical time, the maxiumum error, the maximum exact and computed values and the mean (bulk) values.

A common command to check on the progress of a simulation is

grep tep logfile | tail

which typically produces lines such as

Step    996, t= 9.9600000E-02, DT= 1.0000000E-04, C=  0.015 4.6555E+01 3.7611E-02

indicating, respectively, the step number, the physical time, the timestep size, the Courant (or CFL) number, the cumulative wall clock time (in seconds) and the wall-clock time for the most recent step. Generally, one would adjust Δt to have a CFL of ~0.5.

2.4 Viewing the First 2D Example

The preferred mode for data visualization and analysis with Nek5000 is to use VisIt. For a quick peek at the data, however, we list a few commands for the native Nek5000 postprocessor. Assuming that the maketools script has been executed and that /bin is in the execution path, then typing

postx

in the working directory should open a new window with a sidebar menu. With the cursor focus in this window (move the cursor to the window and left click), hit return on the keyboard accept the default session name and click PLOT with the left mouse button. This should bring up a color plot of the pressure distribution for the first output file from the simulation (here, eddy_uv.fld01), which contains the geometry, velocity, and pressure.

Alternatively one can use the script visnek, to be found in /scripts. It is sufficent to run

visnek eddy_uv (or the name of your session)

to obatain a file named eddy_uv.nek5000 which can be recognized in VisIt 6

Plotting the error: For this case, the error has been written to eddy_uv.fld11 by making a call to outpost() in the userchk() routine in eddy_uv.usr. The error in the velocity components is stored in the velocity-field locations and can be viewed with postx, or VisIt as before.

2.4.1 Modifying the First Example

A common step in the Nek5000 workflow is to rerun with a higher polynomial order. Typically, one runs a relatively low-order case (e.g., lx1=5) for one or two flow-through times and then uses the result as an initial condition for a higher-order run (e.g., lx1=8). We illustrate the procedure with the eddy_uv example.

Assuming that the contents of nek5_svn/trunk/tools/scripts are in the execution path, begin by typing

cp eddy_uv eddy_new

which will copy the requisite eddy_uv case files to eddy_new. Next, edit SIZE and change the two lines defining lx1 and lxd from

      parameter (lx1=8,ly1=lx1,lz1=1,lelt=300,lelv=lelt)  
      parameter (lxd=12,lyd=lxd,lzd=1)

to

      parameter (lx1=12,ly1=lx1,lz1=1,lelt=300,lelv=lelt)  
      parameter (lxd=18,lyd=lxd,lzd=1)

Then recompile the source by typing

makenek eddy_new

Next, edit eddy_new.rea and change the line

            0 PRESOLVE/RESTART OPTIONS  *****

(found roughly 33 lines from the bottom of the file) to

            1 PRESOLVE/RESTART OPTIONS  *****  
eddy_uv.fld12

which tells nek5000 to use the contents of eddy_uv.fld12 as the initial condition for eddy_new. The simulation is started in the usual way:

nekb eddy_new

after which the command

grep err logfile | tail

will show a much smaller error (~ 10 + -9) than the lx1=8 case.

Note that one normally would not use a restart file for the eddy problem, which is really designed as a convergence study. The purpose here, however, was two-fold, namely, to illustrate a change of order and its impact on the error, and to demonstrate the frequently-used restart procedure. However for a higher order timestepping scheme an accurate restart would require a number of field files of the same size (+1) as the order of the multistep scheme

1https://github.com/Nek5000/Nek5000/archive/master.zip

2https://github.com/Nek5000/Nek5000/archive/master.tar.gz

3https://github.com/Nek5000/NekExample/archive/master.zip

4https://github.com/Nek5000/NekExamples/archive/master.tar.gz

5O. Walsh, “Eddy solutions of the Navier-Stokes equations,” The NSE II-Theory and Numerical Methods, J.G. Heywood, K. Masuda, R. Rautmann, and V.A. Solonikkov, eds., Springer, pp. 306–309 (1992)

6https://wci.llnl.gov/simulation/computer-codes/visit/