FAQ

General

Where can I download the code and what version of the code should I use?

We strongly recommend to always use the latest release available. We do not recommend using the master branch on GitHub in a production environment!

How can I properly reference Nek5000?

If you use our software, please cite the following:
NEK5000 Version X.Y. Release date. Argonne National Laboratory, Illinois.
Available: https://nek5000.mcs.anl.gov.

What is the license for Nek5000?

Nek5000 is licensed under BSD. For more information see the LICENSE file included with the distribution in the root level directory.

Where can I get help?

If you have a question, first check the Google group to see if your question is already answered somewhere. The google group also serves as a primary support channel with the Nek5000 user community. Please subscribe to the google group by clicking the button “Apply to join group” right above the “Welcome to the new Nek5000 User Group!” sign.

The past mailing list archive can also be checked for potential answers.

How can I report a bug / feature request?

Nek5000 is hosted on GitHub and all bugs are reported and tracked through the Issues feature on GitHub. However, GitHub Issues should not be used for common troubleshooting purposes. If you are having trouble installing the code or getting your model to run properly, you should first send a message to the User’s Group mailing list. If it turns out your issue really is a bug in the code, an issue will then be created on GitHub. If you want to request that a feature be added to the code, you may create an Issue on GitHub.

How can I contribute to the Nek5000 project?

Our project is hosted on GitHub. Here are the most important things you need to know:

  • follow the usual “fork-and-pull” Git workflow
  • all development happens on the master branch
  • anything in master is always deployable
  • upcoming releases get their own tags out of master

If you are planning a large contribution, we encourage you to discuss the concept here on GitHub and interact with us frequently to ensure that your effort is well-directed.

Installing, Compiling, and Running

Which platforms are supported by Nek5000?

All posix compliant operations system. To run on Windows, we recommend using Ubuntu 20.04 on V2 of the Windows Subsystem for Linux.

Can Nek5000 run on GPUs?

We are currently developing the NekRS code to support GPUs. The repository is available on Github.

Which compilers work?

Currently Intel, PGI, IBM and GNU compilers are supported.

How much memory is required?

The memory footprint of a run depends on many factors and is printed to screen whenever Nek5000 exits. What follows is a first rough guess:
lx1*ly1*lz1*lelt * 3000byte + lelg * 12byte + MPI + optional libraries (e.g. CVODE)

where lelt (the maximum number of local elements) is computed as lelg/lpmin. The memory allocated by MPI will depend heavily on the total number of ranks and the considered MPI implementation. For large rank counts (say \(E>100,000\)) it’s easily 50-100MB.

Note, the output of GNU`s SIZE utility is inaccurate as it does not take into account the dynamic memory alloation of MPI, gslib, CVODE, etc.

Why does the compiler issue relocation errors?

relocation truncated to fit: R_X86_64 against symbol 'foo_' defined in COMMON section in obj/bar.o

This happens when the resultant executable requires more than 2GB of static data. The best way to avoid this is to increase the minimum number of MPI ranks, lpmin in SIZE. Alternatively, you can add -mcmodel=medium to the FFLAGS/CFLAGS variable in makenek. See Compiler Flags for more information.

Why does the code use static memory allocation?

It is easy to program, produces fast code and is less error prone. The drawback of recompilation is not relevant as full rebuild takes less than 1 minute.

How do I launch a parallel run?

Assuming your machine has MPI installed correctly, parallel runs can be launced with the included nekmpi and nekbmpi scripts in Nek5000/bin. If you are running on a machine with a queuing system, consult your sysadmin how to submit a job.

How do I run the examples?

The examples are included by default in the release tarball (see example directory). There is nothing special you need to do as they are ready to run. If you cloned the github repo, you will also need to clone the NekExamples repository separately.

How do I run a case with more than 350,000 elements?

To run a large case, you will need to compile with HYPRE support and use one of the AMG pressure preconditioners. To compile with HYPRE see Preprocessor List. To select an AMG preconditioner see PRESSURE keys in the .par file.

Pre-Processing and Numerics

How can I generate a mesh for use with Nek5000?

Please see quickstart section on Meshing.

What element types are supported?

Conformal curved quadrilateral/hexahedral elements.

How do I import/convert a mesh to Nek5000?

We currently support conversion from the exodusII with the exo2nek converter. This enables the import from popular mesh generators like ANSYS ICEM and CUBIT.

I received a warning message about Lanczos failing to reach a residual target, what does this mean and what should I do about it?

Warning: Lanczos reached a residual of 12.840714 (target: 0.000010) after 50 x 50 iterations in Level=7!

This warning is from parRSB (parallel recursive spectral bisection) which can be used to distribute the elements to MPI ranks for large problems in lieu of genmap (see Preprocessor List). parRSB guarantees “load-balancing”, meaning the numbers of elements for each MPI rank are as close as possible. It also tries to minimize the communication between ranks by finding the minimal number of separators for the graph partition. For each cut, we find the Fiedler vector, the eigenvector of the smallest positive eigenvalue of the graph Laplacian. The warning you simply means Lanczos has hard time to converge at one of the levels, which is commonly shown for complex geometries at scale. As long as it doesn’t produce an error, generally the warning can be ignored. However, it’s recommended to also track the quality of the partition. You can find lines like these

nElements   max/min/bal: 10290 10289 1.00        << num. elem. per MPI rank.
nMessages   max/min/avg: 51 5 15.97              << indicating num. of MPI neighbors
msgSize     max/min/avg: 92191 1 12156.50        << size of each message
msgSizeSum  max/min/avg: 428087 104476 187446.11 << total message size per MPI rank.
There are some knobs we can adjust for the partitioning, but this only matters when you are in the communication dominant region.

Why is it important to non-dimensionalize my case?

Nek5000 can be run with dimensions, but we STRONGLY recommend that the case has been non-dimensionalized properly. An advantage of the nondimensional form is that physical simulation times, tolerances, etc. tend to be easy to set based on prior experience with other simulations.

How do I choose solver tolerances?

Depends on how accurate you need your simulation to be. Typical values (for engineering type of problems) are \(10^{-7}\) for velocity and scalars. In Pn/Pn-2 the pressure tolerance is equal to desired error in divergence. This is in contrast to Pn/Pn where the divergence error is mainly a function of spatial resolution and a tolerance of \(10^{-4}\) is typically good enough. Note the tolerances are related to the residual in the linear solve and do not represent the accuracy of the solution.

What formulation Pn/Pn or Pn/Pn-2 should I use?

There is no simple answer but we typically recommend to use the Pn/Pn formulation altough not all features are supported (at least for now).

What polynomial order should I use?

The code supports a large range of polynomial orders, e.g. \(N=1\) through \(N=32\). You can effectively realize the same number of grid points by using relatively few high-order elements or more low-order elements. For example, a 3D grid with resolution of 64x64x64 could be implemented as a 16x16x16 array of elements of order \(N=3\) or as a 8x8x8 array of elements of order \(N=7\). In Nek5000, the latter is preferred. The solution will be more accurate and the code is optimized for this range of \(N\).

The sweet spot is typically \(N=7\) (lx1=8).

How do I specify/change the polynomial order?

Change lx1 in the SIZE file. Note, the polynomial order is \(N=lx1-1\).

How do I specify/change the solver runtime parameters?

See the section on the Parameter File (.par) file.

Why is ``userbc`` only called for certain element faces?

userbc is ONLY called for element boundary conditions specified with a lower-case letter, e.g. ‘v’, ‘t’, or ‘o’ but NOT ‘W’, ‘E’, or ‘O’. Note that this implies it is not necesarily called on all MPI ranks.

How do I solve for a scalar?

Nek5000 supports solving up to 99 additional scalars. To solve an additional scalar equation, increase ldimt in the SIZE file to accomodate the additional scalar and specify the appropriate parameter in the Parameter File (.par) file. See shear4 example for more details.

Are there any books/papers that describe the numerics of Nek5000?

There are a number of descriptions of the various numerical methods used in Nek5000 available. Probably the best starting point is the book High-Order Methods for Incompressible Fluid Flow by Deville et al. (2002). There are also other, perhaps shorter, expositions of the material. Two papers that we found particularly useful (there are of course many more) are:

  • Fischer. An Overlapping Schwarz Method for Spectral Element Solution of the Incompressible Navier–Stokes Equations. J. Comput. Phys. 133, 84–101 (1997)
  • Fischer et al. Simulation of high-Reynolds number vascular flows. Comput. Methods Appl. Mech. Engrg. 196 (2007) 3049–3060

and also the lecture notes by Paul Fischer (given at KTH in 2016).

Why can I see sometimes the imprint of elements in the solution?

Nek5000 is based on the spectral-element method, which relies on an expansion of the solution in terms of element-local basis functions. These basis functions are the Lagrange interpolants to the Legendre polynomials of a specific order. If using PnPn-2, the velocity is on the Gauss-Lobatto-Legendre mesh (i.e. including the boundary points), and the pressure is on the Gauss-Legendre mesh (without boundary points). These functions are defined within each element, and the continuity between elements is C0, i.e. only the function value is the same. The ansatz functions are polynomials, so you can differentiate them inside each element; however, derivatives are not continuous over element boundaries (even though this difference reduces spectrally fast). Note that for the PnPn-2 method, the pressure is non-continuous.

This means that when visualising e.g. derivatives, one might see discontinuities, which then appear as imprints of the elements. This is due to the mentioned properties of the discretisation, and as such not a sign of a wrong solution. With increasing resolution (either p or h-type) these jumps will most certainly get smaller.

Physical Models

What turbulence models are available in Nek5000?

For LES we provide an explicit filtering approach or a relaxation term model. We currently offer a selection of RANS models from the k-ω class as “experimental” features. These include the k-τ model and regularized versions of the standard k-ω, and k-ω SST models. See the RANSChannel example for further info.

Computational Speed

Are there any compiler specific flags I should use?

Compile with vector instructions like AVX, AVX2 using FFLAGS and CFLAGS in makenek.

How many elements should I have per process?

The upper limit is given by the available memory. The lower limit is (technically) 1 but you may want to have more elements (work) to get a reasonable (whatever that means for you) parallel efficiency. On most machines you need more than 10 elements per MPI rank to get a parallel efficiency of 0.5 (assuming N=7). We recommend you perform a small scaling study on your machine to get a better estimate.

Should I use residual projection?

Typically projection is used for pressure but not velocity, however this is highly case specific and a simple experiment will show if it pays off or not. Projection will speed up the solution to a scalar, but takes time to compute itself. A scalar solve requiring ~40 iterations or greater is a good candidate for use.

What other things can I do to get best performance?

  • Design your mesh for a polynomial order N=7
  • Tune your solver tolerances
  • Increase time step size by switching to 2nd order BDF and OIFS extrapolation (target Courant number 2-5)
  • Use AMG instead of XXT as coarse grid solver
  • Avoid unnecessary time consuming operations in usrchk/userbc
  • Use binary input files e.g. .re2 and .ma2 to minimize solver initialization time
  • Use a high performance MXM implementation for your platform (see makenek options)

Troubleshooting

Why does Nek hang or produce an error when reading the boundary faces from the re2 file?

reading boundary faces                28720 for field 1
  readp_re2_bc:pack/cr/unpack : 0.73E-02 0.18E-0.1 0.61E-02
  readp_re_bc:byte_read_mpi   : 0.65E+01
ERROR: Error reading .re2 boundary data  ierr=120
By default, Nek5000 expects to find a set of boundary conditions for every field in the simulation, i.e. velocity, temperature, and passive scalars. Third party meshes converted with gmsh2nek or exo2nek typically only include a single set of boundary IDs. This error can be resolved by adding the following to the .par file.
[MESH]
numberOfBCFields = 1

My simulation diverges. What should I do?

There are many potential root causes but here are some things you can experiment with:

  • lower the time step (in particular during initial transients)
  • reduce time integration order (e.g. use 2 instead of 3)
  • increase spatial resolution
  • provide a better initial condition
  • check that your boundary conditions are meaningful and correctly implemented
  • visualize the solution and look for anomalies

Post-Processing

What options are available

  • For data analysis you use Nek5000’s internal machinery through the usr file
  • Solution files can be read by VisIt and Paraview (for more information see Visualization)

The local coordinate axes of my elements are not aligned with the global coordinate system, is this normal?

Yes, there is no guarantee that the elements are generated with any particular orientation (except if you use genbox).

Where are my solution files?

By default Nek5000 outputs solution files in binary <casename>0.f%05d.

I have calculated additional fields from my solution, how do I visualize them?

Using the .par file, define an additional scalar and include the solver=none option. For example:
[SCALAR01] # lambda2 vortex criterion
solver = none
Then store the calculated field in t(1,1,1,1,iscal+1) where iscal is your passive scalar index (in this example 1). The scalar will then be output by default with the solution files.

How do I obtain values of variables at a specific point?

The simplest way is through the use of history points. See the section on the History Points file. You can also use the spectral interpolation tool (see examples for more details).