The present work is targeted at performing a strong scaling study of the high-order spectral element uid dynamics solver Nek5000. Prior studies such as [5] indicated a recommendable metric for strong scalability from a theoretical viewpoint, which we test here extensively on three parallel machines with different performance characteristics and interconnect networks, namely Mira (IBM Blue Gene/Q), Beskow (Cray XC40) and Titan (Cray XK7). The test cases considered for the simulations correspond to a turbulent ow in a straight pipe at four different friction Reynolds numbers Reτ = 180, 360, 550 and 1000. Considering the linear model for parallel communication we quantify the machine characteristics in order to better assess the scaling behaviors of the code. Subsequently sampling and profiling tools are used to measure the computation and communication times over a large range of compute cores. We also study the effect of the two coarse grid solvers XXT and AMG on the computational time. Super-linear scaling due to a reduction in cache misses is observed on each computer. The strong scaling limit is attained for roughly 5000 - 10; 000 degrees of freedom per core on Mira, 30; 000 - 50; 0000 on Beskow, with only a small impact of the problem size for both machines, and ranges between 10; 000 and 220; 000 depending on the problem size on Titan. This work aims at being a reference for Nek5000 users and also serves as a basis for potential issues to address as the community heads towards exascale supercomputers.

Hypre, a library for linear algebra, is used to replace a Matlab code for performing the setup step of an Algebraic Multigrid Method (AMG). The AMG method is used to compute part of the preconditioner in Nek5000, a code for Computational Fluid Dynamics based on the spectral element method. However, the solution of the AMG problem is not performed via Hypre but by Nek5000’s internal solver. The new AMG setup is shown to be faster by at least one order of magnitude, while it does not significantly impact the efficiency of the AMG solver, as is shown from its application to relevant test cases.

BoomerAMG, the algebraic multigrid solver from the hypre library, is used to solve a coarse grid problem which is part of the preconditioning strategy for thepressure equation arising from the numerical resolution of the Navier–Stokes equations. A set of optimal parameters for the setup phase is determined and used for selected strong scaling tests on two different supercomputers, namely Mira and Hazel Hen, on up to 131, 072 compute cores. The results are compared to an existing algebraic multigrid solver, designed specifically for the coarse gridproblem at hand. It is shown that the BoomerAMG solver is fast and scalable, and that performance depends on the computer architecture. The test cases considered are the turbulent flow past a NACA4412 airfoil and the turbulent flow inside wire-tapped pin bundles.

Adaptive mesh refinement (AMR) is an important component of modern numerical solvers, as it allows to control the computational error during the simulation, increasing the reliability of the numerical modelling and giving the possibility to study a broad range of different phenomena even without knowing the physics a priori. In this work we present selected aspects of the implementation and parallel performance of a new h−type AMR framework developed for the high-order CFD solver Nek5000; the development was done within the ExaFLOW EU project. We utilise in this case the natural domain decomposition inherent to the spectral element method (SEM), which constitutes the main source of parallelism and provides meshing flexibility that can be exploited in AMR. We use standard libraries for parallel mesh management (p4est) and partitioning (ParMetis) and focus on developing efficient preconditioners for the pressure problem solved on non-conforming meshes. Two different approaches are considered: an additive overlapping Schwarz and a hybrid Schwarz-multigrid method.The strong scaling is shown on the example of the simulation of the turbulent flow around a NACA4412 wing section at Rec = 200, 000.

Adaptive mesh refinement is performed in the framework of the spectral element method augmented by approaches to error estimation and control. The h-refinement technique is used for adapting the mesh, where selected grid elements are split by a quadtree (2D) or octree (3D) structure. Continuity between parent–child elements is enforced by high-order interpolation of the solution across the common faces. Parallel mesh partitioning and grid management respectively, are taken care of by the external libraries ParMETIS and p4est. Two methods are considered for estimating and controling the error of the solution. The first error estimate is local and based on the spectral properties of the solutionon each element. This method gives a local measure of the L2-norm of the solution over the entire computational domain. The second error estimate uses the dual-weighted residuals method — it is based on and takes into account both the local properties of the solution and the global dependence of the error in the solution via an adjoint problem. The objective of this second approach is to optimize the computation of a given functional of physical interest. The simulations are performed by using the code Nek5000 and three steady-state test cases are studied: a two-dimensional lid-driven cavity at Re = 7, 500, a two-dimensional flow past a cylinder at Re = 40, and a three-dimensional lid-driven cavity at Re = 2, 000 with a moving lid tilted by an angle of 30 degrees. The efficiency of both error estimators is compared in terms of refinement patterns and accuracy on the functional of interest. In the case of the adjoint error estimators, the trend on the error of the functional is shown to be correctly represented up to a multiplicative constant.

Unsteady adjoint error estimators based on the dual-weighted residuals method are implemented for the spectral element method in Nek5000. The time-integration of the adjoint problem is performed based on the nonlinear direction solution recomputed via the revolve algorithm, which uses an optimal check-pointing strategy. Adaptive mesh refinement is performed on the flow inside a constricted periodic channel, the so-called periodic hill case, at four different Reynolds numbers, Re = 700, 1400, 2800 and 5600. This case is fully turbulent at all regimes, with significant flow separation, requires curved meshes, but yet has a number of accurate reference solutions in the literature. The chosen method to adapt the mesh is h-refinement, where selected elements are split by an oct-tree structure in three dimensions. The objective function for the adjoint estimators is the integral of the friction forces along the flat bottom wall between the hills, for which the location of the reattachment becomes crucial. The refinement process is compared between the adjoint error estimators and classical straightforward a posteriori spectral error indicators based on the local approximation properties of the solution.The turbulent simulations using mesh adaptation are stable, free of spurious numerical noise and accurate, as shown by comparing the statistical profiles of relevant flow quantities with reference data. The comparison between the error estimators shows that the adjoint error estimators tend to refine the mesh only around localized regions in the computational domain while leaving other areas under-resolved. However, only the locally refined regions are shown to have a significant impact on the value of the objective function and thus on the location of the reattachment point. Conversely, the spectral error indicatorstend to homogenize the error on the solution over the whole domain but have a lesser direct influence on the location of the reattachment point.

The implementation of adaptive mesh refinement (AMR) in Nek5000 is used for the first time on the simulation of the flow over wings. This is done by simulating the flow over a NACA4412 profile with 5 degrees angle of attack at chord-based Reynolds number 200,000. The mesh is progressively refined by means of AMR which allows for high resolution near the wall whereas significantly larger elements are used in the far-field. The resultant mesh shows higher resolution than previous conformal meshes, and it allows for larger computational domains,which avoid the use of RANS to determine the boundary condition, all of this with, approximately, 3 times lower total number of grid points. The results ofthe turbulence statistics show a good agreement with the ones obtained with the conformal mesh. Finally, using AMR on wings leads to simulations at higher Reynolds numbers (i.e. Rec = 850, 000) in order to analyse the effect of adverse pressure gradients at high Reynolds numbers.