Adaptive mesh refinement (AMR) is an important component of modern numerical solvers, as it allows to control the computational error during the simulation, increasing the reliability of the numerical modelling and giving the possibility to study a broad range of different phenomena even without knowing the physics a priori. In this work we present selected aspects of the implementation and parallel performance of a new h−type AMR framework developed for the high-order CFD solver Nek5000; the development was done within the ExaFLOW EU project. We utilise in this case the natural domain decomposition inherent to the spectral element method (SEM), which constitutes the main source of parallelism and provides meshing flexibility that can be exploited in AMR. We use standard libraries for parallel mesh management (p4est) and partitioning (ParMetis) and focus on developing efficient preconditioners for the pressure problem solved on non-conforming meshes. Two different approaches are considered: an additive overlapping Schwarz and a hybrid Schwarz-multigrid method.The strong scaling is shown on the example of the simulation of the turbulent flow around a NACA4412 wing section at Rec = 200, 000.

BoomerAMG, the algebraic multigrid solver from the hypre library, is used to solve a coarse grid problem which is part of the preconditioning strategy for thepressure equation arising from the numerical resolution of the Navier–Stokes equations. A set of optimal parameters for the setup phase is determined and used for selected strong scaling tests on two different supercomputers, namely Mira and Hazel Hen, on up to 131, 072 compute cores. The results are compared to an existing algebraic multigrid solver, designed specifically for the coarse gridproblem at hand. It is shown that the BoomerAMG solver is fast and scalable, and that performance depends on the computer architecture. The test cases considered are the turbulent flow past a NACA4412 airfoil and the turbulent flow inside wire-tapped pin bundles.

Hypre, a library for linear algebra, is used to replace a Matlab code for performing the setup step of an Algebraic Multigrid Method (AMG). The AMG method is used to compute part of the preconditioner in Nek5000, a code for Computational Fluid Dynamics based on the spectral element method. However, the solution of the AMG problem is not performed via Hypre but by Nek5000’s internal solver. The new AMG setup is shown to be faster by at least one order of magnitude, while it does not significantly impact the efficiency of the AMG solver, as is shown from its application to relevant test cases.

Unsteady adjoint error estimators based on the dual-weighted residuals method are implemented for the spectral element method in Nek5000. The time-integration of the adjoint problem is performed based on the nonlinear direction solution recomputed via the revolve algorithm, which uses an optimal check-pointing strategy. Adaptive mesh refinement is performed on the flow inside a constricted periodic channel, the so-called periodic hill case, at four different Reynolds numbers, Re = 700, 1400, 2800 and 5600. This case is fully turbulent at all regimes, with significant flow separation, requires curved meshes, but yet has a number of accurate reference solutions in the literature. The chosen method to adapt the mesh is h-refinement, where selected elements are split by an oct-tree structure in three dimensions. The objective function for the adjoint estimators is the integral of the friction forces along the flat bottom wall between the hills, for which the location of the reattachment becomes crucial. The refinement process is compared between the adjoint error estimators and classical straightforward a posteriori spectral error indicators based on the local approximation properties of the solution.The turbulent simulations using mesh adaptation are stable, free of spurious numerical noise and accurate, as shown by comparing the statistical profiles of relevant flow quantities with reference data. The comparison between the error estimators shows that the adjoint error estimators tend to refine the mesh only around localized regions in the computational domain while leaving other areas under-resolved. However, only the locally refined regions are shown to have a significant impact on the value of the objective function and thus on the location of the reattachment point. Conversely, the spectral error indicatorstend to homogenize the error on the solution over the whole domain but have a lesser direct influence on the location of the reattachment point.

The implementation of adaptive mesh refinement (AMR) in Nek5000 is used for the first time on the simulation of the flow over wings. This is done by simulating the flow over a NACA4412 profile with 5 degrees angle of attack at chord-based Reynolds number 200,000. The mesh is progressively refined by means of AMR which allows for high resolution near the wall whereas significantly larger elements are used in the far-field. The resultant mesh shows higher resolution than previous conformal meshes, and it allows for larger computational domains,which avoid the use of RANS to determine the boundary condition, all of this with, approximately, 3 times lower total number of grid points. The results ofthe turbulence statistics show a good agreement with the ones obtained with the conformal mesh. Finally, using AMR on wings leads to simulations at higher Reynolds numbers (i.e. Rec = 850, 000) in order to analyse the effect of adverse pressure gradients at high Reynolds numbers.

The development of adaptive mesh refinement capabilities in the field of computational fluid dynamics is an essential tool for enabling the simulation of larger and more complex physical problems. In this report, we describe recent developments that have been made to enable adaptive mesh refinement in Nek5000 and we validate the method on simple, two-dimensional, steady test cases.We start by describing the modifications brought to Nek5000 that enable the presence of hanging nodes in the mesh. Thanks to this new feature, we can use the h-refinement technique for mesh adaptation, where selected elements are split via quadtree (2D) or octree (3D) structures. Then, two methods are considered to estimate and control the error. The first method is local and based on the spectral properties of the solution on each element. The second method is goal-oriented and takes into account both the local properties of the solution and the global dependence of the error in the solution via the resolution of an adjoint problem. Finally, the use of automatic mesh refinement is demonstrated in Nek5000 on two test cases: the lid-driven cavity at Re = 7, 500 and the flow past a cylinder at Re = 40. Both error estimation methods are compared andare shown to efficiently reduce the number of degrees of freedom required to reach a given tolerance on the solution compared to conforming refinement. Moreover, the gains in terms of mesh generation, accuracy and computational cost are discussed by analysing the convergence of some functional of interest and the evolution of the mesh as refinement proceeds.

Hypre, a library for linear algebra, is used to replace a Matlab code for performing the setup step of an Algebraic Multigrid Method (AMG). The AMG method is used to compute part of the preconditioner in Nek5000, a code for Computational Fluid Dynamics based on the spectral element method. However, the solution of the AMG problem is not performed via Hypre but by Nek5000’s internal solver. The new AMG setup is shown to be faster by at least one order of magnitude, while it does not significantly impact the efficiency of the AMG solver, as is shown from its application to relevant test cases.

We discuss parallel performance of h-type Adaptive Mesh Refinement (AMR) developed for the high-order spectral element solver Nek5000 within CRESTA project. AMR is a desired feature of the future simulation software, as it gives possibility to increase the accuracy of numerical simulations at minimal computational cost by resolving particular region of the domain. At the same time it increases complexity of the communication pattern and introduces load imbalance, that can have negative effect on the code scalability. In this work we concentrate on the parallel performance of different tools required by AMR and the resulting algorithm limitations. Our implementation is based on available libraries for parallel mesh management (p4est) and partitioning (ParMetis) that provide necessary information for grid refinement/coarsening and redistribution performed within nonconforming version of Nek5000. For simplicity we consider advection-diffusion problem instead of the full Navies-Stokes equations and study both strong and weak scalability for the convected-cone problem. It is a synthetic test case allowing to test AMR with frequent dynamic mesh adjustments.

We study the stability of a jet in cross-flow at low values of the jet to cross-flow velocity ratio R using direct numerical simulations (DNS) and global linear stability analysis adopting a time-stepper method. For the simplified setup without a meshed pipe in the simulations we compare results of a fully-spectral code SIMSON with a spectral-element code Nek5000. We find the use of periodic domains, even with the fringe method, unsuitable due to the large sensitivity of the eigenvalues and due to the large spatial growth of the corresponding eigenmodes. However, we observe a similar sensitivity to reflection from the outflow boundary in the inflow/outflow configuration, and therefore we use an extended domain where reflections are minimal. We apply in our studies both modal and non-modal linear analyses investigating transient effects and their asymptotic fate, and we find a transient wavepacket to develop almost identically in both the globally stable and unstable cases. The final results of the global stability analysis for our numerical setup show the critical value of R, at which the first bifurcation occurs, to lie in the range between 1.5 and 1.6.