Flush-mounted cavity hot-wire probes have emerged as an alternative to classical hot-wire probes mounted several diameters above the surface for wall-shear stress measurements. They aim at increasing the frequency response and accuracy by circumventing the well-known issue of heat transfer to the substrate that hot-wire and hot-film probes possess. Their use, however, depends on the assumption that the cavity does not influence the flow field. In this study, we show that this assumption does not hold, and that turbulence statistics are modified by the presence of the cavity with sizes that are practically in use. The mean velocity and fluctuations increase near the cavity while the shear stress decreases in its surroundings, all seemingly stemming from the fact that the no-slip condition is not present anymore and that flow reversal occurs. Overall, the energy spectra and the probability density function of the wall shear stress fluctuations indicate a change of nature of turbulence by the presence of the cavity.

High-Performance Computing (HPC) systems provide input/output (IO) performance growing relatively slowly compared to peak computational performance and have limited storage capacity. Computational Fluid Dynamics (CFD) applications aiming to leverage the full power of Exascale HPC systems, such as the solver Nek5000, will generate massive data for further processing. These data need to be efficiently stored via the IO subsystem. However, limited IO performance and storage capacity may result in performance, and thus scientific discovery, bottlenecks. In comparison to traditional post-processing methods, in-situ techniques can reduce or avoid writing and reading the data through the IO subsystem, promising to be a solution to these problems. In this paper, we study the performance and resource usage of three in-situ use cases: data compression, image generation, and uncertainty quantification. We furthermore analyze three approaches when these in-situ tasks and the simulation are executed synchronously, asynchronously, or in a hybrid manner. In-situ compression can be used to reduce the IO time and storage requirements while maintaining data accuracy. Furthermore, in-situ visualization and analysis can save Terabytes of data from being routed through the IO subsystem to storage. However, the overall efficiency is crucially dependent on the characteristics of both, the in-situ task and the simulation. In some cases, the overhead introduced by the in-situ tasks can be substantial. Therefore, it is essential to choose the proper in-situ approach, synchronous, asynchronous, or hybrid, to minimize overhead and maximize the benefits of concurrent execution.

The computational power of High-Performance Computing (HPC) systems is constantly increasing, however, their input/output (IO) performance grows relatively slowly, and their storage capacity is also limited. This unbalance presents significant challenges for applications such as Molecular Dynamics (MD) and Computational Fluid Dynamics (CFD), which generate massive amounts of data for further visualization or analysis. At the same time, checkpointing is crucial for long runs on HPC clusters, due to limited walltimes and/or failures of system components, and typically requires the storage of large amount of data. Thus, restricted IO performance and storage capacity can lead to bottlenecks for the performance of full application workflows (as compared to computational kernels without IO). In-situ techniques, where data is further processed while still in memory rather to write it out over the I/O subsystem, can help to tackle these problems. In contrast to traditional post-processing methods, in-situ techniques can reduce or avoid the need to write or read data via the IO subsystem. They offer a promising approach for applications aiming to leverage the full power of large scale HPC systems. In-situ techniques can also be applied to hybrid computational nodes on HPC systems consisting of graphics processing units (GPUs) and central processing units (CPUs). On one node, the GPUs would have significant performance advantages over the CPUs. Therefore, current approaches for GPU-accelerated applications often focus on maximizing GPU usage, leaving CPUs underutilized. In-situ tasks using CPUs to perform data analysis or preprocess data concurrently to the running simulation, offer a possibility to improve this underutilization.

We detail our developments in the high-fidelity spectral-element code Neko that are essential for unprecedented large-scale direct numerical simulations of fully developed turbulence. Major inno- vations are modular multi-backend design enabling performance portability across a wide range of GPUs and CPUs, a GPU-optimized preconditioner with task overlapping for the pressure-Poisson equation and in-situ data compression. We carry out initial runs of Rayleigh–Bénard Convection (RBC) at extreme scale on the LUMI and Leonardo supercomputers. We show how Neko is able to strongly scale to 16,384 GPUs and obtain results that are not pos- sible without careful consideration and optimization of the entire simulation workflow. These developments in Neko will help resolv- ing the long-standing question regarding the ultimate regime in RBC. 

Turbulence data sets produced from computational fluid dynamics (CFD), especially from fine-resolved direct numerical simulations (DNS) and large eddysimulations (LES) of turbulent flows, tend to be very large due to high resolutions adopted to accurately resolve the smallest scales. While the computational capacity of high-performance computing (HPC) platforms has kept increasing, storage capacity has lagged to the point that more data is being produced than what can be efficiently managed. Among the several methods emerged to deal with this problem, an efficient technique is data compression. In this study, we present a proof of concept of a novel data compression approach that relies on Gaussian process regression (GPR) within a Bayesian framework to handle data sets in such a way that initially discarded information can be recovered a posteriori. The approach can be used to supplement existing compression algorithms with measures of uncertainty and we show that it can be applied to compress not only the 3D spatial fields of turbulence but also the discrete sets of time series data. The compression algorithm has been designed for data from spectral element method (SEM) simulations but can be extended to spatiotemporal fields obtained from other methods arising in engineering andphysics. Our investigation shows that it is possible to use Gaussian process regression for data compression, however also highlights several of its limitations, in particular, that efficient implementations of GPR are crucial for its adoption, and that, while it is unlikely that the method can compete in terms of throughput with state of the art methods, given the cost of GPR, there is potential in termsof compression performance, as long as efficient bit-plane coding is integrated.

Proper orthogonal decomposition (POD) is a quintessential tool in the analysis of flow fields in experimental and computational fluid dynamics as a whole and turbulence in particular. It is a data driven technique that can be used to extract coherent structures in the flow and to create reduced-order dynamical systems that represent the most energetic flow structures. While extremely useful, its main drawback is that, at least traditionally, the POD algorithm is executed in serial using all the available data at once, while the original full-order system (simulation or experiment) is typically conducted and concluded before, either using a parallel simulation code or an (optical field-based) experiment. This can potentially limit its applications, in particular for large-scale data sets, e.g., from three-dimensional turbulent flows. In this article, we collect and review some of the more flexible methods for calculation of the POD (in particular, based on singular value decomposition, SVD) that have appeared over the yearsi n order to tackle these limitations. We summarize an implementation that combines the characteristics of the (distributed-memory) parallel and streaming methods and conduct detailed uncertainty quantification studies to address the effect of the hyper-parameters that typically appear with the introduction of streaming and parallel computations. We find that in general, the number of parallel processes used to compute the POD has no significant effect. On the other hand, the accuracy of the results depends on the number of updated modes and the number of snapshots treated simultaneously, with 60% and 30% of global sensitivity, respectively, in the error in the obtained POD modes. Given these findings, we formulate the recommendation that it is beneficial that more POD modes than those desired for later analysis are continuously updated. We find that the errors introduced by using streaming and parallel algorithms are however minimal in all cases. Therefore, the use of these algorithms for either computation on smaller memory systems, or online (in-situ) computation of POD modes as simulations advance is a reliable and a robust way forward to increase the rate of scientific discovery in large-scale fluid-mechanics problems.

High-order finite element methods, such as the spectral element method (SEM), are often used in scientific computing due to their low dissipation and dispersion errors. While the high-order meshes used with these methods are crucial for numerical accuracy, their disposition is generally cumbersome to perform post-processing analysis, and interpolation on to structured meshes is often necessary. In this work we demonstrate how high-order interpolation is performed on large distributed datasets in Python by using PySEMTools, a library designed to help post-process data from SEM simulations. We focus on simulations of turbulence, which necessitates fine unstructured meshes. We use these flow cases to show how different search methods and data structures perform on unstructured SEM meshes of up to 800,000 elements and provide insights into the expensive operations when interpolating meshes of up to 3 million elements. Furthermore, we show how our use of the modal representation of the data is robust. In general, we found that it is possible to have scalable parallel interpolation in Python while keeping the flexibility of scripting languages.

PySEMTools is a Python-based library for post-processing simulation data produced with high-order hexahedral elements in the context of the spectral element method in computational fluid dynamics. It aims to minimize intermediate steps typically needed when analyzing large files. Specifically, the need to use separate codebases (like the solvers themselves) at post-processing. For this effect, we leverage the use of message passing interface (MPI) for distributed computing to perform typical data processing tasks such as spectrally accurate differentiation, integration, interpolation, and reduced order modeling, among others, on a spectral element mesh. All the functionalities are provided in self-contained Python code and do not depend on the use of a particular solver. We believe that PySEMTools provides tools to researchers to accelerate scientific discovery and reduce the entry requirements for the use of advanced methodsin computational fluid dynamics.

Rayleigh–Bénard convection (RBC) is the canonical flow case in the studyof natural convection. Due to the search for numerical data approaching theso-called ultimate regime, simulations on narrow cells have been performedin the past in order to reach the Rayleigh number (Ra) range where such aphenomenon is expected to appear. While it can be argued that narrow cellsare not optimal for the study of such a regime, it is still an interesting flow caseto study given that the constrained domain introduces certain simplifications tothe flow, namely, only one large circulation structure (LCS) can appear. Thisprovides ample opportunity to understand the effect of such structure in theheat transfer and to isolate production mechanisms for turbulence. In this studywe perform direct numerical simulations of RBC in the Ra range of 109 to1014. We perform a detailed statistical analysis and uncertainty quantificationof time series of the non-dimensional heat transfer, i.e., the Nusselt number N u,and validate the results obtained by Iyer et al. (2020) using slightly differentnumerical methods and different meshes. Furthermore, we identify that thedissipation of turbulent kinetic energy in the domain is balanced mostly by thesource term associated with temperature fluctuations and do not find a strongindication that shear is one of the dominating mechanisms in this range. Furtherinvestigation on the shear dominated regions indicate that less than 50% of thein-plane cross sections present evidence of a shear layer forming. Our studiesof the dominant structures with the use of proper orthogonal decompositionalso indicate that most of the dynamics in this geometry stem from the largecirculation structure.

This study introduces vector autoregression (VAR) as a linear procedure that can be used for synthesizing turbulence time series over an entire plane, allowing them to be imposed as an efficient turbulent inflow condition in simulations requiring stationary and cross-correlated turbulence time series. VAR is a statistical tool for modelling and prediction of multivariate time series through capturing linear correlations between multiple time series. A Fourier-based proper orthogonal decomposition (POD) is performed on the two-dimensional (2-D) velocity slices from a precursor simulation of a turbulent boundary layer at a momentum thickness-based Reynolds number, Re-theta=790. A subset of the most energetic structures in space are then extracted, followed by applying a VAR model to their complex time coefficients. It is observed that VAR models constructed using time coefficients of 5 and 30 most energetic POD modes per wavenumber (corresponding to 66% and 97% of turbulent kinetic energy, respectively) are able to make accurate predictions of the evolution of the velocity field at Re-theta=790 for infinite time. Moreover, the 2-D velocity fields from the POD-VAR when used as a turbulent inflow condition, gave a short development distance when compared with other common inflow methods. Since the VAR model can produce an infinite number of velocity planes in time, this enables reaching statistical stationarity without having to run an extremely long precursor simulation or applying ad hoc methods such as periodic time series.