SELF-Fluids

About SELF

SELF is an object-oriented Fortran library that support the implementation of Spectral Element Methods for solving partial differential equations.

The SELF API is designed based on the assumption that SEM developers and researchers need to be able to implement derivatives in 1-D and divergence, gradient, and curl in 2-D and 3-D on scalar, vector, and tensor functions using spectral collocation, continuous galerkin, and discontinuous galerkin spectral element methods. Additionally, as we enter the exascale era, we are currently faced with a zoo of compute hardware that is available. Because of this, SELF routines provide support for GPU acceleration through AMD's HIP and support for multi-core, multi-node, and multi-GPU platforms with MPI.

SELF-Fluids

The "Fluids" component of "SELF-Fluids" pieces together the SELF for solving the compressible Navier-Stokes using a Discontinuous Galerkin Spectral Element Method for the spatial discretization. For the time integration, we use Williamson's 3rd Order Low Storage Runge-Kutta. We are currently working on transitioning SELF to support multi-core, multi-node, GPU, and multi-GPU platforms, including AMD and Nvidia GPUs. Once this work is complete (expected July 2021), SELF-Fluids will be transitioned and a new round of validation experiments will begin.

The end result of SELF-Fluids is a fluids software that can be compiled for a variety of platforms and with a variety of compilers.

The video on the right shows a well studied setup, known as the "dam break" or "lock exchange". Cold and hot fluids are placed side-by-side and left to evolve. Gravity causes the cold, heavy fluid to sink and subduct underneath the hot, light fluid. Sufficient gradients in velocity result in Kelvin-Helmholtz instability and a turbulent forward cascade of energy ensues, ultimately mixing the fluid.

We have been setting up other well-studied scenarios to develop an understanding of the code's performance, scalability, and accuracy for a range of fluid applications. The breadth of applications include blood-flow, aerodynamics, shock tubes, planetary boundary layers, and topographically driven flows.

Singularity Containerization

Within the development of the SELF-Fluids code, we uncovered a requirement to provide a containerized product that deploys and interfaces with resources with programmatic behavior. As we developed our internal cloud compute resources we have realized a need to enable orchestration management systems to inter-operate and scale a code base at will, on optimized resources, in order to leverage the capabilities of the system entirely. To streamline this process internally with SELF-Fluids, we set out to decouple, distribute and repackage the application in a way that would integrate well with burst scalable resources. Some of the most popular options that we considered were Docker and Singularity, which we ultimately use in conjunction in order to provide a seamless serverless deployment that is ready to scale per our application's needs rather than a generic resource assignment.

This progression in the structure of our code enables us to empower a Platform as a Service to operate a Software and provision Data as the end product to potential customers or end users. It allows an interested user base to basically demonstrate proof of concept without the complexity of administrating, managing, and executing a project internally. This use case could potentially find a successful result without leaving their on-premise workstation.

Roadmap

The roadmap below shows the features we plan on implementing with a timeline for the expected completion.

February 2021

  • New SELF API Specification

    • Routines defined by the cross product of
      [ interpolation, differentiation, divergence, gradient, curl ] X [scalar, vector, tensor] X [1D, 2D,3D] X [Strong Form, DG Weak Form, CG Weak Form ] x [Serial, Single GPU (HIP), MPI-only, MPI+HIP (multi-gpu) ]

  • All routines for Serial CPU and Single GPU implemented

April 2021

  • MPI-only support for all routines

  • HOPr file IO support

  • VTK file IO support

  • SpecMesh file IO support

June 2021

  • MPI+HIP support for all routines

  • OBJ file IO support

  • CGNS (with higher order extension) file IO support

August 2021

  • Nektar++ file IO support

  • Example Applications : SELF-Fluids (Compressible Navier-Stokes)

  • Board of Directors formation campaign

    • Formally establish board of direction & operating policies

    • Formalize Specifications & Contributing guidelines