Building HDF5 with PGI

Dr. Joseph Schoonover

Summary

HDF5 has long been a standard for sharing scientific data across High Performance Computing (HPC) platforms. From the HDF website "HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety of datatypes, and is designed for flexible and efficient I/O and for high volume and complex data. HDF5 is portable and is extensible, allowing applications to evolve in their use of HDF5." Other data models, like NetCDF from UCAR, are built on top of HDF5 and are heavily used in oceanography, meteorology, and other earth science domains. Given this, many applications rely on having a usable working version of HDF5 that they can link into their applications.

The Portland Group Compilers are gaining popularity with the growing interest in accelerating HPC applications with Graphics Processing Units (GPUs). PGI compilers ship completely with the CUDA toolkit and a Fortran compiler (pgfortran) that support OpenACC and CUDA-Fortran, languages for writing code that operates on GPUs. This is a plus for Fortran programmers, who have a large presence in Department of Energy labs and in earth science departments around the world. Other options for GPU acceleration include FortranCL, CUDA (C/C++), OpenCL, and HIP. It is fair to say that the lack of activity on the main FortranCL repository for the last 6 years suggests the project has long been forgotten; this makes it unattractive for developers to latch on to this as a solution. The other languages are supported in C/C++ syntax and require Fortran programmers to develop C/C++ in addition to an ISO_C_BINDING layer to call their C-kernels from Fortran.

Building HDF5 with parallel and Fortran support with PGI compilers is not as straight-forward as building with other compilers, like GCC. I came across this issue while setting up clusters for hackathons, and defining build instructions for SELF-Fluids ( personal software project ). Through this experience, I have discovered that there are hints at the solution on the HDF Forums; in fact, this is where I found that a GNU C-preprocessor needs to be used in place of PGI's. This did enable a build of HDF5 with serial support, but the OpenMPI build that ships with PGI compilers cannot be used for parallel support, and instead OpenMPI was built from scratch.

This document provides details necessary for compiling HDF5 from source with the PGI compilers. HDF5 is built using an autotools build system. Template configure incantations are provided with the install notes along with expected output at the configure stage of the build. Ultimately, this was a roughly 16 hour exploration into this build issue that ultimately led to its resolution.

Install Notes

These notes cover how to install serial and parallel implementations of HDF5-1.10.2 with PGI-18.4.

Summary

In short, to build HDF5 with the PGI compilers ( v18.4 )

  1. The C-preprocessor is replaced with the GCC preprocessor
  2. At a minimum, the -fPIC flag must be added for C, C++, and Fortran compilers. Here, we have added “-fPIC -m64 -tp=px”, per Carl Ponder’s script that can be found on this forum.
  3. When building in parallel, the precompiled OpenMPI libraries shipped with PGI community edition compilers cannot be used. To work around this, we’ve opted to compile OpenMPI (v3.1.1) from scratch.


Below, I’ll provide the arguments that need to be passed at the configure stage of the serial and parallel builds. For each build, I’ll provide the summary output that was obtained at the end of the configure stage.

Prerequisites

HDF5 depends on zlib, and parallel builds depend on some flavor of MPI.


Serial Build

$ CPP=cpp CFLAGS="-fPIC -m64 -tp=px" CXXFLAGS="-fPIC -m64 -tp=px" FCFLAGS="-fPIC -m64 -tp=px" CC=pgcc CXX=pgc++ FC=pgfortran ./configure --with-zlib=<path/to/zlib> --enable-threadsafe --enable-cxx --enable-fortran --enable-unsupported --prefix=<path/to/hdf5>


$ make

$ make install


Below shows the output of the configure stage when invoking a similar incantation to that shown above.

SUMMARY OF THE HDF5 CONFIGURATION

=================================


Compiling Options:

------------------

Build Mode: production

Debugging Symbols: no

Asserts: no

Profiling: no

Optimization Level: high


Linking Options:

----------------

Libraries: static, shared

Statically Linked Executables:

LDFLAGS:

H5_LDFLAGS:

AM_LDFLAGS: -L/apps/spack/opt/spack/linux-ubuntu16.04-x86_64/pgi-18.4/zlib-1.2.11-yy6235b5kltm4qta7wnzvtvm4mol4nfc//lib

Extra libraries: -lpthread -lz -ldl -lm

Archiver: ar

AR_FLAGS: cr

Ranlib: ranlib


Languages:

----------

C: yes

C Compiler: /apps/pgi-18.4/linux86-64/18.4/bin/pgcc ( pgcc 18.4-0 64-bit target on x86-64 Linux -tp px )

CPPFLAGS:

H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200112L -DNDEBUG -UH5_DEBUG_API

AM_CPPFLAGS: -I/apps/spack/opt/spack/linux-ubuntu16.04-x86_64/pgi-18.4/zlib-1.2.11-yy6235b5kltm4qta7wnzvtvm4mol4nfc//include

C Flags: -fPIC -m64 -tp=px

H5 C Flags: -c99 -Minform=inform -fast -s

AM C Flags:

Shared C Library: yes

Static C Library: yes


Fortran: yes

Fortran Compiler: /apps/pgi-18.4/linux86-64/18.4/bin/pgf90 ( pgf90 18.4-0 64-bit target on x86-64 Linux -tp px )

Fortran Flags: -fPIC -m64 -tp=px

H5 Fortran Flags: -fast -Mnoframe -s

AM Fortran Flags:

Shared Fortran Library: yes

Static Fortran Library: yes


C++: yes

C++ Compiler: /apps/pgi-18.4/linux86-64/18.4/bin/pgc++

C++ Flags: -fPIC -m64 -tp=px

H5 C++ Flags:

AM C++ Flags:

Shared C++ Library: yes

Static C++ Library: yes


Java: no


Features:

---------

Parallel HDF5: no

High-level library: yes

Threadsafety: yes

Default API mapping: v110

With deprecated public symbols: yes

I/O filters (external): deflate(zlib)

MPE: no

Direct VFD: no

dmalloc: no

Packages w/ extra debug output: none

API tracing: no

Using memory checker: no

Memory allocation sanity checks: no

Metadata trace file: no

Function stack tracing: no

Strict file format checks: no

Optimization instrumentation: no


Parallel Build


For the parallel build, you’ll first want to build OpenMPI with PGI. To do so, with CUDA enabled, you can do something similar to the following

Build MPI with CUDA support

$ CPP=cpp ./configure --with-cuda=<path-to-pgi-install>/linux86-64/2018/cuda/9.0 --prefix=<path-to-mpi-install>


$ make

$ make install

Note that PGI-18.4 ships with a CUDA-9.0 toolkit and you can use that installation to provide CUDA libraries.


The config summary is

$ Open MPI configuration:

-----------------------

Version: 3.1.1

Build MPI C bindings: yes

Build MPI C++ bindings (deprecated): no

Build MPI Fortran bindings: mpif.h, use mpi, use mpi_f08

MPI Build Java bindings (experimental): no

Build Open SHMEM support: yes

Debug build: no

Platform file: (none)


Miscellaneous

-----------------------

CUDA support: yes

Transports

-----------------------

Cisco usNIC: no

Cray uGNI (Gemini/Aries): no

Intel Omnipath (PSM2): no

Intel SCIF: no

Intel TrueScale (PSM): no

Mellanox MXM: no

Open UCX: no

OpenFabrics Libfabric: no

OpenFabrics Verbs: no

Portals4: no

Shared memory/copy in+copy out: yes

Shared memory/Linux CMA: yes

Shared memory/Linux KNEM: no

Shared memory/XPMEM: no

TCP: yes

Resource Managers

-----------------------

Cray Alps: no

Grid Engine: no

LSF: no

Moab: no

Slurm: yes

ssh/rsh: yes

Torque: no


Build HDF5 with parallel support

Make sure that the previous build of MPI is in your path. Additionally, be sure to fill in the path to your zlib installation (--with-zlib) and where you want to install HDF5 (--prefix)

$ CPP=cpp CFLAGS="-fPIC -m64 -tp=px" CXXFLAGS="-fPIC -m64 -tp=px" FCFLAGS="-fPIC -m64 -tp=px" CC=mpicc CXX=mpic++ FC=mpif90 ./configure --with-zlib=<path-to-zlib> --enable-threadsafe --enable-cxx --enable-fortran --enable-unsupported --enable-parallel --prefix=<path-to-hdf5-install>


$ make

$ make install


Below is the configuration summary for our build.

SUMMARY OF THE HDF5 CONFIGURATION

=================================


Compiling Options:

------------------

Build Mode: production

Debugging Symbols: no

Asserts: no

Profiling: no

Optimization Level: high


Linking Options:

----------------

Libraries: static, shared

Statically Linked Executables:

LDFLAGS:

H5_LDFLAGS:

AM_LDFLAGS: -L/apps/spack/opt/spack/linux-ubuntu16.04-x86_64/pgi-18.4/zlib-1.2.11-yy6235b5kltm4qta7wnzvtvm4mol4nfc//lib

Extra libraries: -lpthread -lz -ldl -lm

Archiver: ar

AR_FLAGS: cr

Ranlib: ranlib


Languages:

----------

C: yes

C Compiler: /apps/openmpi/3.1.1/pgi-18.4/bin/mpicc

CPPFLAGS:

H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200112L -DNDEBUG -UH5_DEBUG_API

AM_CPPFLAGS: -I/apps/spack/opt/spack/linux-ubuntu16.04-x86_64/pgi-18.4/zlib-1.2.11-yy6235b5kltm4qta7wnzvtvm4mol4nfc//include

C Flags: -fPIC -m64 -tp=px

H5 C Flags: -c99 -Minform=inform -fast -s

AM C Flags:

Shared C Library: yes

Static C Library: yes



Fortran: yes

Fortran Compiler: /apps/openmpi/3.1.1/pgi-18.4/bin/mpif90

Fortran Flags: -fPIC -m64 -tp=px

H5 Fortran Flags:

AM Fortran Flags:

Shared Fortran Library: yes

Static Fortran Library: yes


C++: yes

C++ Compiler: /apps/openmpi/3.1.1/pgi-18.4/bin/mpic++

C++ Flags: -fPIC -m64 -tp=px

H5 C++ Flags:

AM C++ Flags: -DOLD_HEADER_FILENAME -DHDF_NO_NAMESPACE -DNO_STATIC_CAST

Shared C++ Library: yes

Static C++ Library: yes


Java: no



Features:

---------

Parallel HDF5: yes

High-level library: yes

Threadsafety: yes

Default API mapping: v110

With deprecated public symbols: yes

I/O filters (external): deflate(zlib)

MPE:

Direct VFD: no

dmalloc: no

Packages w/ extra debug output: none

API tracing: no

Using memory checker: no

Memory allocation sanity checks: no

Metadata trace file: no

Function stack tracing: no

Strict file format checks: no

Optimization instrumentation: no