Chapter 12 - MODIS HDF4 Remote Sensing Data in Python
In this chapter, you will learn how to work with MODIS remote sensing data stored in the HDF4 file format using open source Python.
After completing this chapter, you will be able to:
- Explain how data are structured and stored in HDF4 files.
- Explore HDF4 files (and their metadata) in a free HDF viewer.
- Open and process HDF4 files in Python.
What You Need
You will need a computer with internet access to complete this lesson and the Cold Springs MODIS data, which you can download using earthpy:
You will also need to download a free HDF viewer from the HDF Group website. Additional download and installation information are provided on this page.
Be sure to download the data above, so that you can access the data in the free HDF viewer.
What are HDF4 Files?
On the landing page of this section, you learned about the general characteristics of Hierarchical Data Format (HDF) files and that there are many types of HDF files including HDF4, HDF5, and NetCDF.
HDF files are open source file formats that support large, complex, heterogeneous data, using a “file directory” like structure. HDF formats also allow for embedding of metadata, making them self-describing.
HDF4 is an older hierarchical data format as compared to HDF5, which is the latest version promoted by the HDF Group, the publisher of the libraries and standards for these formats.
While the transition to HDF5 has occurred for many remote sensing products, HDF4 is still the primary data format that is adapted for MODIS data products published by NASA.
Data Tip: HDF4 is an older hierarchical data format. Most MODIS data are still delivered in an adapted version of this format. HDF5 (and NetCDF which is similar to the HDF5 format with different associated standards) are the preferred data structures to use for new data products.
HDF4 EOS Format for MODIS
Both the HDF4 and HDF5 formats have been adapted by NASA to publish data from Earth Observing System (EOS) missions. These adapted formats, referred to as HDF-EOS, contain additional geolocated data types (point, grid, swath) that can be used to store spatial information that are not supported within the original HDF structure.
HDF4-EOS, the format adapted from HDF4, is the currently used format for MODIS data products. You can review the HDF4-EOS User Guide (section 3.1) to learn more about how the HDF4 format has been adapted to support these additional spatial data types.
HDF4-EOS files are composed of a directory containing data objects (what we might think of as individual files in a computer directory). Each data object is listed as an individual entry in the directory, which allows the data object to be linked with related metadata.
Related data objects can be grouped into datasets consisting of multiple data objects (what we might think of subdirectories to organize files within a computer directory).
Explore HDF4 Files Using HDFView
To familiarize yourself with the HDF4 structure, and explore a particular file’s data objects, you can use the free HDFView tool published by the HDF group.
The sections below walk you through downloading and installing the tool as well as exploring an HF4-EOS file included in the dataset that you downloaded at the top of this page (see: section on What You Need).
Download and Install HDF Viewer
In order to download the free HDF viewer from the HDF Group website, you will need to first create a free account.
You can create a free account by clicking on the “Create Free Account” button on the top-right corner of the download page.
During the process to create a free account, you will be asked to confirm your email address by entering a code that is emailed to the address that you provided.
Once you have finished creating an account, you can select the appropriate installer for your operating system from the download page.
Windows Users: you will need to select the appropriate installer based on your version of Windows. For example,
HDFView-3.1.0-win10vs14_64.zip is the appropriate installer for Windows version 10.0.14. After downloading the .zip file, you can extract the file and double-click on the .msi file to run the installation.
Open HDF4 Files in HDF Viewer
Once installed, open the HDFView tool on your computer.
In the menu bar, click the first button
Open to open a file.
Navigate to directory for the data download (see: section on What You Need).
Recall that earthpy datasets are downloaded to the
data directory under
cold-springs-modis-h5 directory, you will see two directories containing
.hdf files (e.g.
Select and open
MOD09GA.A2016189.h09v05.006.2016191073856.hdf within the
Explore HDF4 Data Objects in HDF Viewer
View HDF File Structure
Once opened, the HDF file structure will appear in the left panel of the HDFView interface.
Click on the
> next to the dataset name for
MODIS_Grid_500m_2D to expand it.
Data Fields to see the data objects that are stored in that dataset.
From the list of surface reflectance bands, click (or select)
sur_refl_b01_1 for band 1.
Notice that when you select an individual data object, you see the associated metadata in the
Object Attribute Info tab, such as the fill value of
-28672 and valid range of
View Image of Surface Reflectance Band
Click on the second tab titled
General Object Info.
Notice that the path to the file is displayed as well as other attributes such as the dimensions.
To preview the image, click on the button for
Show Data with Options (bottom center).
Display As option, select the radio button for
On the right hand side of the window, enter values into the box for Valid Range:
Last, click on the
OK button (bottom center) to open the image.
You now know how to open and explore HDF4 files using the free HDFView tool.
To explore additional functionality in the HDF viewer such as converting and exporting files, review the HDF Viewer User Guide.