pytables – Make Me Engineer

Is there an analysis speed or memory usage advantage to using HDF5 for large array storage (instead of flat binary files)?

June 7, 2023 by Tarik

HDF5 Advantages: Organization, flexibility, interoperability Some of the main advantages of HDF5 are its hierarchical structure (similar to folders/files), optional arbitrary metadata stored with each item, and its flexibility (e.g. compression). This organizational structure and metadata storage may sound trivial, but it’s very useful in practice. Another advantage of HDF is that the datasets can … Read more

How to get faster code than numpy.dot for matrix multiplication?

May 31, 2023 by Tarik

np.dot dispatches to BLAS when NumPy has been compiled to use BLAS, a BLAS implementation is available at run-time, your data has one of the dtypes float32, float64, complex32 or complex64, and the data is suitably aligned in memory. Otherwise, it defaults to using its own, slow, matrix multiplication routine. Checking your BLAS linkage is … Read more

Storing numpy sparse matrix in HDF5 (PyTables)

May 30, 2023 by Tarik

The answer by DaveP is almost right… but can cause problems for very sparse matrices: if the last column(s) or row(s) are empty, they are dropped. So to be sure that everything works, the “shape” attribute must be stored too. This is the code I regularly use: import tables as tb from numpy import array … Read more

How can I combine multiple .h5 file?

July 23, 2022 by Tarik

These examples show how to use h5py to copy datasets between 2 HDF5 files. See my other answer for PyTables examples. I created some simple HDF5 files to mimic CSV type data (all floats, but the process is the same if you have mixed data types). Based on your description, each file only has one … Read more