recursive_diff: Compare two Python data structures

JSON, JSONL, YAML and MessagePack are massively popular formats used to represent nested data. A problem arises when you want to compare two large JSON-like data structures, because the == operator will tell you if the two structures differ somewhere, but won’t tell you where. Additionally, if the structures contain floating-point numbers, == won’t allow to set a tolerance: 1.00000000000001 is different from 1.0, which is majorly problematic as floating point arithmetic is naturally characterised by noise around the 15th decimal position (the size of the double-precision mantissa). Tests on floating point numbers are typically performed with math.isclose() or numpy.isclose(), which however are not usable if the numbers to be tested lie deep inside a nested structure.

A second problem that data scientists need to face routinely is comparing huge NumPy-based data structures, such as pandas.DataFrame objects or data loaded from Zarr, netCDF, or HDF5 datastores. Again, it is very frequently needed to identify where differences are, and apply tolerance to the comparison.

This module offers the function recursive_diff(), which crawls through two arbitrarily large nested JSON-like structures and dumps out all the differences. Python-specific data types, such as set and tuple, are also supported. NumPy, Pandas, and Xarray are supported and optimized for speed. Two variant functions, display_diffs() and recursive_eq(), are designed to be used in Jupyter Notebooks and unit tests respectively.

You can load a whole directory tree of JSON, JSONL, YAML, MessagePack, netCDF, or Zarr files with one call to recursive_open() and then pass it to recursive_diff() or recursive_eq() to compare it to another directory tree.

Finally, a CLI tool allows comparing two files in any of the formats above, or two directory trees full of files, as long as they can be loaded with xarray.open_dataset().

Index

• Quick Start
• Installation
  • Required dependencies
  • Optional dependencies
  • Installing with conda
  • Installing with pip
  • Minimum dependency versions
• Working with Jupyter notebooks
  • Comparing directories
• Dask integration
  • Dask clusters
• Extending recursive_diff/recursive_eq
• API Reference
  • recursive_diff()
  • recursive_eq()
  • diff_arrays()
  • display_diffs()
  • cast()
  • open()
  • recursive_open()
• CLI tool
  • Usage
  • Memory design
  • Limitations
• Development Guidelines
  • Reporting issues
  • Deploying a development environment
  • Test
  • Code Formatting
  • Documentation
• What’s New
  • v2.2.0 (unreleased)
  • v2.1.0 (2026-03-12)
  • v2.0.0 (2026-02-22)
    • CLI changes
    • Breaking changes
  • v1.3.0 (2025-10-14)
  • v1.2.0 (2024-03-16)
  • v1.1.0 (2022-03-26)
  • v1.0.0 (2019-01-02)

Credits

• recursive_diff, recursive_eq and ncdiff were originally developed by Legal & General and released to the open source community in 2018.

• All boilerplate is from python_project_template, which in turn is from Xarray.

License

This software is available under the open source Apache License.