I/O Module (stx.io)

Unified file I/O for 30+ scientific data formats. A single save() and load() interface handles format detection by file extension, so you never need to remember which library reads .npy vs .h5 vs .parquet.

Note

Core format handlers are provided by the standalone scitex-io package. stx.io adds integration with stx.session (provenance tracking), stx.clew (claim verification), and automatic CSV export for figures.

Supported Formats

Category	Extensions
Tabular	.csv, .tsv, .xlsx, .xls, .parquet, .feather
Array	.npy, .npz, .mat, .zarr, .h5 / .hdf5
Config	.yaml, .yml, .json, .toml, .ini, .conf
Figure	.png, .jpg, .jpeg, .svg, .pdf, .tiff
Text	.txt, .log, .md, .rst
Audio	.wav, .mp3, .flac
Serialized	.pkl, .pickle, .joblib
Bibliography	.bib

Quick Start

import scitex as stx
import pandas as pd
import numpy as np

# --- Save and load a DataFrame ---
df = pd.DataFrame({"x": [1, 2, 3], "y": [4, 5, 6]})
stx.io.save(df, "results.csv")
df_loaded = stx.io.load("results.csv")

# --- Save and load a NumPy array ---
arr = np.random.randn(100, 3)
stx.io.save(arr, "data.npy")
arr_loaded = stx.io.load("data.npy")

# --- Save a figure (PNG + CSV data export) ---
fig, ax = stx.plt.subplots()
ax.stx_line([1, 2, 3, 4, 5], id="my_data")
stx.io.save(fig, "plot.png")
# Creates: plot.png, plot.csv, plot.yaml

Key Functions

save(obj, path, **kwargs)

Save any supported object to a file. The format is determined by the file extension. Built-in features beyond simple save:

• Auto directory creation – no os.makedirs() needed

• Path resolution – relative paths resolve to <script_name>_out/<path>

• Symlinks – symlink_from_cwd=True for short access paths

• Save logging – prints file path and size on success

• Clew hash tracking – file hashes recorded automatically for verification

• Figure CSV export – saves plot data alongside image files

# Format detected from extension
stx.io.save(fig, "figure.pdf")       # Also exports CSV + YAML recipe
stx.io.save(df, "output.parquet")    # DataFrame
stx.io.save({"lr": 0.001}, "config.yaml")  # Dict
stx.io.save(arr, "weights.npy")      # NumPy array

# Symlink for convenient access
stx.io.save(df, "results/data.csv", symlink_from_cwd=True)

load(path)

Load data from a file. Returns the appropriate Python object (DataFrame, ndarray, dict, etc.).

df = stx.io.load("results.csv")          # -> pd.DataFrame
arr = stx.io.load("weights.npy")         # -> np.ndarray
cfg = stx.io.load("config.yaml")         # -> dict
data = stx.io.load("experiment.h5")      # -> dict of arrays

load_configs(pattern="./config/*.yaml")

Load and merge multiple YAML configuration files into a single DotDict. Used internally by @stx.session to build the CONFIG object.

CONF = stx.io.load_configs("./config/*.yaml")
print(CONF.MODEL.hidden_size)

list_formats()

List all registered format handlers.

fmts = stx.io.list_formats()
# ['.csv', '.h5', '.hdf5', '.json', '.mat', '.npy', ...]

HDF5 and Zarr Exploration

For hierarchical formats, use the explorer interface:

# HDF5
stx.io.explore_h5("data.h5")            # Print tree structure
stx.io.has_h5_key("data.h5", "/group/dataset")

# Zarr
stx.io.explore_zarr("data.zarr")
stx.io.has_zarr_key("data.zarr", "/group/array")

Format Registry

Register custom loaders and savers for new file extensions:

@stx.io.register_loader(".custom")
def load_custom(path, **kwargs):
    with open(path) as f:
        return parse_custom(f.read())

@stx.io.register_saver(".custom")
def save_custom(obj, path, **kwargs):
    with open(path, "w") as f:
        f.write(serialize_custom(obj))

Caching

Built-in load caching for repeated reads of the same file:

stx.io.configure_cache(maxsize=128)
data = stx.io.load("big_file.h5")    # First call: reads from disk
data = stx.io.load("big_file.h5")    # Second call: from cache

stx.io.get_cache_info()               # Cache hit/miss statistics
stx.io.clear_load_cache()             # Flush the cache

API Reference

scitex-io — Universal scientific data I/O with plugin registry.

Functionalities

• save(obj, “path.ext”) / load(“path.ext”) — extension-dispatched one-call I/O for 30+ formats (CSV, Parquet, Feather, NumPy, pickle, YAML, JSON, HDF5, Zarr, MATLAB, images, matplotlib figures, PyTorch, MNE, EDF, video).

• register_saver(“.ext”) / register_loader(“.ext”) — plugin hooks for user-defined formats; dispatch lookup follows the same registry.

• load_configs() — collect every <project-root>/config/*.yaml into a single DotDict with UPPER_CASE normalisation + DEBUG_ overrides.

• glob / parse_glob — natural-sorted globbing with {placeholder} parsing; cache / reload / flush — load-cache management.

IO

• Reads: any registered extension; ./config/*.yaml; $SCITEX_DIR cache; figure metadata (PNG tEXt, JPEG EXIF, SVG XML, PDF XMP).

• Writes: relative paths resolve under {caller}_out/ (script / notebook) or $SCITEX_DIR/io/runtime/cache/ (REPL); absolute paths pass through unchanged.

Dependencies

• Hard: tqdm, PyYAML, ruamel.yaml, mne, numpy, pandas, click, rich, natsort, scitex-dev, scitex-logging.

• Optional ([scientific]): scipy, h5py, zarr>=3, numcodecs, matplotlib. ([mcp]): fastmcp.

Register custom handlers:

from scitex_io import register_saver, register_loader

@register_saver(".myformat")
def save_myformat(obj, path, **kw): ...

@register_loader(".myformat")
def load_myformat(path, **kw): ...

Top-level imports are PEP 562 lazy — import scitex_io is cheap. Public symbols load on first attribute access. See _skills/general/03_interface_01_python-api/04_lazy-imports-and-optional-deps.md.

scitex.io.register_saver(ext, fn=None, *, builtin=False)

Register a save handler for a file extension.

Can be used as a decorator or called directly:

@register_saver(".json")
def my_json_saver(obj, path, **kwargs): ...

register_saver(".json", my_json_saver)

Parameters:

• ext (str) – File extension (e.g., “.json”, “json” — dot is optional).

• fn (Callable, optional) – Handler function (obj, path, **kwargs) -> None. If None, returns a decorator.

• builtin (bool) – If True, registers as built-in (lower priority). User registrations always override built-ins.

scitex.io.register_loader(ext, fn=None, *, builtin=False)

Register a load handler for a file extension.

Same API as register_saver().

Parameters:

• ext (str) – File extension (e.g., “.json”, “json” — dot is optional).

• fn (Callable, optional) – Handler function (path, **kwargs) -> Any.

• builtin (bool) – If True, registers as built-in (lower priority).

scitex.io.get_saver(ext)

Look up a save handler. User overrides take priority.

Lazy builtin specs ((module_path, attr_name) tuples) are resolved on first access and memoised in place.

Return type:

Optional[Callable]

scitex.io.get_loader(ext)

Look up a load handler. User overrides take priority.

Lazy builtin specs ((module_path, attr_name) tuples) are resolved on first access and memoised in place.

Return type:

Optional[Callable]

scitex.io.list_formats()

List all registered formats.

Returns:

A dict with keys "save" and "load", each containing "builtin" and "user" format lists.

Return type:

dict

Notes

Builtin entries are listed regardless of whether they have been lazy-resolved yet — registration is what counts.

scitex.io.unregister_saver(ext)

Remove a user-registered saver. Returns True if found.

Return type:

bool

scitex.io.unregister_loader(ext)

Remove a user-registered loader. Returns True if found.

Return type:

bool

scitex.io.save(obj, specified_path, makedirs=True, verbose=True, symlink_from_cwd=False, symlink_to=None, dry_run=False, no_csv=False, use_caller_path=False, env_detector=None, **kwargs)

Save obj by extension; specified_path is caller-anchored.

The file format is selected from specified_path’s extension via the plugin registry — .csv, .npy, .pkl, .yaml, .png, .h5, … 30+ formats are built in; custom extensions can be added with register_saver.

Path resolution rules (when specified_path is relative):

• Called from a script /path/to/analysis.py → /path/to/analysis_out/<specified_path>.

• Called from a notebook /path/to/exp.ipynb → /path/to/exp_out/<specified_path>.

• Called from python -i / IPython / interactive REPL → $SCITEX_DIR/io/runtime/cache/<specified_path> (default ~/.scitex/io/runtime/cache/). Honours the canonical scitex local-state convention; see scitex-dev skills/general 01_ecosystem_06_local-state-directories.md.

• Absolute path → used as-is, no routing.

Intermediate directories are created automatically — callers do not need os.makedirs() / Path.mkdir().

Parameters:

• obj (Any) – The object to be saved.

• specified_path (Union[str, Path]) – The filename or relative path under which to save obj. May contain subdirectories ("sub/dir/file.csv"); intermediates are auto-created. Absolute paths bypass routing.

• makedirs (bool, optional) – Create parent directories on demand. Default True.

• verbose (bool, optional) – Print a one-line success message. Default True.

• symlink_from_cwd (bool, optional) – Drop a symlink at ./<specified_path> pointing into the auto-routed location. Default False.

• symlink_to (Union[str, Path], optional) – Plant a symlink at this custom path pointing to the saved file.

• dry_run (bool, optional) – Print the resolved path without writing. Default False.

• no_csv (bool, optional) – Skip the auto-CSV sidecar for figure saves. Default False.

• use_caller_path (bool, optional) – Resolve the anchor from the calling script, not the immediate caller — needed when save is wrapped by a library. Default False.

• **kwargs – Passed through to the per-format handler.

Returns:

Path to saved file on success, None/False on error.

Return type:

Path or None

scitex.io.load(lpath, ext=None, show=False, verbose=False, cache=True, **kwargs)

Public wrapper around _load_impl() that fires post-load hooks.

Glob expansion is handled inside _load_impl; the inner per-file recursion already routes through load so each match fires its own hook. Only the outer non-glob path fires once here.

Return type:

Any

scitex.io.load_configs(IS_DEBUG=None, show=False, verbose=False, config_dir=None)

Load and merge every YAML under config_dir into one DotDict.

Filename stems become top-level keys; YAML keys become nested attributes. Every string key (filename stem and every nested key) is normalised to UPPER_CASE at load time so the in-memory tree is case-stable regardless of source casing — model.yaml with hidden_dim: 256 lands at CONFIG.MODEL.HIDDEN_DIM. Lookups on the returned DotDict are case-insensitive for string keys, so CONFIG.SEIZURE.STR2COLOR["seizure"] resolves the stored "SEIZURE" entry — no surprise KeyError for the lowercase key the author wrote (non-string keys are matched exactly).

If two keys inside one mapping fold to the same UPPER form (e.g. MODEL.yaml next to model.yaml, or HIDDEN_DIM next to hidden_dim, or "seizure" next to "SEIZURE" in one string-mapping), a loud ValueError is raised at load time naming the source file, the mapping path, and both offending keys. The collision is never silently merged or dropped.

Debug mode promotes any DEBUG_<KEY> sibling over its non-debug counterpart, so a single IS_DEBUG.yaml flips the whole project between production and debug values. Equivalent triggers: IS_DEBUG.yaml with IS_DEBUG: true, the IS_DEBUG=True kwarg, or running under CI=True.

Parameters:

• IS_DEBUG (bool, optional) – Force debug mode. If None (default), inferred from IS_DEBUG.yaml inside config_dir or from the CI env var.

• show (bool) – Echo the DEBUG_<KEY> -> <KEY> substitutions to stdout.

• verbose (bool) – Print detailed information.

• config_dir (Union[str, Path], optional) – Directory containing the YAML files. Defaults to "./config".

Returns:

Merged configuration tree with UPPER_CASE keys throughout.

Return type:

DotDict

Raises:

ValueError – If two keys inside one mapping fold to the same UPPER form (a case collision). Raised at load time, naming the file, the mapping path, and both offending keys.

Examples

>>> CONFIG = load_configs()                       # ./config/*.yaml
>>> CONFIG.MODEL.HIDDEN_DIM                       # 256
>>> CONFIG = load_configs(IS_DEBUG=True)
>>> CONFIG.MODEL.HIDDEN_DIM                       # 32 (DEBUG_ promoted)

scitex.io.glob(expression, parse=False, ensure_one=False)

Perform a glob operation with natural sorting and extended pattern support.

This function extends the standard glob functionality by adding natural sorting and support for curly brace expansion in the glob pattern.

Parameters:

• expression (str or Path) – The glob pattern. Supports standard glob and {a,b} expansion.

• parse (bool, optional) – Whether to parse the matched paths. Default is False.

• ensure_one (bool, optional) – Ensure exactly one match is found. Default is False.

Returns:

If parse=False: naturally sorted file paths. If parse=True: tuple of (paths, parsed_results).

Return type:

list or tuple

Examples

>>> glob('data/*.txt')
['data/file1.txt', 'data/file2.txt', 'data/file10.txt']

>>> glob('data/{a,b}/*.txt')
['data/a/file1.txt', 'data/a/file2.txt', 'data/b/file1.txt']

>>> paths, parsed = glob('data/subj_{id}/run_{run}.txt', parse=True)
>>> paths
['data/subj_001/run_01.txt', 'data/subj_001/run_02.txt']
>>> parsed
[{'id': '001', 'run': '01'}, {'id': '001', 'run': '02'}]

>>> paths, parsed = glob('data/subj_{id}/run_{run}.txt', parse=True, ensure_one=True)
AssertionError  # if more than one file matches

scitex.io.parse_glob(expression, ensure_one=False)

Convenience function for glob with parsing enabled.

Parameters:

• expression (str or Path) – The glob pattern.

• ensure_one (bool, optional) – Ensure exactly one match is found. Default is False.

Returns:

Matched paths and parsed results.

Return type:

tuple

Examples

>>> paths, parsed = pglob('data/subj_{id}/run_{run}.txt')
>>> paths
['data/subj_001/run_01.txt', 'data/subj_001/run_02.txt']
>>> parsed
[{'id': '001', 'run': '01'}, {'id': '001', 'run': '02'}]

>>> paths, parsed = pglob('data/subj_{id}/run_{run}.txt', ensure_one=True)
AssertionError  # if more than one file matches

scitex.io.reload(module_or_func, verbose=False)

Reload a module or the module containing a given function.

This function attempts to reload a module directly if a module is passed, or reloads the module containing the function if a function is passed. This is useful during development to reflect changes without restarting the Python interpreter.

Parameters:

module_or_funcmodule or function

The module to reload, or a function whose containing module should be reloaded.

verbosebool, optional

If True, print additional information during the reload process. Default is False.

Returns:

: None

Raises:

Exception

If the module cannot be found or if there’s an error during the reload process.

Notes:

• Reloading modules can have unexpected side effects, especially for modules that maintain state or have complex imports. Use with caution.

• This function modifies sys.modules, which affects the global state of the Python interpreter.

Examples:

>>> import my_module
>>> reload(my_module)

>>> from my_module import my_function
>>> reload(my_function)

scitex.io.flush(sys=<module 'sys' (built-in)>, *, sync_fn=None)

Flushes the system’s stdout and stderr, and syncs the file system. This ensures all pending write operations are completed.

sync_fn lets callers (typically tests) substitute os.sync() with a no-op or recording callable. Defaults to os.sync().

scitex.io.cache(id, *args, cache_root=None)

Store or fetch data using a pickle file.

This function provides a simple caching mechanism for storing and retrieving Python objects. It uses pickle to serialize the data and stores it in a file with a unique identifier. If the data is already cached, it can be retrieved without recomputation.

Parameters:

• id (str) – A unique identifier for the cache file.

• *args (str) – Variable names to be cached or loaded.

• cache_root (Path or None, optional) – Explicit cache directory. Defaults to $SCITEX_DIR/io/runtime/cache/ (~/.scitex/io/runtime/cache/ fallback), honouring the canonical scitex local-state convention.

Returns:

A tuple of cached values corresponding to the input variable names.

Return type:

tuple

Raises:

ValueError – If the cache file is not found and not all variables are defined.

Example

>>> import scitex
>>> import numpy as np
>>>
>>> # Variables to cache
>>> var1 = "x"
>>> var2 = 1
>>> var3 = np.ones(10)
>>>
>>> # Saving
>>> var1, var2, var3 = scitex.io.cache("my_id", "var1", "var2", "var3")
>>> print(var1, var2, var3)
>>>
>>> # Loading when not all variables are defined and the id exists
>>> del var1, var2, var3
>>> var1, var2, var3 = scitex.io.cache("my_id", "var1", "var2", "var3")
>>> print(var1, var2, var3)

scitex.io.configure_cache(enabled=None, max_size=None, verbose=None)

Configure cache settings.

Parameters:

• enabled (Optional[bool]) – Enable or disable caching

• max_size (Optional[int]) – Maximum number of files to cache

• verbose (Optional[bool]) – Enable verbose logging

Return type:

None

scitex.io.get_cache_info()

Get cache statistics and configuration.

Returns:

Cache information including stats and config

Return type:

Dict[str, Any]

scitex.io.clear_load_cache()

Clear all cached data.

Return type:

None

class scitex.io.DotDict(dictionary=None)

Bases: object

A dictionary-like object that allows attribute-like access (for valid identifier keys) and standard item access for all keys (including integers, etc.).

Case-insensitive on string-key lookup, storage-stable

Keys are stored exactly as set (load_configs separately normalises every config key to UPPER on load). Lookups, however, are case-insensitive for string keys: d["seizure"], d["SEIZURE"], d.seizure and d.SEIZURE all resolve to the same stored value regardless of the stored case, and "seizure" in d matches a stored "SEIZURE" (and vice versa).

This means a config written STR2COLOR: {"seizure": "red"} — which load_configs stores as {"SEIZURE": "red"} — can still be looked up with the lowercase key the user wrote (CONFIG.X.STR2COLOR["seizure"]) without a surprise KeyError.

keys() / values() / items() / iteration return the stored (canonical) form — they are NOT case-folded. Non-string keys (ints, etc.) are left untouched and matched exactly.

to_dict(include_private=False)

Recursively convert to plain dict.

scitex.io.register_post_save_hook(fn)

Register a function to run after every successful scitex_io.save.

Hooks fire in registration order. They MUST NOT raise — exceptions are swallowed with logger.debug. A misbehaving observer must never break the host’s I/O.

Return type:

None

scitex.io.register_post_load_hook(fn)

Register a function to run after every successful scitex_io.load.

Return type:

None

class scitex.io.H5Explorer(filepath, mode='r')

Bases: object

Interactive HDF5 file explorer.

This class provides convenient methods to explore HDF5 files, inspect their structure, and load data.

Example

>>> explorer = H5Explorer('data.h5')
>>> explorer.explore()  # Display file structure
>>> data = explorer.load('group1/dataset1')  # Load specific dataset
>>> explorer.close()

__init__(filepath, mode='r')

Initialize H5Explorer.

Parameters:

• filepath (str) – Path to HDF5 file

• mode (str) – File opening mode (‘r’ for read, ‘r+’ for read/write)

__enter__()

Context manager entry.

__exit__(exc_type, exc_val, exc_tb)

Context manager exit.

close()

Close the HDF5 file.

explore(path='/', max_depth=None)

Explore HDF5 file structure interactively.

Return type:

None

show(path='/', max_depth=None, indent='', _current_depth=0)

Display HDF5 file structure.

Parameters:

• path (str) – Starting path in HDF5 file

• max_depth (Optional[int]) – Maximum depth to explore (None for unlimited)

• indent (str) – Indentation string (used internally)

• _current_depth (int) – Current depth (used internally)

Return type:

None

keys(path='/')

Get keys at specified path.

Parameters:

path (str) – Path in HDF5 file

Return type:

List[str]

Returns:

List of keys at the specified path

load(path)

Load data from specified path.

Parameters:

path (str) – Path to dataset or group in HDF5 file

Return type:

Any

Returns:

Data from the specified path

get(path)

Alias for load() method for compatibility.

Parameters:

path (str) – Path to dataset or group in HDF5 file

Return type:

Any

Returns:

Data from the specified path

get_info(path='/')

Get information about an item.

Parameters:

path (str) – Path to item in HDF5 file

Return type:

Dict[str, Any]

Returns:

Dictionary with item information

find(pattern, path='/')

Find items matching pattern.

Parameters:

• pattern (str) – Pattern to search for in item names

• path (str) – Starting path for search

Return type:

List[str]

Returns:

List of paths matching the pattern

get_shape(path)

Get shape of a dataset.

Parameters:

path (str) – Path to dataset

Return type:

Optional[tuple]

Returns:

Shape tuple or None if not a dataset

get_dtype(path)

Get dtype of a dataset.

Parameters:

path (str) – Path to dataset

Return type:

Optional[dtype]

Returns:

Numpy dtype or None if not a dataset

scitex.io.explore_h5(filepath)

Explore HDF5 file structure.

Parameters:

filepath (str) – Path to HDF5 file

Return type:

None

scitex.io.has_h5_key(h5_path, key, max_retries=3, action_on_corrupted='delete')

Robust version of has_h5_key that handles corrupted files and lock conflicts.

scitex.io.embed_metadata(image_path, metadata)

Embed metadata into an existing image or PDF file.

Parameters:

• image_path (str) – Path to the image/PDF file (PNG, JPEG, SVG, or PDF)

• metadata (Dict[str, Any]) – Dictionary containing metadata (must be JSON serializable)

Raises:

• ValueError – If file format is not supported or metadata is not JSON serializable

• FileNotFoundError – If file doesn’t exist

Return type:

None

Example

>>> metadata = {
...     'experiment': 'seizure_prediction_001',
...     'session': '2024-11-14',
...     'analysis': 'PAC'
... }
>>> embed_metadata('result.png', metadata)
>>> embed_metadata('result.pdf', metadata)

scitex.io.read_metadata(image_path)

Read metadata from an image or PDF file.

Parameters:

image_path (str) – Path to the file (PNG, JPEG, SVG, or PDF)

Return type:

Optional[Dict[str, Any]]

Returns:

Dictionary containing metadata, or None if no metadata found

Raises:

• FileNotFoundError – If file doesn’t exist

• ValueError – If file format is not supported

Example

>>> metadata = read_metadata('result.png')
>>> print(metadata['experiment'])
'seizure_prediction_001'
>>> metadata = read_metadata('result.pdf')

scitex.io.has_metadata(image_path)

Check if an image file has embedded metadata.

Parameters:

image_path (str) – Path to the image file

Return type:

bool

Returns:

True if metadata exists, False otherwise

Example

>>> if has_metadata('result.png'):
...     print(read_metadata('result.png'))