Metadata

Metadata provides key-value storage for datasets and columns.

Dataset Metadata

deeplake.Metadata

Bases: ReadOnlyMetadata

Writable access to dataset and column metadata for ML workflows.

Stores important information about datasets like:

• Model parameters and hyperparameters
• Preprocessing statistics
• Data splits and fold definitions
• Version and training information

Changes are persisted immediately without requiring commit().

Examples:

Storing model metadata:

ds.metadata["model_name"] = "resnet50"
ds.metadata["hyperparameters"] = {
    "learning_rate": 0.001,
    "batch_size": 32
}

Setting preprocessing stats:

ds["images"].metadata["mean"] = [0.485, 0.456, 0.406]
ds["images"].metadata["std"] = [0.229, 0.224, 0.225]

__contains__

__contains__(key: str) -> bool

Checks if the metadata contains the given key.

__getitem__

__getitem__(key: str) -> Any

Gets metadata value for the given key.

Parameters:

Name	Type	Description	Default
key	str	Metadata key to retrieve	required

Returns:

Type	Description
Any	The stored metadata value

Examples:

mean = ds["images"].metadata["mean"]
std = ds["images"].metadata["std"]

__setitem__

__setitem__(key: str, value: Any) -> None

Sets metadata value for given key. Changes are persisted immediately.

Parameters:

Name	Type	Description	Default
key	str	Metadata key to set	required
value	Any	Value to store	required

Examples:

ds.metadata["train_split"] = 0.8
ds.metadata["val_split"] = 0.1
ds.metadata["test_split"] = 0.1

keys

keys() -> list[str]

Lists all available metadata keys.

Returns:

Type	Description
list[str]	list[str]: List of metadata key names

Examples:

# Print all metadata
for key in metadata.keys():
    print(f"{key}: {metadata[key]}")

# Set dataset metadata
ds.metadata["description"] = "Training dataset"
ds.metadata["version"] = "1.0"
ds.metadata["params"] = {
    "image_size": 224,
    "mean": [0.485, 0.456, 0.406],
    "std": [0.229, 0.224, 0.225]
}

# Read dataset metadata
description = ds.metadata["description"]
params = ds.metadata["params"]

# List all metadata keys
for key in ds.metadata.keys():
    print(f"{key}: {ds.metadata[key]}")

Column Metadata

deeplake.ReadOnlyMetadata

Read-only access to dataset and column metadata for ML workflows.

Stores important information about datasets like: - Model parameters and hyperparameters - Preprocessing statistics (mean, std, etc.) - Data splits and fold definitions - Version and training information

Examples:

Accessing model metadata:

metadata = ds.metadata
model_name = metadata["model_name"]
model_params = metadata["hyperparameters"]

Reading preprocessing stats:

mean = ds["images"].metadata["mean"]
std = ds["images"].metadata["std"]

__contains__

__contains__(key: str) -> bool

Checks if the metadata contains the given key.

__getitem__

__getitem__(key: str) -> Any

Gets metadata value for the given key.

Parameters:

Name	Type	Description	Default
key	str	Metadata key to retrieve	required

Returns:

Type	Description
Any	The stored metadata value

Examples:

mean = ds["images"].metadata["mean"]
std = ds["images"].metadata["std"]

keys

keys() -> list[str]

Lists all available metadata keys.

Returns:

Type	Description
list[str]	list[str]: List of metadata key names

Examples:

# Print all metadata
for key in metadata.keys():
    print(f"{key}: {metadata[key]}")

# Set column metadata
ds["images"].metadata["mean"] = [0.485, 0.456, 0.406]
ds["images"].metadata["std"] = [0.229, 0.224, 0.225]
ds["labels"].metadata["class_names"] = ["cat", "dog", "bird"]

# Read column metadata
mean = ds["images"].metadata["mean"]
class_names = ds["labels"].metadata["class_names"]
ds.commit() # Commit the changes to the dataset