Weights and Biases

Deep Lake’s Weights and Biases integration allows you to track and improve reproducibility of your machine learning experiments. Deep Lake will automatically push all information required to reproduce the snapshot of the data like your dataset’s URI, commit ID, and view IDs of any views that you have used in your training workflow.

Learn more about Weights and Biases here.

Logging Dataset Creation

If you create a Deep Lake dataset using any of the functions mentioned in Creating Datasets, just perform a commit on the dataset to log its creation on W&B.

>>> run = wandb.init(project="deeplake_wandb", job_type="dataset_upload")
>>> ds = deeplake.empty("hub://fayazrahman4u/my_dataset") # create dataset
>>> ds.create_tensor("images", htype="image", sample_compression="jpg") # create a tensor
>>> ds.images.append(deeplake.read("files/images/dog.jpg")) # add a sample
>>> ds.commit("creation") # commit -> trigger logging
>>> run.finish()

Note

If you created your dataset using deeplake.deepcopy(), perform the commit only if you have head changes.

Note

If you make changes to an existing dataset, commit the changes with an active Weights and Biases run to log it’s state.

Logging Dataset Read

A dataset read will be logged if you iterate over a dataset or call Dataset.pytorch() or Tensor.numpy() on its tensors.

>>> run = wandb.init(project="deeplake_wandb", job_type="torch dataloader")
>>> train_loader = ds.pytorch()
>>> run.finish()

>>> run = wandb.init(project="deeplake_wandb", job_type="iteration")
>>> for sample in ds:
>>>     print(sample["images"].shape)
>>> run.finish()