Weights and Biases
Deep Lake’s Weights and Biases integration allows you to track and improve reproducibility of your machine learning experiments. Deep Lake will automatically push all information required to reproduce the snapshot of the data like your dataset’s URI, commit ID, and view IDs of any views that you have used in your training workflow.
Learn more about Weights and Biases here.
Logging Dataset Creation
If you create a Deep Lake dataset using any of the functions mentioned in Creating Datasets, just perform a commit on the dataset to log its creation on W&B.
>>> run = wandb.init(project="deeplake_wandb", job_type="dataset_upload") >>> ds = deeplake.empty("hub://fayazrahman4u/my_dataset") # create dataset >>> ds.create_tensor("images", htype="image", sample_compression="jpg") # create a tensor >>> ds.images.append(deeplake.read("files/images/dog.jpg")) # add a sample >>> ds.commit("creation") # commit -> trigger logging >>> run.finish()
If you created your dataset using
deeplake.deepcopy(), perform the commit only if you have head changes.
If you make changes to an existing dataset, commit the changes with an active Weights and Biases run to log it’s state.
Logging Dataset Read
>>> run = wandb.init(project="deeplake_wandb", job_type="torch dataloader") >>> train_loader = ds.pytorch() >>> run.finish()
>>> run = wandb.init(project="deeplake_wandb", job_type="iteration") >>> for sample in ds: >>> print(sample["images"].shape) >>> run.finish()