Object Storage Dataset

Contents

• Object Storage Dataset
  • Introduction
  • Python API
  • Use Cases
  • Indices

To get an an iterable style object from large/out-of-memory CSV files stored in local file-system or Cloud storage platforms like Amazon-S3, Google cloud storage, and Azure blob storage.

The ObjectStorageDataset provides support for tensor-based, out-of-memory datasets for the iterable-style interface. The ObjectStorageDataset is not available by default when you install PyTorch, so you need to install it separately in your Python environment using:

pip install osds

and once installed, import the class in your runtime using:

from osds.utils import ObjectStorageDataset

Introduction

• Quick Start: Introduction
  • Intiate URL File Path
  • Batch Size
  • Eager Load Batches
  • Data Type

Python API

• Object_Storage_Dataset Class
  • Glob
  • Storage_Options:
  • Batch Size
  • DataType (dtype)
  • Eager_load_batches
  • fits_in_node_memory
  • worker
  • cache_dir
  • tensor_cache_size
  • partition_cache_size
  • batch_cache_size
  • replicas

Use Cases

• Iterable object from: Google Storage
• Iterable object from: Amazon Cloud Storage
• Iterable object from: local file system (Linux)
• Iterable object from: local file system (Windows)

Indices

• License

• Search Page