Skip to content

General Configuration

The General Configuration section defines parameters that affect the overall behavior of WarpRec. It allows customization of numerical precision, device usage, logging verbosity, and integration of custom models or callbacks.

Available Keywords

  • device: Device used for training and evaluation. Supports cpu or cuda devices. Defaults to cpu.
  • backend: Backend to use for data reading and writing. Supports polars or pandas. Defaults to polars.
  • ray_verbose: Verbosity level of Ray Tune. Acceptable values are integers from 0 (silent) to 3 (very verbose). Defaults to 1.
  • ray_address: The address of the Ray cluster to connect to. Defaults to auto, which means that Warprec will automatically detect the Ray cluster.
  • cpu_data_prep: The cpu resource to request for data preparation phase. Defaults to 1.
  • custom_resources_data_prep: The custom resources to request for data preparation phase.
  • label_selector_data_prep: The label selector used for data preparation phase.
  • time_report: Whether to report the time taken by each step. Defaults to True.
  • custom_modules: Python modules to import custom implementations into WarpRec. Can be a string or a list of strings.
  • callback: Nested section to configure a custom callback.
  • azure: Nested section to configure Azure information needed for reading and writing data from/to Azure Blob Storage.

Callback Configuration

The callback section allows pointing to a custom callback implementation and passing initialization parameters directly from the configuration:

  • callback_path: Path to the Python script containing the callback implementation.
  • callback_name: Name of the callback class. Must inherit from WarpRecCallback.
  • args: List of positional arguments to pass to the callback constructor.
  • kwargs: Dictionary of keyword arguments to pass to the callback constructor.

Important

Custom callbacks must inherit from WarpRecCallback and be compatible with the provided arguments and keyword arguments. Follow this guide on how to implement your first callback.

Azure Configuration

The azure section is required when using Azure Blob Storage for reading or writing data. The remote-io extra must be installed to enable this functionality. The section includes:

  • storage_account_name: Name of the Azure Storage Account.
  • container_name: Name of the Azure Blob container.

Example General Configuration

Below is a complete example of a general configuration including precision, verbosity, and a custom callback:

general:
    device: cuda
    ray_verbose: 0
    callback:
        callback_path: path/to/the/script.py
        callback_name: class_name
        args: [arg_1, arg_2]
        kwargs:
            kwargs_1: kwargs_value_1
            kwargs_2: kwargs_value_2