Skip to content

Swarm Pipeline

The Swarm Pipeline represents an alternative execution strategy to the standard train pipeline, enabling full parallelization across both models and hyperparameter trials. Prerequisites and configuration requirements remain identical to those defined for the train pipeline.

Warning

This pipeline is specifically optimized for maximum throughput, orchestrating multiple tuning processes concurrently to minimize overall training time. As a consequence of this high degree of parallelism, terminal logs may become non-deterministic and difficult to interpret. For reliable insights, users should refer to the consolidated outputs generated by the writer module.

When to Use

  • Performing large-scale model training and hyperparameter optimization tasks
  • Aiming to maximize utilization of all available cluster resources
  • Minimizing end-to-end training time is a primary objective