Introducing MLBench

07 Sep 2018

MLBench is a framework for distributed machine learning. Its purpose is to improve transparency, reproducibility, robustness, and to provide fair performance measures as well as reference implementations, helping adoption of distributed machine learning methods both in industry and in the academic community.

MLBench is public, open source and vendor independent, and has two main goals:

1. to be an easy-to-use and fair benchmarking suite for algorithms as well as for systems (software frameworks and hardware).
2. to provide re-usable and reliable reference implementations of distributed ML algorithms.

Main Features

MLBench is based on Kubernetes to ease deployment in a distributed setting, both on public clouds and on dedicated hardware. It supports several standard machine-learning frameworks and algorithms

• Easy setup: MLBench can be installed with a single shell command.
• Convenient Dashboard: MLBench comes with a dashboard that allows easy access and management for running experiments
  • Monitoring: The dashboard allows you to monitor resource usage of all worker nodes participating in experiments
  • Experiment Setup: Easily start one of the reference experiments or define your own, specifying resource usage, number of nodes, type of experiment and more.
  • Visualizations: Quickly get visualizations of your runs, including quality metrics and resources usage in total and on individual workers
• Expandable: We aim to easily allow users to implement and add their own models and algorithms to MLBench, with minimal changes, while still benefiting from all the features of the benchmark environment.
• Fairness and Reproducibility: By providing precise specifications of the benchmark ML tasks, metrics as well as reference implementations, MLBench provides fair baselines and improves transparency.
• Versatile: We aim to support as many of the platforms, ML frameworks and common machine learning task as possible.

Community

Our project is open, vendor independent and backed by academic standards, and we highly value contributions from the community

• Github

• Mailing list

Getting Started

For installation and quickstart, please refer to the documentation.