Survey on Recent MoE systems

Table of Contents

Background
- MoE
- NCCL
Switch Transformer
FastMoE
Tutel

Background

MoE

NCCL

SC15: https://images.nvidia.com/events/sc15/pdfs/NCCL-Woolley.pdf

Switch Transformer

link: https://arxiv.org/pdf/2101.03961.pdf

FastMoE

link: https://arxiv.org/pdf/2103.13262.pdf
author's blog post (in Chinese): https://laekov.com.cn/view/6b400a

Tutel

link: https://github.com/microsoft/tutel

A recent implementation from Microsoft.

Author: expye(Zihao Ye)

Email: expye@outlook.com

Date: 2021-11-24 Wed 00:00

Last modified: 2024-07-04 Thu 10:35

Licensed under CC BY-NC 4.0