Fine-tuning Large MoE Models#
Compared to PyTorch FSDP, Megatron-LM supports full 5D parallelism, delivering better scaling and efficiency. AReaL fully supports customized RL training with Megatron-LM as the backend. This guide explains how to harness the Megatron training backend and train large MoE models for your application.
Enabling Megatron Backend#
Shifting from FSDP to Megatron requires only a single line of change: the
actor.backend field from fsdp:d4 to megatron:d4.
For a complete guide on allocation mode syntax, parallelism dimensions, and GPU calculations, see the Allocation Mode Reference.
Bridge Backend Selection#
MegatronEngine supports two bridge backends configured by
actor.megatron.bridge_type:
actor:
megatron:
bridge_type: mbridge # default (backward compatible)
Set bridge_type: megatron-bridge to use the new backend.
For trade-offs and migration guidance, see the Megatron Bridge Backend Reference.
Freezing Vision Parameters#
Qwen3-VL and Qwen3.5 (including Qwen3.8-27B) using megatron-bridge freeze the vision
encoder and its projector/merger by default. No environment variable or CLI override is
needed. The explicit equivalent is:
actor:
megatron:
bridge_type: megatron-bridge
freeze_vision_model: true
freeze_vision_projection: true
The equivalent CLI overrides are actor.megatron.freeze_vision_model=true and
actor.megatron.freeze_vision_projection=true. These settings reach the native model
provider before DDP and optimizer construction. To freeze only the encoder while keeping
the merger trainable, set freeze_vision_model: true and
freeze_vision_projection: false. Freeze both components to keep cached visual
embeddings valid across language-model updates, assuming the image preprocessing is
unchanged.
Both options default to null: Qwen3-VL/Qwen3.5 resolve this to true, while other
models preserve their provider’s freezing policy. Set either option to false to train
that component. Explicit settings with an unsupported bridge or provider raise an error.
Qwen4-Exp (Qwen3.8 Flash Next) uses a separate mcore-bridge adapter that already
freezes its entire visual module, including its merger, without these options.
MoE Parallel Strategy#
For MoE models, Megatron supports separate parallelism for attention and FFN modules using the hybrid syntax. For example:
megatron:(attn:d1p4t2c2|ffn:d1p4t1e4)
This 16-GPU configuration uses PP=4, with attention modules using TP=2 and CP=2, while expert modules use TP=1 and EP=4. See MoE Parallel Folding for details on this feature.
Tuning Guides:
Aligning Inference and Training Precision#
Due to the sparse nature of MoE models, the logits calculated by forward passes during
inference and training could be severely misaligned, leading to unstable training
results. To mitigate this instability, it is highly recommended to set
actor.megatron.use_deterministic_algorithms=True to disable nondeterministic
calculations in Megatron, although this may cause a ~10-20% slowdown in training steps.
As an example, you can run GRPO on the Qwen3 30B-A3B MoE model and GSM8K dataset (on a 32-GPU ray cluster) directly with the following command:
# NOTE: Allocation mode here is only for illustration purposes. It is not optimized.
python3 examples/math/gsm8k_rl.py --config <megatron_config.yaml> \
scheduler.type=ray \
experiment_name=megatron-moe-gsm8k-grpo trial_name=trial-0 \
rollout.backend=sglang:d4t4 actor.backend=megatron:(attn:d1p4t2c2|ffn:d1p4t1e4) \
cluster.n_nodes=4 cluster.n_gpus_per_node=8 actor.path=Qwen/Qwen3-30B-A3B \
actor.megatron.use_deterministic_algorithms=True