Huginn recurrent policy training

Huginn recurrent policy training#

Train Huginn-0125 at checkpoint revision bb6621b65e90b6a4b9b29ef88dc83866d450470c. The provider preserves the prelude, recurrent core, coda, sandwich norms and checkpoint’s fixed mean_recurrence. Each completion carries its own like-init-cpu-f32-v1 latent seed/position trace, replayed under current actor and critic weights. Its RoPE buffer remains FP32 when parameters move to FP16.

--algorithm

Objective

Critic

Defaults

ppo

Clipped rollout-policy ratio

Yes

GAE, one completion/prompt

grpo

Clipped group-relative advantages

No

Four completions/prompt

dppo

Decoupled PPO with proximal/behavior correction

Yes

Recompute proximal scores; correction clipped to [0.5, 2]

flow-dppo

Categorical full-vocabulary KL gate

Yes

Budget 0.01, two optimizer steps/rollout

The shared recipe defines each objective, environment and checkpoint contract. DPPO uses VIME’s three-policy importance correction; Flow-DPPO applies its divergence gate to categorical token probabilities.

Use the pinned native engine and patched Megatron from the shared recipe, with one training GPU and one dedicated rollout GPU. The model defaults to FP16, fixed loss scale 128 and FP32 gradient accumulation. JSONL rows contain string prompt and label fields for the existing math reward.

python examples/huginn/run.py --algorithm grpo \
  --model /models/huginn-0125 --data /data/math.jsonl \
  --output /runs/huginn-grpo --updates 3 --recompute \
  --model-revision bb6621b65e90b6a4b9b29ef88dc83866d450470c \
  --engine-revision d2a358933393f25d74dd2bdd1068a741cd5d9226

Run each objective in a separate directory. For a fresh-process resume, add --stop-after 2 to an interrupted run, then rerun with --resume and without that flag, keeping total updates and all settings fixed. Compare full actor/critic, optimizer, scheduler and RNG state, tokens, rewards, latent traces and publication digests with uninterrupted training. GRPO restores only its actor; equal group rewards correctly produce zero advantages.

The tied embedding is published under both native aliases, checked for equality by the engine. The fixed RoPE buffer must match the model revision. Complete weight publication commits before generation resumes. Persist PyTorch version, dtype and CPU architecture with the latent profile when comparing fresh runs.