High-performance reinforcement learning framework optimized for speed and scale. Use when you need fast parallel training, vectorized environments, multi-agent systems, or integration with game environments (Atari, Procgen, NetHack). Achieves 2-10x speedups over standard implementations. For quick prototyping or standard algorithm implementations with extensive documentation, use stable-baselines3 instead.
SKILL.md
PufferLib
Use PufferLib with an explicit version profile. Upstream currently has two
incompatible surfaces:
Upstream default branch; not the latest stable PyPI artifact
Native C Ocean environments, native CUDA trainer, optional Torch fallback
Do not combine 3.0 imports with 4.0 config/CLI examples. The 4.0 redesign
removed the 3.0 emulation, vector, and pytorch modules from the current
package tree.
Safe defaults
Start with bundled synthetic, CPU-only, network-free tools.
Do not import an arbitrary environment by dotted path. Bundled tools accept
only allowlisted built-ins and slug identifiers.
Do not install or execute an unreviewed environment package, native
extension, ROM, map, checkpoint, or pickle file.
Verify official source, immutable revision, licenses, checksums or
attestations, and build hooks. Sandbox native builds and first execution.
Cap steps, environments, agents, workers, threads, buffers, memory, disk,
render size, and wall time.
Keep training and evaluation environments/seeds separate.
Default logging to local/none. External logging requires explicit opt-in,
disclosure acknowledgment, and separate artifact-upload approval.
Never pass W&B or Neptune credentials via CLI, INI, JSON, tags, run names, or
logger configuration. Never print them.
Never dump all environment variables or recursively search for .env.
Hash checkpoint bytes before trusted, sandboxed loading; metadata inspection
is not proof of safety.
First local checks
All bundled CLIs are dependency-free and emit strict JSON:
Commit pyproject.toml and uv.lock; verify the archive digest and every
resolved dependency. The source build can compile native code and fetch build
assets, so resolve/build in a sandbox without credentials or sensitive mounts.
The uploaded metadata does not pin Torch or CUDA; do not claim a supported CUDA
matrix that PyPI does not declare.
The current package declares Python >=3.10 and Torch >=2.9. Upstream
PufferTank currently uses Ubuntu 24.04, Python 3.12, and an NVIDIA CUDA
13.0.2/cuDNN development image with the cu130 Torch index, but does not pin
the exact Torch wheel or all system packages. Treat it as a reference, not a
complete lock. Never execute a remote installer directly from a pipe.
Read references/training.md before any installation or build.
Validate spaces, shapes, dtypes, finite rewards, booleans, reset-before-step,
reset-after-end, seeding, and cleanup. terminated is an MDP terminal;
truncated is an external cutoff such as a time limit. Preserve the distinction
for bootstrapping and metrics.
There is no supported 3.0 pufferlib.emulate(...) shortcut matching the old
skill. Read references/environments.md and references/integration.md.
3. Native environments
Published 3.0 PufferEnv requires
single_observation_space, single_action_space, and num_agents before
super().__init__(buf). It uses in-place vector buffers and returns separate
terminal/truncation arrays plus a list of info dictionaries.
Current 4.0 uses C bindings. Start from upstream ocean/squared (single-agent)
or ocean/target (multi-agent), build one environment in local/sanitized mode,
and verify every buffer size/type/index before optimization.
Move to Multiprocessing only after serial traces pass. Record
num_envs, num_workers, batch_size, zero-copy mode, start method, agent
count, masks, and actual returned shapes. For multi-agent environments, batch
length is based on agent slots, not necessarily num_envs.
Read references/vectorization.md. Benchmark fixed work with warmup and at least
three repeats; report simulation and end-to-end training SPS separately. The
bundled benchmark measures only its synthetic harness.
Policy workflow
Published 3.0 policies are Torch modules sized from
single_observation_space/single_action_space. Stable recurrent composition
uses encode_observations and decode_actions; structured emulation uses
pufferlib.pytorch.nativize_dtype and nativize_tensor.
It provides MLP, MinGRU, LSTM, and GRU network choices; --slowly selects this
fallback instead of the native backend. Check output/state shapes, masks,
finite values, gradients, and eager-versus-compiled behavior. See
references/policies.md.
Training and evaluation
Published 3.0 trainer import:
from pufferlib import pufferl
trainer = pufferl.PuffeRL(train_config, vecenv, policy)
The schema rejects secret-bearing keys, unbounded resources, dotted environment
paths, invalid vector divisibility, mixed-version options, and coupled
train/eval seeds. See references/training.md.
Logging
PufferLib 3.0 exposes W&B and Neptune; current 4.0 CLI exposes W&B. Both are
optional external services. They may transmit configuration, metrics, source
metadata, hardware telemetry, output, and approved artifacts, with privacy,
retention, access-control, and cost implications.
W&B credential: named environment variable WANDB_API_KEY.
Neptune credential: named environment variable NEPTUNE_API_TOKEN.
Never put values in arguments/config/logs.
Sanitize config keys before logging.
Keep source/model upload off unless explicitly approved.
It reports only the required variable name and never reads its value.
Checkpoint workflow
PufferLib 3.0 and the 4.0 Torch fallback use Torch serialization; current native
4.0 writes opaque .bin weights. PyTorch warns that untrusted models are
programs and that torch.load uses unpickling.
The inspector hashes and classifies only. It does not call torch.load, import
pickle/Torch, inspect archive members, or extract files. Verify source, license,
architecture, environment revision, sidecar metadata, and checksum before any
sandboxed load. Never use latest in a reproducible evaluation.
