PyTorch FSDP: sharding integration and bounded resource scheduling

What changed

Integrated fully sharded data parallelism with PyTorch tensor/dispatcher/allocator components, deferred model initialization, topology-aware sharding choices and parameter prefetch/memory scheduling.

Scientific scope

Integrated fully sharded data parallelism with PyTorch tensor/dispatcher/allocator components, deferred model initialization, topology-aware sharding choices and parameter prefetch/memory scheduling. Inherits ZeRO-style state sharding and data parallelism; system co-design is distinct from inventing all sharding. Original template's2020placeholder is not the2023preprint date. Near-linear TFLOPS scaling is tested, not universal throughput, numerical equivalence or current PyTorch release-status verification.

Evidence

The linked original sources describe the stated contribution and its evidence limits. WRS has not independently rerun the experiments or certified a proof. Inherits ZeRO-style state sharding and data parallelism; system co-design is distinct from inventing all sharding. Original template's2020placeholder is not the2023preprint date. Near-linear TFLOPS scaling is tested, not universal throughput, numerical equivalence or current PyTorch release-status verification.

Contributors

• Source-listed authors and technical contributors · SCIENCE: Source-listed authors or explicitly credited technical contributors for the cited versions. Family entries retain source-specific attribution; a combined list is not one original-paper byline or equal individual work. Names alone do not establish resolved identities.

• PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Provisional WRS assessment, edition 2026-10-16.1. Independent expert calibration is pending. This grade does not certify correctness or assign ownership.

Full scientific record · Browse contribution grades · Request a correction