Durable
jobs
Jobs are saved before work starts and split into retry-safe pieces. After a worker or node failure, they resume from the last completed piece. Forward-only status, counters, manifests, and structured errors show progress and outcomes.
Data Orchestrator
Limestone Data Orchestrator helps AI labs move datasets, model weights, containers, artifacts, and logs across fragmented storage and GPU compute environments with high throughput, reliable and secure execution, and clear monitoring.
Data Orchestrator jobs
Control plane view
12.8 PB
found
9.4 PB
copied
2
errors
Problem
AI workloads span clouds, regions, and private GPU clusters, while the data they need is scattered across cluster storage, object stores, file systems, databases, lakes, warehouses, and registries.
Teams connect these systems with fragile scripts and cloud-specific pipelines. Slow or silent failures leave accelerators idle and pull platform engineers away from infrastructure work.
Solution
A vendor-neutral control plane tracks data movement jobs across storage environments, while nearby data-plane workers execute transfers.
It delivers fast, scalable, observable, reliable, and secure petabyte-scale movement across private clusters and public clouds. This keeps customers in control of their data while reducing accelerator idle time.
Product
Data Orchestrator makes data movement fast, reliable, observable, and easier to operate as AI infrastructure becomes more distributed.
Move data between storage and compute to hydrate training clusters, sync post-training loops, and stage inference models.
Durable
Jobs are saved before work starts and split into retry-safe pieces. After a worker or node failure, they resume from the last completed piece. Forward-only status, counters, manifests, and structured errors show progress and outcomes.
Unified
Register each storage location once; keep credentials in your secrets manager. Assign admin, operator, or viewer roles. Jobs use validated locations that only authorized users can change.
Fast
Workers near storage and compute run pipelined, parallel transfers that scale horizontally to petabyte workloads.
Copy workflows are submitted as durable jobs with explicit state, counters, manifests, and structured errors. The CLI and Python SDK interface with one unified control plane across storage and compute resources for humans and their agents.
do job copy \ --origin s3://training-data/frontier-v4 \ --destination cluster://h100-pool-a/datasets \ --follow-symlinks=false job_id: job_01HX9B7M6R state: RUNNING throughput: 20.1 GB/s files_copied: 18,402,117
Use cases
01
Stage base-model artifacts, containers, configuration, and datasets on the run's cluster. Return checkpoints, logs, and evaluation outputs to durable storage.
02
Supply fine-tuning and reinforcement-learning loops with environments, trajectories, rewards, checkpoints, and evaluations. Sync updated weights to rollout fleets and move work between training and generation pools with recovery state intact.
03
Stage full serving bundles for autoscaling, failover, and cold starts. Move versioned models with dependencies to available compute, keep prior versions for rollback, and return batch outputs to durable storage.
04
Give agents fast, safe access to local data copies in isolated sandboxes. Sync changes through the CLI to central storage as the durable source of truth.
Results
Increase GPU-fleet utilization and reduce wasted GPU-hours and spend.
Shorten time from job submission to execution start, returning valuable time to researchers.
Free platform engineers from custom pipeline work so they can focus on infrastructure.
Speed model iteration and bring model releases forward.
FAQ
AI workloads often run far from the data they need, leaving expensive GPUs idle while engineers move data by hand. Data Orchestrator enables your team to move large datasets, models, and other artifacts durably to available compute, then deliver outputs to your desired long-term storage, unblocking your training and inference workloads. Our solution gives AI researchers and engineers time back while freeing up precious GPU cycles across the fleet.
Data Orchestrator is built for AI teams working with diverse compute and storage systems across clusters, regions, and clouds. It helps researchers move faster with their experiments while giving infrastructure teams a reliable, consistent way to manage data across training, inference, and agentic workloads.
Researchers get faster access to the datasets, checkpoints, model weights, containers, and artifacts their workloads require. Infrastructure engineers get durable data ingestion, fine-grained observability, and a consistent way to stage inputs and collect outputs. The result is faster experimentation and less compute time lost waiting for data.
Data Orchestrator separates data movement from the underlying compute and storage environments. Workloads can easily move across GPU clusters while retaining a consistent view into the data, allowing teams to use the compute that best meets their performance, availability, residency, or cost requirements.
Data Orchestrator moves data only between the storage and compute environments you authorize. In the managed offering, DO encrypts credentials, stores them securely, and uses them only for authorized operations. BYOC means Bring Your Own Cloud: data movement execution remains within your environment, preserving your data-residency and security boundaries. We integrate with Kubernetes and will support Slurm, SkyPilot, or SUNK runtimes in the future.
Today, Data Orchestrator supports Object Stores through Amazon S3 and S3-compatible object stores, Azure Blob Storage, and Google Cloud Storage. We plan to support other storage systems, such as file systems, in the future.