In AI agent training, you can pay for powerful GPUs and still spend much of your time waiting. Agents need to attempt tasks and produce results before the model can learn from them. A few slow attempts can hold up the next training update, making the systems that run those tasks a major factor in training speed and cost.

Di Jin, VP of AI Training at Amsterdam-headquartered Nebius, says teams coming from pretraining or supervised fine-tuning can underestimate the work these systems do.

“If you staff and design it as a training project, inference becomes the bottleneck nobody owns.”

Rollout generation often takes most of a training step’s wall-clock time, making batching, cache management and scheduling central to how fast the model can learn. “If you staff and design it as a training project, inference becomes the bottleneck nobody owns,” Jin says.

Time to a trainable batch

He adds that engineers should measure time to a trainable batch alongside throughput. Average tokens per second can hide the long tail: in a synchronous batch, the step cannot finish until the final trajectory is completed. 

Agent trajectories do not all have the same length. Many finish quickly, while others take longer, leaving training GPUs waiting on a handful of stragglers. For this reason, Jin advises engineers to treat the inference engine as part of the post-training reinforcement learning optimizer.

Data misunderstood

Learning from data generated by an older model version involves a tradeoff. Some policy lag can improve GPU efficiency by letting generation and training overlap, Jin says, but its safety depends on how much the policy has changed and how the training algorithm handles the difference.

Jin tells The New Stack, “In fully synchronous RL, inference GPUs wait while training runs, and training GPUs wait while generation runs. Letting generation use weights that are one or a few updates old allows both to run in parallel. That overlap is how most large-scale systems reach reasonable utilization.” When success on a hard task is rare, he adds, an already successful trajectory can be valuable to reuse even if it is slightly stale.

Staleness becomes harmful when the policy that generated a trajectory has drifted too far from the policy being trained. The useful measure is divergence, not a universal limit on how many training steps old the data can be.

“Staleness isn’t the enemy. Unmeasured staleness is.”

Rather than seeking to eradicate old data, engineers and organizations should measure and manage staleness. They can apply importance-sampling corrections, usually truncated to limit variance; drop or down-weight samples beyond a staleness threshold; and track divergence between the generating policy and the current policy. Jin says: “Staleness isn’t the enemy. Unmeasured staleness is.”

Environments are the new data

In Jin’s experience, data and environments take up most of the work in successful RL post-training. Algorithm design is the smallest slice, while infrastructure enables fast iteration. “Infrastructure determines how fast you can iterate. But what the model actually learns is decided by the tasks, environments, and rewards you give it,” he says.

For agents, that means focusing on the environments that generate training experience. Supervised fine-tuning uses a collected dataset; in agentic RL, the model generates trajectories by interacting with an environment. 

Reproducible containers, codebases, websites, and APIs with realistic states help the model’s learning transfer to real use. Engineers also need reliable verifiers and tasks at a useful difficulty level. As the model improves, the task set needs refreshing so tasks don’t saturate and stop producing a learning signal. Jin says: “Building environments is the new data curation.”

As with previous technologies, problems occur where systems hand over to one another. The trainer, inference engine, sandbox, and verifier may each work on their own, yet failures emerge at the interfaces. A model can exploit verifier loopholes and receive rewards for behavior that doesn’t accomplish the intended task. Sandboxes can also fail or misbehave, corrupting the training data.

Engineers should record whether an episode ended in model success, model failure, infrastructure failure, or an unknown outcome. Exclude failures attributable to infrastructure from policy-gradient updates. Failures caused by the model’s own actions should still be scored as model failures, so triggering a timeout or another error doesn’t become a way to avoid a penalty.

Long-horizon agentic RL combines trajectories, tool calls, and external latencies, creating bursts of GPU demand and periods of waiting. For GRPO-style training, where sibling attempts at a task are scored against one another, engineers should schedule around group completion rather than launch order. A group enters a ready bank once all sibling attempts finish; the trainer then takes eligible complete groups regardless of when they were launched. Time to a trainable batch is the governing measure, while tokens per second remains useful for diagnosing inference. Routing matters because keeping a conversation close to its cached history can avoid repeated prefill work.

In a Nebius experiment on a Terminal-Bench-based workload, changes to routing, sandbox execution, and batch assembly reduced fixed-weight collection time for a 256-trajectory batch by 66.9%. That measured batch collection, excluding the trainer’s forward, backward, and optimizer work; it did not measure complete training speed.

​For agentic AI rollouts, an infrastructure goal is to reduce the time expensive GPUs spend waiting on sandboxes, browsers, code execution, and other non-GPU dependencies.

“Your most expensive GPU is often waiting on your cheapest container.”

Jin recommends scaling CPU and sandbox capacity independently of GPUs and provisioning it generously, because idle containers are comparatively inexpensive. “Your most expensive GPU is often waiting on your cheapest container,” he says.

​Keeping sandboxes ready, using snapshot and fork methods, can take environment startup off the critical path. It doesn’t remove the time needed to run tools. Jin recommends putting sandboxes on the same network fabric as the GPUs where possible to minimize the cumulative latency of the hundreds of tool calls an agent may make. Per-step timeouts and budgets can limit the disruption from a hung browser or process. Engineers should track GPU-hours spent waiting on environments, alongside utilization and time to a trainable batch, to identify where additional investment would help.

When the maths does not match

During RL post-training, the rollout engine and the trainer can assign different probabilities to the same tokens, even when they start from the same weights. Quantization, kernels, batching, and, in mixture-of-experts models, expert routing can all contribute. Those differences can introduce systematic bias into training updates.

Jin recommends aligning the numerics first, then measuring and correcting the remaining mismatch. That can mean matching the quantization scheme and computation used in the trainer’s forward pass to the rollout engine, while retaining higher-precision weights for optimization. Importance-sampling corrections can help manage residual differences, but clipping alone should not be assumed to solve every form of mismatch.

Evaluation should use the precision, inference engine, and version that will serve the model. Changing quantization or kernels calls for another evaluation.

Keeping the weights in step

As training updates the policy, rollout workers need a reliable way to receive new weights while keeping policy lag within acceptable limits. Jin describes three broad patterns: resharding weights in memory when training and inference share GPUs; transferring weights directly to separate inference workers; and publishing checkpoints that inference engines hot-load through storage. The best choice depends on the model, hardware topology, and freshness requirements.

A full service restart can introduce avoidable downtime, but storage-based publication does not have to require one. Asynchronous transfer and compact updates can move much of the work off the critical path. Teams should measure end-to-end publication latency separately from the pause needed to apply the weights.

Weight publication also needs an explicit plan for active requests and cached state. Depending on the engine, teams can finish, abort, or pause requests before applying an update, and clear or recompute cached context as needed. If a trajectory spans policy versions, record which version generated its tokens so the training system can handle that mixture deliberately. Coordinate transfers with rollout traffic where they share resources, and monitor publication delays alongside policy age.

Designing these components together helps teams improve iteration speed while keeping training data and policy updates reliable. Jin advises smaller teams to start with a simple setup and add asynchronous complexity when measured bottlenecks justify it.

Before scaling, check:

• Batch readiness: How long does it take to assemble a complete, usable batch, including the straggler tail?
• Policy freshness: Can every rollout be traced to the policy version that generated it, and what drift makes it too stale to use?
• Numerical alignment: How different are the trainer’s and rollout engine’s probabilities, and what correction handles the remaining gap?
• Environment reliability: Can you distinguish infrastructure failures from model failures and reproduce an affected episode?
• Weight publication: What happens to active requests and KV caches during an update, and how long does publication actually take?
• Evaluation: Are you testing the model in the precision and serving stack users will encounter?

The post Nebius cut rollout collection time by 66.9% — here’s what changed appeared first on The New Stack.

Voxel51 has launched version 3.0 of its data platform, adding new tools designed to help robotics and physical AI developers manage the huge quantities of camera, video and sensor data generated by machines operating in the real world. The basic problem Voxel51 is trying to solve is straightforward: robots need large amounts of real-world data […]

Impressive benchmark results should make it easy for engineers to select the right open model for the outcome they are tasked with developing. However, that is not the case. 

Benchmarks indicate an open model’s optimal performance, but in production, a model must work repeatedly. The challenge is that in production, other factors matter, such as the information provided to the model and its accuracy. So benchmarks can’t tell an engineer whether the AI they are developing will complete the task accurately, or even at all. This is especially important for long-horizon AI tasks.

Ryan Hanrui Wang, SVP of Applied AI at AI cloud provider Nebius, tells The New Stack it is important to measure an open model by the industry benchmarks in conjunction with internal benchmarks and harnesses to ensure you get the inference outcomes the organization requires, something he says the Amsterdam, Netherlands-based company always does in partnership with its clients. 

Wang joined Nebius when it acquired Eigen AI, the inference-optimization startup he co-founded and led as CEO, in a roughly $643 million deal announced in May; Nebius sells inference through its Token Factory platform. 

“Customers often have their own benchmarks, so we can use their agent workflow and its traces to see the level of success we can work with. We always work closely with them to define the benchmarks.”

Wang goes on to add: “The public benchmark may not cover the multi-domain tasks that an engineer wants inference to do in the actual production environment.” He says this can misalign agent tasks with the true cost of inference calls. “The workflow is not tested by the benchmark design. There needs to be in-context learning.”

Engineers and organizations should therefore not spend too much time selecting the open model; they should look at the entire inference stack. Model selection should depend on whether the engineer needs a lower-cost, flexible open-model stack or a frontier model for inference. 

For those who need lower cost, Wang says Nebius evaluates open models such as Kimi and DeepSeek, as well as lighter “flash” models, to see whether they can meet inference accuracy needs and whether fine-tuning is required. 

Those who want superior accuracy and quality usually narrow it down to two or three large open models and then fine-tune them to improve accuracy. Wang says open model selection is only the first step; after selection, they need to ensure long-context tasks and cached inputs deliver the expected outcome.

Inference and post-training

In post-training, Wang says the organization uses its data to improve accuracy and task fit, while the serving layer, quantization, batching, caching, and kernel optimization drive inference speed. In addition, engineers should develop the cache system for production traffic, ensuring the hit rate is maximized for the workload so repeated inputs hit the cache rather than being recomputed. 

“The other important thing is throughput, the measure of how many tokens are generated in a certain time span, as they are a major trade-off against interactivity,” Wang adds. Engineers need to balance better utilization and cheaper tokens with acceptable interactivity– per-user speed.

Getting concurrency right ensures the organization uses the open model effectively; it is a fine balancing act for engineers. If compute isn’t fully utilized, token cost is high. But high utilization with increased concurrency can reduce end-user token-per-second speed.

Engineers also need to watch tail latency, not just averages. “P95 and P99 are important metrics for the per-user token speed and time to first token (TTFT); the time they need to wait needs to be below five seconds for some of our customers,” Wang says. A system can post a great average response time and still be unusable if one request in twenty stalls.

Modeling for accuracy

Engineers need to be careful when considering benchmark results and accuracy. Benchmarks may test reasoning on a mathematical problem, which isn’t very useful if the engineer is developing an AI agent for customer support, where high emotional intelligence (EQ) matters more. “So there is a misalignment between the different benchmarks and actual production,” Wang says.

On cost per successful task, Wang says engineers must remember that cheap tokens are useless if the model is inaccurate.

Engineers need to consider the end-to-end process the AI will perform. This lets engineers select the right-sized open model for the AI task. Wang gives the example of a specialized assistant agent where Nebius gathered 6,000 traces to refine a small model and improved accuracy by 11% while executing 22% fewer tool calls. 

He raises this point because it counters the tendency to think bigger is better, but accuracy gains from increasing model size diminish. “There is typically a tipping point for this kind of task,” Wang says. For a task like customer support, he says, 10 billion parameters could be enough, while 100 billion or even a trillion parameters yields only minimal accuracy gains. “What we want to do is find the sweet spot.”

Engineers can use large models as a reference point, then tune the reasoning with their organizational data to find an open model with the right accuracy and size for the task. 

“We typically do a sweep of several models, using the largest model as a reference to know where the sweet spot for model size is,” Wang says. “For a complicated task, we can dissect the complex task into smaller subtasks, working with the customer, which connects workflows and can then use smaller models.” 

Wang describes this process as a division of labor, referencing the famous economist Adam Smith.

This approach shows the course of AI engineering maturity. Like every sector in technological and engineering history, the initial gold rush targets bigger-is-better. As the engineering matures, it focuses on outcomes and engineering excellence, and therefore on working with partners who understand how to achieve them. 

Sometimes that is a bigger model; other times a small niche development is best.

The post Assess the entire inference stack — not just open model benchmarks appeared first on The New Stack.