NVIDIA Scale-In: The Fifth Pillar of the AI Factory

NVIDIA Scale-In: The Fifth Pillar of the AI Factory

For more than a decade, cloud data centers were built around a straightforward principle: pool general-purpose compute, storage, and networking resources, then allocate them dynamically through software. That architecture powered the growth of the internet, mobile applications, and modern cloud computing.

The rise of agentic AI is changing that model.

AI systems are no longer limited to responding to individual prompts. Autonomous agents can plan tasks, invoke tools, retrieve memory, access enterprise databases, enforce security policies, call multiple models, and exchange data across distributed infrastructure. A single user request can therefore generate a large number of compute, storage, networking, and security operations.

At AI-factory scale, those infrastructure operations can consume significant CPU resources and introduce latency into the very workloads that generate tokens.

NVIDIA’s answer is Scale-In, the fifth pillar of its AI Factory networking architecture. The concept moves infrastructure services such as networking, security, storage acceleration, telemetry, and management onto dedicated hardware so host CPUs and GPUs can remain focused on AI workloads.

🏭 The Five Pillars of NVIDIA’s AI Factory

#

Scale-In makes more sense when viewed alongside NVIDIA’s broader AI Factory architecture.

Pillar	Primary Role
Scale-Up	Connects GPUs within a server or compute node using extremely high-bandwidth, low-latency interconnects
Scale-Out	Connects large numbers of compute nodes for distributed AI training and inference
Scale-Across	Connects AI factories across multiple data centers and geographic locations
Context Memory (CMX)	Provides infrastructure for sharing and accessing reusable inference context and KV-cache data
Scale-In	Offloads networking, security, storage, management, and infrastructure services from host CPUs

A useful analogy is a large industrial park.

Scale-Up connects equipment inside individual workshops. Scale-Out provides the logistics network between workshops. Scale-Across connects multiple industrial parks. Context Memory functions like a nearby warehouse for frequently reused materials.

Scale-In operates the infrastructure surrounding the entire park: gates, security checkpoints, logistics, traffic control, and centralized operations.

The goal is to ensure that infrastructure services do not consume the computing resources needed to manufacture AI tokens.

🌐 Why Traditional North-South Networking Is No Longer Enough

#

Traditional cloud architectures were designed around north-south traffic between external users, applications, and data centers. Software-defined networking made it possible to dynamically allocate resources as workloads changed.

Agentic AI introduces a much more demanding traffic pattern.

An AI agent may simultaneously:

• Receive requests from users or other agents
• Invoke multiple models and tools
• Retrieve information from databases
• Read and write object or file storage
• Access enterprise applications
• Perform security and policy checks
• Exchange information with other agents
• Retrieve and update long-lived context

The resulting traffic can involve both north-south communication with external resources and east-west communication between services inside the AI factory.

If every infrastructure operation runs through the host CPU, AI servers increasingly spend their resources on networking, storage, virtualization, security, and management rather than model execution.

Scale-In changes the division of labor by moving these functions to dedicated infrastructure processors.

Instead of asking an AI worker to simultaneously perform production work, guard the factory entrance, manage inventory, and operate logistics, the factory assigns those responsibilities to specialized infrastructure systems.

⚡ BlueField-4: The Hardware Foundation of Scale-In

#

At the center of Scale-In is NVIDIA’s BlueField-4 DPU.

The DPU creates a dedicated infrastructure-processing domain inside each AI server, allowing networking, security, storage, and management services to operate independently from the host system.

Key components include:

• 64-core Grace CPU: Runs infrastructure software for policy enforcement, configuration, telemetry, orchestration, and management.
• Inline acceleration engines: Accelerate packet processing, RDMA, storage protocols, encryption, firewall functions, and policy enforcement directly in the data path.
• LPDDR5X memory: Provides high-bandwidth memory for infrastructure services while improving power efficiency.
• PCIe Gen6: Provides a high-speed connection between the DPU and host system.
• 800 Gb/s networking: Provides the external connectivity required by high-bandwidth AI infrastructure.

Compared with BlueField-3, NVIDIA positions BlueField-4 as delivering approximately 6× the compute performance, 4× the memory bandwidth, and 2× the network bandwidth.

That additional capacity allows a single DPU to host more infrastructure services while processing substantially greater traffic volumes.

🔀 BlueField-4 and the Vera Rubin NVL72 Architecture

#

The division of responsibilities becomes particularly important in NVIDIA’s next-generation AI systems.

Within a Vera Rubin NVL72 rack, the networking architecture separates workload traffic from infrastructure services.

ConnectX-9 SuperNIC handles high-performance tenant workload traffic across the Scale-Out network, while BlueField-4 handles infrastructure functions such as connectivity, security, storage, and management.

Each compute tray can provide aggregate network connectivity of up to 7.2 Tb/s, with the architecture allocating:

• 800 Gb/s for north-south connectivity through BlueField-4
• Four 1.6 Tb/s paths for east-west traffic

This separation prevents east-west workload traffic from being forced through the lower-bandwidth north-south path.

It also creates a common policy framework covering both directions of traffic.

BlueField Astra Extends Policy Enforcement

#

BlueField Astra further connects Scale-In and Scale-Out policy management.

BlueField-4 can install and update policies while collecting telemetry. ConnectX-9 can then enforce those policies directly in the network data path.

The result is a closed-loop infrastructure model in which policy configuration, enforcement, and monitoring operate together without requiring tenant software to directly manage networking hardware.

🧩 DOCA Turns BlueField-4 Into a Programmable Infrastructure Platform

#

Hardware acceleration alone is not enough. NVIDIA’s software layer, DOCA, provides the programming and orchestration framework that turns BlueField-4 into a general infrastructure platform.

Important components include:

• DOCA Microservices: Containerized infrastructure services that run directly on BlueField DPUs.
• DOCA Flow: Programs hardware packet-processing pipelines.
• DOCA PCC: Provides programmable congestion-control capabilities.
• DOCA Telemetry: Exposes infrastructure health and performance information.
• DOCA Platform Framework (DPF): Handles DPU configuration, deployment, lifecycle management, and updates.

BlueField-4 also supports Service Function Chaining, allowing traffic to pass through multiple infrastructure services according to policy.

For example, a traffic flow could be classified, inspected, encrypted, routed, and monitored through a programmable sequence without requiring every function to run independently on the host CPU.

This gives NVIDIA a unified software model for deploying infrastructure services throughout an AI factory.

🛣️ Spectrum-X: The Network Behind Scale-In

#

BlueField-4 handles infrastructure processing at the server, but Scale-In also depends on the network connecting AI factories to users, applications, storage systems, and enterprise data.

That is where NVIDIA Spectrum-X Ethernet enters the architecture.

Large AI factories generate enormous numbers of simultaneous traffic flows. Conventional Ethernet can experience congestion, packet loss, and unpredictable latency when storage and compute workloads compete for the same resources.

Spectrum-X is designed to provide more predictable behavior through mechanisms such as intelligent load balancing and congestion management.

NVIDIA says the combination of BlueField-4 and Spectrum-X can deliver up to 1.45× higher storage throughput compared with standard Ethernet in its targeted storage architecture.

The objective is not simply higher peak bandwidth. It is maintaining predictable performance when thousands of concurrent AI workloads compete for infrastructure resources.

🏗️ Five Major Scale-In Use Cases

#

Isolated AI Factory VPCs

#

Multi-tenant AI infrastructure requires strong isolation between customers and workloads.

Scale-In can build virtual private cloud environments on shared physical infrastructure by combining:

• DOCA Host-Based Networking (HBN) for accelerated Layer 3 routing
• DOCA Flow for traffic classification and access control
• OVS-DOCA for east-west policy enforcement
• BlueField Astra for extending VPC policies across east-west traffic

This allows a common policy model to control ingress, egress, and lateral traffic while reducing networking overhead on tenant hosts.

Security Enforcement in Hardware

#

One of Scale-In’s most important ideas is moving security enforcement outside the host operating system.

BlueField-4 can provide infrastructure security controls independently of tenant software, reducing the ability of compromised or misconfigured workloads to interfere with enforcement mechanisms.

Relevant NVIDIA technologies include:

• DOCA Argus for runtime threat detection
• DOCA Vault for file-access policy management
• DOCA Flow for line-rate network policy enforcement
• BlueField Astra for coordinating encryption and isolation policies

Because these functions execute on dedicated infrastructure hardware, they reduce the CPU resources consumed by security processing.

Accelerating External Storage

#

AI factories depend on enormous external data stores containing training datasets, model files, enterprise knowledge, and application data.

If storage access becomes a bottleneck, expensive GPUs can remain idle while waiting for data.

BlueField-4 is designed to accelerate:

• NVMe over Fabrics
• File storage protocols
• Object storage
• RDMA
• TCP-based data movement
• Storage virtualization

Spectrum-X then provides congestion management and performance isolation across the network.

NVIDIA’s reported up to 1.45× storage-throughput improvement illustrates the intended benefit of combining endpoint acceleration with network optimization.

This function complements Context Memory: CMX focuses on reusable inference state and shared KV-cache data inside the AI factory, while Scale-In connects compute resources to external data sources.

Operating the AI Factory Control Plane

#

An AI factory must be configured before it can execute workloads.

Servers need to be discovered, authenticated, provisioned, connected to networks and storage, and loaded with their operating environments.

BlueField-4 can perform many of these tasks independently of the host CPU.

The DPU can participate in:

• Node onboarding
• Security provisioning
• Network configuration
• Storage provisioning
• Remote server boot
• Host OS deployment
• Infrastructure policy installation

DOCA Platform Framework (DPF) extends this model into Kubernetes environments by treating DPUs as manageable infrastructure nodes.

The result is a more centralized approach to deployment and lifecycle management while preserving host resources for AI computation.

Observing and Optimizing AI Operations

#

Large AI factories require continuous visibility into network traffic, storage performance, service health, and infrastructure utilization.

DOCA Telemetry collects operational information from infrastructure devices and services without requiring tenant software to provide the data.

When combined with GPU and network telemetry, operators can identify problems such as:

• Storage bottlenecks
• Network congestion
• Poor workload placement
• Policy-processing overhead
• Resource underutilization
• Infrastructure service failures

This turns Scale-In into an operational feedback system rather than simply a networking offload mechanism.

🧠 Scale-In vs. the Other AI Factory Pillars

#

The five pillars address different layers of AI infrastructure.

Pillar	Primary Problem Solved	Main Infrastructure Focus
Scale-Up	Communication inside a compute node	GPU-to-GPU interconnect
Scale-Out	Distributed AI workloads	Cluster networking
Scale-Across	Geographic expansion	Data-center-to-data-center connectivity
Context Memory	Reusing inference state	KV-cache and context access
Scale-In	Infrastructure overhead	Security, storage, networking, and management

This makes Scale-In complementary rather than competitive with the other four pillars.

Its purpose is to ensure that supporting infrastructure scales alongside compute instead of becoming the limiting factor.

🚀 Why Scale-In Matters for Agentic AI

#

The fundamental change is workload complexity.

A traditional application might generate a relatively predictable sequence of compute and storage operations. An autonomous AI agent can dynamically create new operations depending on what it discovers during execution.

One request can therefore trigger an unpredictable chain of model inference, database queries, API calls, storage accesses, policy evaluations, and network transfers.

At small scale, conventional infrastructure can absorb this overhead.

At AI-factory scale, however, even a small amount of infrastructure overhead multiplied across thousands of GPUs can become a major performance and cost problem.

Scale-In attempts to solve this through hardware offload, programmable infrastructure services, and centralized policy management.

📌 Conclusion: Infrastructure Must Scale With Intelligence

#

The core idea behind NVIDIA’s Scale-In architecture is straightforward: AI compute cannot scale efficiently if the infrastructure surrounding it remains dependent on general-purpose host CPUs.

BlueField-4 provides dedicated infrastructure compute and acceleration. DOCA turns those capabilities into programmable services. Spectrum-X provides the network foundation for high-bandwidth, predictable connectivity.

Together, they move networking, security, storage, management, and telemetry away from the host and into a dedicated infrastructure layer.

NVIDIA’s broader AI Factory strategy therefore goes beyond simply adding more GPUs.

Scale-Up makes individual systems faster. Scale-Out connects more systems. Scale-Across connects more data centers. Context Memory makes inference state easier to reuse. And Scale-In ensures that the infrastructure surrounding all that compute can keep pace.

The larger the AI factory becomes, the more important that final layer becomes. Without it, adding more GPUs may simply produce a larger infrastructure bottleneck rather than a faster AI factory.