TECH : Building with the Network

Dominic Robinson
ARTICLE BY:
POSTED:

TAGS: Build, SN-DBS

English writer John Heywood coined the phrase "Many hands make light work". We've applied this proverb to game development, replaced 'hands' with 'computers', and came up with SN-DBS, our distributed build system.

SN-DBS significantly reduces compile times for games console development projects, by sharing the work between computers on a network.

SN-DBS overview diagram

Each computer with SN-DBS installed is called an Agent. There can be many Agents, but only one Broker that coordinates all Agents.

SN-DBS runs in the background with minimal impact on the normal performance of the host computer. The Broker distributes builds within the SN-DBS network. Each Agent can initiate builds, in addition to offering its own processing power for use by others.

How it works

The complete set of source files, header files, and any compiler tools are marked as dependency files for the job, and a description of the job is sent to an Agent to process.

When a job is scheduled on an Agent, the Agent requests any files missing from its cache. When the Agent has received all necessary files, it performs the compilation and returns any resulting files back to the originating client. SN-DBS receives the compiled file and places it into the file system at the expected location. Job done!

Performance

SN-DBS can drastically reduce overall build times for games console development projects. The optimized job scheduler pipelines jobs to all available CPU cores on each Agent. Files are also cached to minimize network traffic.

The actual results will depend on the number of computers in the SN-DBS network, we've got some customers with over 200 Agents enabled for distributed builds. The table below shows a fairly typical scenario.

Table 1. Local build vs. SN-DBS Build

Build TypeNumber of CoresTime Taken
Local 4 5 minutes
SN-DBS 100 26 seconds

SN-DBS is designed to help keep game developers doing what they love to do, and not waiting for builds to complete!

Back to top