Minimal required:

backplane:
  redisUri: "redis://localhost:6379"
  queues:
    - name: "cpu"
      properties:
        - name: "min-cores"
          value: "*"
        - name: "max-cores"
          value: "*"
worker:
  publicName: "localhost:8981"

Configuration files also support includes via !include tag:

backplane: !include "backplane.yml"
server: !include "server.yml"
worker: !include "worker.yml"

The configuration can be provided to the server and worker as a CLI argument or through the environment variable CONFIG_PATH For an example configuration containing all of the configuration values, see examples/config.yml.

All Configurations

Common

Example:

digestFunction: SHA1
defaultActionTimeout: 1800
maximumActionTimeout: 1800
prometheusPort: 9090
server:
  ...
worker:
  ...

Server

Example:

server:
  instanceType: SHARD
  name: shard
  actionCacheReadOnly: true
  recordBesEvents: true

GRPC Metrics

Example:

server:
  grpcMetrics:
    enabled: false
    provideLatencyHistograms: false
    labelsToReport: []

Server Caches

Example:

server:
  caches:
    directoryCacheMaxEntries: 10000
    commandCacheMaxEntries: 10000
    digestToActionCacheMaxEntries: 10000
    recentServedExecutionsCacheMaxEntries: 10000

Admin

Example:

server:
  admin:
    deploymentEnvironment: AWS
    clusterEndpoint: "grpc://localhost"

Metrics

Example:

server:
  metrics:
    publisher: log
    logLevel: INFO

server:
  metrics:
    publisher: aws
    topic: buildfarm-metadata-test
    topicMaxConnections: 1000
    secretName: buildfarm-secret

Correlated Invocations Index Scopes

A set of names to extract per-correlatedInvocationsId fields from a valid URI to create indices which will contain the fragment uuid

Example:

server:
  correlatedInvocationsIndexScopes: !!set
    ? host
    ? username

Redis Backplane

Example:

backplane:
  type: SHARD
  redisUri: "redis://localhost:6379"
  priorityQueue: true

Execution Queues

Example:

backplane:
  type: SHARD
  redisUri: "redis://localhost:6379"
  queues:
    - name: "cpu"
      allowUnmatched: true
      properties:
        - name: "min-cores"
          value: "*"
        - name: "max-cores"
          value: "*"

Worker

worker:
  port: 8981
  publicName: "localhost:8981"
  linkedInputDirectories:
    - "^path/to/common/directory"

Capabilities

Example:

worker:
  capabilities:
    cas: true
    execution: true

Sandbox Settings

Using the sandbox can be configurable by the client via exec_properties. However, sometimes it is preferred to enable it via buildfarm config to prevent users from running actions outside the sandbox.

Example:

worker:
  sandboxSettings:
    alwaysUseSandbox: true
    alwaysUseAsNobody: false
    alwaysUseCgroups: true
    alwaysUseTmpFs: true
    additionalWritePaths: []
    tmpFsPaths: []
    selectForBlockNetwork: false
    selectForTmpFs: false

Note: In order for these settings to take effect, you must also configure limitGlobalExecution: true.

Dequeue Match

Example:

worker:
  dequeueMatchSettings:
    allowUnmatched: false
    properties:
      - name: "gpu"
        value: "nvidia RTX 2090"

Resources

A list of limited resources that are available to the worker to be depleted by actions which execute containing a “resource:": "N" property. The dequeueMatchSettings may also further limit executions to contain "resource:" in properties, with either a specific limited resource count as the only accepted value for the action property. The use case here is one where executions are not allowed to request any value other than the one specified. There are no operators currently for asserting an execution requests 'less than' a particular number of resources.

The default resource type is SEMAPHORE. The expected use case is that a resource is internally allocated and managed by an execution, and the exhaustion prevents executions from starting which would block or fault if they could not consume the resource. Common examples include licensed software with tokens, and this mechanism can work for singleton resources like one gpu on a worker.

The POOL resource type is intended for externally defined resource allotment. The Integer range [0,amount) will be created in a pool of free ids provided to The common use case here is multiple gpus on a system, where the concurrent execution on all gpus can take place, but each execution completion frees a different gpu that must be targetted with, for example, the CUDA_VISIBLE_DEVICES env var. The pool takes a resource out of a queue of available ids, and replaces it in the queue when the execution is complete. Since externally defined resources need to be injected into an execution, the depletion of a pool resource applies an automatic execution policy, named ‘pool-', and supplies transformations for both '<_name_>' - expanded to all ids in multiple arguments, or '<_name_-_N_>' for 0-N addressable ids, up to the amount requested. Subsequent literal arguments can provide a delimiter for the wrapper.

Example:

worker:
  dequeueMatchSettings:
    properties:
      - name: "resource:special-compiler-license"
        value: "1" # only actions which request one compiler license at a time will be accepted
  resources:
    name: "special-compiler-license"
    amount: 3

Worker CAS

Unless specified, options are only relevant for FILESYSTEM type

Example:

This definition will create a filesystem-based CAS file cache at the path “/cache" on the worker that will reject entries over 2GiB in size, and will expire LRU blobs when the aggregate size of all blobs exceeds 2GiB in order to insert additional entries.

worker:
  storages:
    - type: FILESYSTEM
      path: "cache"
      maxSizeBytes: 2147483648 # 2 * 1024 * 1024 * 1024

This definition elides FILESYSTEM configuration with ‘…’, will read-through an external GRPC CAS supporting the REAPI CAS Services into its storage, and will attempt to write expiring entries into the GRPC CAS (i.e. pushing new entries into the head of a worker LRU list will drop the entries from the tail into the GRPC CAS).

worker:
  storages:
    - type: FILESYSTEM
      ...
    - type: GRPC
      target: "cas.external.com:1234"

Execution Policies

Example:

worker:
  executionPolicies:
    - name: as-nobody
      prioritized: true
      executionWrapper:
        path: /app/build_buildfarm/as-nobody
        arguments:
          - "-u"
          - "<exec-owner>"
    - name: unshare
      executionWrapper:
        path: /usr/bin/unshare
        arguments:
          - "-n"
          - "-r"
    - name: linux-sandbox
      executionWrapper:
        path: /app/build_buildfarm/linux-sandbox
        arguments:
          # use "--" to signal the end of linux-sandbox args. "--" should always be last!
          - "--"
    - name: test
      executionWrapper:
        path: /YOUR/WRAPPER
        arguments:
          - arg1
          - arg2
          - "<platform-property-name>"

arg1 and arg2 are interpreted literally. <platform-property-value> will be substituted with the value of a property named "platform-property-name" from a Command’s Platform or the requested pool resources for the execution. If a matching property or pool resource is not found for a specified name, the entire wrapper will be discarded and have no effect on the execution.

<exec-owner> is an automatically provided pool resource when execOwner or execOwners is specified, and will contain the value of the execution’s owner selected for exec tree creation.

An execution with as-nobody, unshare, and linux-sandbox execution policies enabled would produce a command line like:

/app/build_buildfarm/as-nobody -u <exec-owner> /usr/bin/unshare -n -r /app/build_buildfarm/linux-sandbox -- /YOUR/WRAPPER arg1 arg2 <platform-property-name> ACTION

where ACTION is the Command from remote execution action.

Table of contents

• Authentication