LowCbfProcessor

LowCbfProcessor is a Tango device server for monitoring and control of registers in the Low.CBF signal processing FPGAs. Processors are shared by and may perform correlation or beamforming for multiple subarrays simultaneously.

The processor device uses the ska-low-cbf-fpga python package to represent registers within the FPGA design as “fields” and groupings of fields (“peripherals”) it can interact with. The ska-low-cbf-fpga package is a general framework for interacting with registers in Low.CBF FPGA designs. Registers within a FPGA are read from the address map file associated with each FPGA design (fpgamap_NNNNNNNN.py).

The processor device repository contains code to control and monitor two specific FPGA design personalities

  • Low.CBF PST Beamformer FPGAs

  • Low.CBF correlator FPGAs

  • Low.CBF PSS Beamformer FPGAs

Processor communication with Allocator

Processor Tango devices subscribe to Allocator events that represent the desired state of the processor device. On receipt of an event, a processor device will attempt to conform itself to the desired state. It will translate the attribute parameters into appropriate register values for the FPGA design personality. There are two attributes to which a Processor subscribes:

  • “internal_alveo” conveys per-Alveo data

  • “internal_subarray” conveys per-subarray settings that are used globally by Alveos

The Allocator “internal_alveo” attribute conveys information about the internal register settings of every Alveo. The attribute data is a JSON-encoded dictionary, using Alveo serial numbers as the key. If an Alveo card’s serial number is not present then the Alveo is not currently in use by any subarray. Each Alveo uses only the data listed under its own ID. The data includes the FPGA personality the Alveo should run and an abbreviated description of its register settings.

The Allocator’s “internal_subarray” attribute conveys information about the currently operating subarrays, including delay polynomial sources. The attribute data is a JSON-encoded dictionary using Subarray ID [1-16] as key. If a subarray’s key is not present the the subarray is not in use. Values for each subarray provide information about subarray stations, beams, frequencies, and destinations for the Alveo’s output products. The information is potentially used by, and common to, every Alveo.

Delay polynomials

The processor subscribes to tango attributes that provide station-beam delay polynomials (currently a 5th order) for each station contributing to a station beam. It examines time in the incoming SPS packets and chooses polynomials with appropriate start-of-validity to use for delay calculations from an internal queue of polynomias it received The “stats_delay” attribute provides information about delay polynomials in the FPGA registers and whether the polynomials are valid (ie not being used before start-of-validity and not being used after end-of-validity).

Design

Processors load a FPGA executable for the particular Personality (correlator, beamformer, etc) they are to run. The executable is downloaded from SKA CAR or Gitlab. Since the download takes longer time than a Tango command is allowed to execute, firmware download and load is handled by a separate thread, allowing the Tango thread to return before Tango times it out. Because FPGA personality can change at any time, code is designed to receive requests to download a new personality any time, even while a download is in progress.

Register values in an Alveo are only updated when any subarray that the Processor is handling changes between scanning and not-scanning states and when incoming packets contain time that matches more recent delay polynomial validity. Scanning state is derived from “internal_subarray” events, and any event that indicates scanning state has changed cause registers to be reprogrammed immediately.

Data from “internal_alveo” events however is simply recorded for use later, i.e. when scanning state changes. The SKA observing state machine ensures that a subarray receives configuration (affecting “internal_alveo” data) before it begins to scan, so the last recorded version of “internal_alveo” event data always contains the required Alveo register configuration to be applied when an “internal_subarray” event is received.

Tango attribute/command list

Processor device Tango attributes

Processor device Tango commands

Test Mode

If Test mode is active, the value of some attributes can be temporarily changed to any desired value for testing.

Continuous Integration

The current CI tests of this device use DeviceTestContext and do not require a full Tango system.

Environment Variables

Some runtime behaviours could be configured through environment variables.

  • INITIAL_ADMINMODE variable can be used to select the default value of adminMode Tango attribute on startup.

  • ALLOW_ADMIN_CHANGE_WHILE_USED allows adminMode to be altered at any time if set to true. If false (the default), adminMode is unable to be changed unless the FPGA is not in use by any subarray.

  • FPGA_XRT_TIMEOUT allows the timeout (milliseconds) for each FPGA register read or write to be extended (default 5 if not set)

  • FPGA_POST_FW_LOAD_DELAY specifies time in seconds after firmware load before register reads will time out. (Allows for busy server CPU after loading a FPGA)

  • CACHE_DIR allows firmware to be cached on-server, saving download bandwidth. Our Helm chart sets this to /app, meaning downloads will be cached in the pod’s ephemeral storage. For best results, override this to something persistent & shared between pods (e.g. a volume mount on the FPGA host server).

An example Helm values.yaml file configuration would look something like:

environment_variables:

  - name: CACHE_DIR
    value: /test-data/firmware_image_cache/
  - name: FPGA_POST_FW_LOAD_DELAY
    value: 0.1
  - name: FPGA_XRT_TIMEOUT
    value: 15
  - name: INITIAL_ADMINMODE
    value: online
  - name: ALLOW_ADMIN_CHANGE_WHILE_USED
    value: true