Each card supplier may design their FPGA board with different FPGA chips, circuit components, memory and IOs, so BSP (Board support package) is different from card to card. That's why an open-sourced project is so helpful: It allows card supplier and every developer to explore the functions of the card freely, and get benefits from CAPI technology.

CAPI2.0 BSP (capi_bsp_wrap) contains following modules: PSL9 (PSL9_WRAP) PCIe hard IP core (pcie3/4_ultrascale_plus_0) Flash Controller (capi_flash_spi_mt25qt) VSEC: Vendor Specific Extended Capability (capi_vsec) Xilinx MultiBoot control logic (capi_xilmltbt) Other miscelleneous logic In this diagram, psl_fpga.vhdl is the top design. How to connect the ports to FPGA pins is different from card to card, and the information is provided by Card Supplier. They may include: Flash interface (usually SPIx4 or dual SPIx4) PCIe interface: perst, refclk, TX and RX data lanes (PCIe Gen3.0x16, or Gen4.0x8) Peripheral IPs: I2C, LED, DDR, Ethernet, etc. At least Flash interface pins and PCIe interface pins are required to be assigned in xdc files precisely. Between capi_bsp_wrap and user AFU (psl_accel), there are 6 groups of signals: Command, DMA, Buffer, Response, MMIO and Control. Please refer to CAPI2.0 PSL/AFU interface Spec for the details.

capi2-bsp is a public Github repository. You need to have a Github account first. Then create a "fork" (Click the "fork" button) on https://github.com/open-power/capi2-bsp. git clone https://github.com/[YOUR_USERNAME]/capi2-bsp capi2-bsp is also a submodule of snap. Keep working on your own capi2-bsp fork, when it has been validated to work well, submit a pull request to "open-power/capi2-bsp" and request merging into the public upstream.

First, define a FPGACARD name. It can start from the company's name, following with the card name and be short. For example, ADKU3 = Alpha-Data ADM-PCIE-KU3. Get information from the card supplier. Item Description FPGACARD Short card name FPGACHIP FPGA part name, for example, xcvu9p-fsgd2104-2L-e Flash Type and IO pins Flash chip that attached to FPGA, for example mt28gu01gaax1e-bpi-x16. And the related xdc files for FPGA config. PCIe Config and IO pins The tcl/xdc information about the PCIe hardware IP for this card. DDR MC IP DDR memory controller Vivado IP tcl/xdc file. Other peripherals NVMe IP, Ethernet IP and so on (Optional) DDR MC and other peripheral IP's configurations are not needed at the beginning. They are not in the capi_bsp_wrap diagram. However, when you query information from the FPGA Supplier, it's wise to include them also.

Download PSL9 Zip package from OpenPower Portal and put it in "capi2-bsp/psl" directory.

Choose an existing card as a base. Copy the folder to your new FPGACARD name. Then modify the contents according to the information collected from FPGA supplier. Make sure the information in xdc/tcl files are permitted to be open-source. There are some other modifications you should pay attention to: PCIe core IP creation: "Vendor ID" and "Device ID" have to be 0x1014 and 0x0477, so kernel module cxl can recognize the card as a CAPI device. (See in pci.c) If the card vendor has a code allocated by PCISIG (See in PCISIG Member companies), use it as "Subsystem Vendor ID". "Subsystem Device ID" can be chosen freely. If the card vendor doesn't have a code allocated by PCISIG, or just for testing and evaluation purpose, please use default "Subsystem Vendor ID" = 0x1014, and send email to aclwg-chair@openpowerfoundation.org to get a distinct "Subsystem Device ID" to differentiate this card from others. Example: (in create_ip.tcl) create_ip -name pcie4_uscale_plus -vendor xilinx.com -library ip -module_name pcie4_uscale_plus_0 -dir $ip_dir >> $log_file set_property -dict [list \ CONFIG.PF0_CLASS_CODE {1200ff} \ CONFIG.PF0_REVISION_ID {02} \ CONFIG.VENDOR_ID {1014} \ CONFIG.PF0_DEVICE_ID {0477} \ CONFIG.PF0_SUBSYSTEM_VENDOR_ID {1014} \ CONFIG.PF0_SUBSYSTEM_ID {0661} \ ...... \ ...... \ ] [get_ips pcie4_uscale_plus_0] >> $log_file The corresponding "Subsytem Vendor ID" and "Subsystem Device ID" need to be added into capi-utils, file "psl-devices". Product Family support: If the FPGA chip types are Xilinx VU33P or VU37P who have HBM, this is actually a new FPGA family virtexuplushbm. For a new FPGA Production family, additional steps need to take: "capi2-bsp/psl/create_ip.tcl": set_property supported_families ..., add new family name like "virtexuplushbm Production" "capi2-bsp/common/tcl/create_capi_bsp.tcl": set_property supported_families ..., do the same as above. Add family support to PSL9 ZIP package: unzip the package, do the modifications, and zip them back again. Commands: $ unzip ibm.com_CAPI_PSL9_WRAP_2.00.zip (modify compnent.xml to add new family name, search "supportedFamilies") $ zip -r ibm.com_CAPI_PSL9_WRAP_2.00.zip component.xml src/ xgui/ $ rm -fr component.xml src/ xgui/ VSEC starting address: VSEC (Vendor Specific Extended Capability Structure) is a part of PCIe capability list architecture. It needs to be properly linked in PCIe config space. "capi2-bsp/[FPGACARD]/src/capi_vsec.vhdl": vsec_addr[21:32] defines the address for VSEC. It should be matched with PCIe core value PF0_SECONDARY_PCIE_CAP_NEXTPTR. Take card U200 for example, its vsec_addr[21:32] starts from 12'h400 (12'b0100_0000_0000), and "tcl/patch_ip.tcl" modifies it from default value 12'h480 to 12'h400." exec /bin/bash -c "sed -i \"s/PF0_SECONDARY_PCIE_CAP_NEXTPTR=0x480/PF0_SECONDARY_PCIE_CAP_NEXTPTR=0x400/\" $pcie_source" exec /bin/bash -c "sed -i \"s/PF0_SECONDARY_PCIE_CAP_NEXTPTR('H480)/PF0_SECONDARY_PCIE_CAP_NEXTPTR('H400)/\" $pcie_source" Xilinx PCIe code information for extended configuration space can be found on PG156 (for Ultrascale Device) or PG213 (for Ultrascale+ Device). For Ultrascale+ HBM device's pcie4c core, the VSEC starts from 12'hE80. At this time vsec_addr[21:32] must be changed in "capi_vsec.vhdl". And the above two lines in "patch_ip.tcl" are not needed anymore. Vital Product Data: This step is optional. "capi2-bsp/[FPGACARD]/src/capi_vsec.vhdl": Edit the hardcoded vpd44data_be to add VPD (Vital Product Data) information. Ideally this information should be read from an I2C EEPROM. The FPGA supplier wrote the content of EEPROM before shipping. Today the simpliest way is taken -- writing some hard coded value. "capi2-bsp/common/script" has a script "gen_vsec.sh" to do this. User Image Address: "capi2-bsp/[FPGACARD]/src/capi_xilmltbt.vhdl": Edit the User image starting address wbstart_addr. wbstart_addr <= "User_image_address" when (cpld_user_bs_req = '1') else "00000000000000000000000000000000"; "capi_xilmltbt.vhdl" has a Xilinx multi-boot core. That means you can create two kinds of images: Factory image and User image. Factory images will be placed at address 0 of FPGA Flash, and User image will be placed at "User_image_address" on the flash. When power-on or the FPGA card is reset, the multiboot core knows where to load the image. Usually a Golden factory image is put on address 0 and is never changed. Multiboot core always tries to load user image first. If the user image has something wrong, multiboot logic will tell the FPGA to "fallback" to factory image. You still see the card in the system and you can just program a new user image to try again. Check Vivado Version: Make sure this version of Vivado tool supports the FPGA part name you have assigned in "capi2-bsp/[FPGACARD]/Makefile". For some very new FPGA chip types, in one Vivado version they may have a suffix of "es" (engineering sample), and in a newer Vivado version the "es" suffix is removed.

cd capi2-bsp make [FPGACARD] If it is successfully done, the generation of BSP for CAPI2.0 is completed. Developers using HDK mode can create their own Vivado project and import capi_bsp_wrap as an IP. But for SNAP developers there are some other work to do, see in next chapter.