Each card supplier may design their FPGA board with different FPGA chips, circuit components, memory and IOs, so BSP (Board support package) is different from card to card. That's why an open-sourced project is so helpful: It allows card supplier and every developer to explore the functions of the card freely, and get benefits from CAPI technology.

CAPI2.0 BSP (capi_bsp_wrap) contains following modules: PSL9 (PSL9_WRAP) PCIe hard IP core (pcie3/4_ultrascale_plus_0) Flash Controller (capi_flash_spi_mt25qt) VSEC: Vendor Specific Extended Capability (capi_vsec) Xilinx MultiBoot control logic (capi_xilmltbt) Other miscelleneous logic In this diagram, psl_fpga.vhdl is the top design. How to connect the ports to FPGA pins is different from card to card, and the information is provided by Card Supplier. They may include: Flash interface (usually SPIx4 or dual SPIx4) PCIe interface: perst, refclk, TX and RX data lanes (PCIe Gen3.0x16, or Gen4.0x8) Peripheral IPs: I2C, LED, DDR, Ethernet, etc. At least Flash interface pins and PCIe interface pins are required to be assigned in xdc files precisely. Between capi_bsp_wrap and user AFU (psl_accel), there are 6 groups of signals: Command, DMA, Buffer, Response, MMIO and Control. Please refer to CAPI2.0 PSL/AFU interface Spec for the details. The logic in CAPI2.0 snap_core implemented the data path with DMA interface. Buffer interface is not used.

capi2-bsp is a public Github repository. You need to have a Github account first. Then create a "fork" (Click the "fork" button) on https://github.com/open-power/capi2-bsp. git clone https://github.com/[YOUR_USERNAME]/capi2-bsp Actually capi2-bsp is a submodule of snap. That will be introduced later. Keep working on your own capi2-bsp fork, when it has been validated to work well, submit a pull request to "open-power/capi2-bsp" and require merging into the public upstream.

First, define a FPGACARD name. It can start from the company's name, followed with the card name and be short. For example. ADKU3 = Alpha-Data ADM-PCIE-KU3. Get information from the card supplier. Item Description FPGACARD Short card name FPGACHIP FPGA part name, for example, xcvu9p-fsgd2104-2L-e Flash Type and IO pins Flash chip that attached to FPGA, for example mt28gu01gaax1e-bpi-x16. And the related xdc files for FPGA config. PCIe Config and IO pins The tcl/xdc information about the PCIe hardware IP for this card. DDR MC IP DDR memory controller Vivado IP tcl/xdc file. Other peripherals NVMe IP, Ethernet IP and so on (Optional) DDR MC and other peripheral IP's configurations are not needed at the beginning. They are not in the capi_bsp_wrap diagram. However, when you query information from the FPGA Supplier, it's wise to include them also.

Download PSL9 Zip package from OpenPower Portal and put it in "capi2-bsp/psl" directory.

Choose an existing card as a base. Copy the folder to your new FPGACARD name. Then modify the contents according to the information collected from FPGA supplier. Make sure the information in xdc/tcl files are permitted to be open-source. There are some other modifications you should pay attention to: Send email to OpenPower Acceleration Workgroup or contact your representative to apply for a subsystem device ID for the new card. For example, ADKU3 uses 0x0605. S241 uses 0x0660. The information needs to be filled in "[FPGACARD]/tcl/create_ip.tcl", CONFIG.PF0_SUBSYSTEM_ID As a CAPI device, you need to make sure PF0 (physical function) has PF0_DEVICE_ID=0477 and PF0_SUBSYSTEM_VENDOR_ID=1014. This is required by the linux kernel module cxl (pci.c), otherwise the card will not be recognized as a CAPI card by system. If you are using Xilinx VU33P or VU37P who have HBM, this is actually a new FPGA family "virtexuplushbm". Or if you are using other new FPGA Production family, additional steps need to take: capi2-bsp/psl/create_ip.tcl: "set_property supported_families ...", add new family name like "virtexuplushbm Production" capi2-bsp/common/tcl/create_capi_bsp.tcl: "set_property supported_families ...", do the same as above. Add family support to PSL9 ZIP package: unzip the package, do the hacking, and zip them back again. unzip ibm.com_CAPI_PSL9_WRAP_2.00.zip (modify compnent.xml to add new family name, search "supportedFamilies") zip -r ibm.com_CAPI_PSL9_WRAP_2.00.zip component.xml src/ xgui/ rm -fr component.xml src/ xgui/ VSEC starting address: VSEC (Vendor Specific Extended Capability Structure) is a part of PCIe capability list architecture. It needs to be properly linked in PCIe config space. File "capi2-bsp/[FPGACARD]/src/capi_vsec.vhdl, vsec_addr[21:32] defines the address for VSEC. It should be matched with PCIe core value PF0_SECONDARY_PCIE_CAP_NEXTPTR. Take card U200 for example, its vsec_addr[21:32] starts from 12'h400 (12'b0100_0000_0000), and there is a patch_ip.tcl to modify the pcie4 core's default value 12'h480 to 12'h400. (Actually this is a historical workaround, because Xilinx had modified its extended capability starting address from 12'h400 to 12'h480 on one version. The patch is "tcl/patch_ip.tcl" exec /bin/bash -c "sed -i \"s/PF0_SECONDARY_PCIE_CAP_NEXTPTR=0x480/PF0_SECONDARY_PCIE_CAP_NEXTPTR=0x400/\" $pcie_source" exec /bin/bash -c "sed -i \"s/PF0_SECONDARY_PCIE_CAP_NEXTPTR('H480)/PF0_SECONDARY_PCIE_CAP_NEXTPTR('H400)/\" $pcie_source" About Xilinx PCIe code information for extended configuration space, you can find it on PG156 (for Ultrascale Device) or PG213 (for Ultrascale+ Device). For Ultrascale+ HBM device, pcie4c core, the VSEC starts from 12'hE80. At this time vsec_addr must be changed in capi_vsec.vhdl. And the above two lines in patch_ip.tcl should be disabled. Vital Product Data: Source files under "capi2-bsp/[FPGACARD]/src": capi_vsec.vhdl. This step is optional. Edit the hardcoded "vpd44data_be" to add VPD (Vital Product Data) information. Ideally this information should be read from an I2C EEPROM. The FPGA supplier wrote the content of EEPROM before shipping. However, today we take the simpliest way to write some hard coded value. "capi2-bsp/common/script" has a script "gen_vsec.sh" to help you do this. User Image Address: Source files under "capi2-bsp/[FPGACARD]/src": capi_xilmltbt.vhdl. Edit the User image starting address "wbstart_addr". wbstart_addr <= "User_image_address" when (cpld_user_bs_req = '1') else "00000000000000000000000000000000"; capi_xilmltbt.vhdl has a Xilinx multi-boot core. That means you can create two kinds of images: Factory image and User image. Factory images will be placed at address 0 of FPGA Flash, and User image will be placed at "User_image_address" on the flash. When power-on or the FPGA card is reset, the multiboot core knows where to load the image. Usually we put a Golden factory image on address 0 and never change it, and multiboot core always tries to load user image first. If the user image has something wrong, multiboot logic will tell the FPGA to "fallback" to factory image. You still see the card in the system and you can just program a new user image to try again. Check Vivado Version. Make sure this version of Vivado tool supports the FPGA part name you have assigned in "capi2-bsp/[FPGACARD]/Makefile". For some very new FPGA chip types, in one Vivado version they may have a suffix of "es" (engineering sample), and in a newer Vivado version the "es" suffix is removed.

cd capi2-bsp make [FPGACARD] If it is successfully done, the generation of BSP for CAPI2.0 is completed. For HDK developers, they can create their own Vivado project and import capi_bsp_wrap as an IP. But for SNAP developers, there are some other work to do, see in next chapter.