diff --git a/enable_capi_snap/bk_main.xml b/enable_capi_snap/bk_main.xml index 2617392..861fbf3 100644 --- a/enable_capi_snap/bk_main.xml +++ b/enable_capi_snap/bk_main.xml @@ -32,7 +32,7 @@ xml:id="bk_main"> - Enable new FPGA card for CAPI2.0 + Enable new FPGA cards for CAPI2.0 BSP and SNAP diff --git a/enable_capi_snap/ch_capi20_bsp.xml b/enable_capi_snap/ch_capi20_bsp.xml index f3f5d8d..0da338d 100644 --- a/enable_capi_snap/ch_capi20_bsp.xml +++ b/enable_capi_snap/ch_capi20_bsp.xml @@ -116,14 +116,14 @@ Choose an existing card as a base. Copy the folder to your new FPGACARD name. Then modify the contents according to the information collected from FPGA supplier. Make sure the information in xdc/tcl files are permitted to be open-source. There are some other modifications you should pay attention to: - + Send email to OpenPower Acceleration Workgroup or contact your representative to apply for a subsystem device ID for the new card. For example, ADKU3 uses 0x0605. S241 uses 0x0660. The information needs to be filled in "[FPGACARD]/tcl/create_ip.tcl", CONFIG.PF0_SUBSYSTEM_ID As a CAPI device, you need to make sure PF0 (physical function) has PF0_DEVICE_ID=0477 and PF0_SUBSYSTEM_VENDOR_ID=1014. This is required by the linux kernel module cxl (pci.c), otherwise the card will not be recognized as a CAPI card by system. If you are using Xilinx VU33P or VU37P who have HBM, this is actually a new FPGA family "virtexuplushbm". Or if you are using other new FPGA Production family, additional steps need to take: - + capi2-bsp/psl/create_ip.tcl: "set_property supported_families ...", add new family name like "virtexuplushbm Production" capi2-bsp/common/tcl/create_capi_bsp.tcl: "set_property supported_families ...", do the same as above. Add family support to PSL9 ZIP package: unzip the package, do the hacking, and zip them back again. @@ -132,21 +132,30 @@ zip -r ibm.com_CAPI_PSL9_WRAP_2.00.zip component.xml src/ xgui/ rm -fr component.xml src/ xgui/ - + + + + VSEC starting address: VSEC (Vendor Specific Extended Capability Structure) is a part of PCIe capability list architecture. It needs to be properly linked in PCIe config space. File "capi2-bsp/[FPGACARD]/src/capi_vsec.vhdl, vsec_addr[21:32] defines the address for VSEC. It should be matched with PCIe core value PF0_SECONDARY_PCIE_CAP_NEXTPTR. Take card U200 for example, its vsec_addr[21:32] starts from 12'h400 (12'b0100_0000_0000), and there is a patch_ip.tcl to modify the pcie4 core's default value 12'h480 to 12'h400. (Actually this is a historical workaround, because Xilinx had modified its extended capability starting address from 12'h400 to 12'h480 on one version. The patch is "tcl/patch_ip.tcl" + exec /bin/bash -c "sed -i \"s/PF0_SECONDARY_PCIE_CAP_NEXTPTR=0x480/PF0_SECONDARY_PCIE_CAP_NEXTPTR=0x400/\" $pcie_source" +exec /bin/bash -c "sed -i \"s/PF0_SECONDARY_PCIE_CAP_NEXTPTR('H480)/PF0_SECONDARY_PCIE_CAP_NEXTPTR('H400)/\" $pcie_source" + About Xilinx PCIe code information for extended configuration space, you can find it on PG156 (for Ultrascale Device) or PG213 (for Ultrascale+ Device). + For Ultrascale+ HBM device, pcie4c core, the VSEC starts from 12'hE80. At this time vsec_addr must be changed in capi_vsec.vhdl. And the above two lines in patch_ip.tcl should be disabled. + - Source files under "capi2-bsp/[FPGACARD]/src": capi_xilmltbt.vhdl. Edit the User image starting address "wbstart_addr". - wbstart_addr <= "User_image_address" when (cpld_user_bs_req = '1') else "00000000000000000000000000000000"; - capi_xilmltbt.vhdl has a Xilinx multi-boot core. That means you can create two kinds of images: Factory image and User image. Factory images will be placed at address 0 of FPGA Flash, and User image will be placed at "User_image_address" on the flash. When power-on or the FPGA card is reset, the multiboot core knows where to load the image. Usually we put a Golden factory image on address 0 and never change it, and multiboot core always tries to load user image first. If the user image has something wrong, multiboot logic will tell the FPGA to "fallback" to factory image. You still see the card in the system and you can just program a new user image to try again. + Vital Product Data: Source files under "capi2-bsp/[FPGACARD]/src": capi_vsec.vhdl. This step is optional. Edit the hardcoded "vpd44data_be" to add VPD (Vital Product Data) information. Ideally this information should be read from an I2C EEPROM. The FPGA supplier wrote the content of EEPROM before shipping. However, today we take the simpliest way to write some hard coded value. "capi2-bsp/common/script" has a script "gen_vsec.sh" to help you do this. + - Source files under "capi2-bsp/[FPGACARD]/src": capi_vsec.vhdl. This step is optional. Edit the hardcoded "vpd44data_be" to add VPD (Vital Product Data) information. Ideally this information should be read from an I2C EEPROM. The FPGA supplier wrote the content of EEPROM before shipping. However, today we take the simpliest way to write some hard coded value. "capi2-bsp/common/script" has a script "gen_vsec.sh" to help you do this. + User Image Address: Source files under "capi2-bsp/[FPGACARD]/src": capi_xilmltbt.vhdl. Edit the User image starting address "wbstart_addr". + wbstart_addr <= "User_image_address" when (cpld_user_bs_req = '1') else "00000000000000000000000000000000"; + capi_xilmltbt.vhdl has a Xilinx multi-boot core. That means you can create two kinds of images: Factory image and User image. Factory images will be placed at address 0 of FPGA Flash, and User image will be placed at "User_image_address" on the flash. When power-on or the FPGA card is reset, the multiboot core knows where to load the image. Usually we put a Golden factory image on address 0 and never change it, and multiboot core always tries to load user image first. If the user image has something wrong, multiboot logic will tell the FPGA to "fallback" to factory image. You still see the card in the system and you can just program a new user image to try again. Check Vivado Version. Make sure this version of Vivado tool supports the FPGA part name you have assigned in "capi2-bsp/[FPGACARD]/Makefile". For some very new FPGA chip types, in one Vivado version they may have a suffix of "es" (engineering sample), and in a newer Vivado version the "es" suffix is removed. - +
Generate capi_bsp_wrap cd capi2-bsp diff --git a/enable_capi_snap/ch_capi20_snap.xml b/enable_capi_snap/ch_capi20_snap.xml index c73da36..f5a97ae 100644 --- a/enable_capi_snap/ch_capi20_snap.xml +++ b/enable_capi_snap/ch_capi20_snap.xml @@ -289,20 +289,29 @@ git submodule update When you download and install Vivado Lab, please pick up as same version as the Vivado (SDx) that you are using to build images. - - Seeing 0477 by "lspci" is the most important milestone. If not, please do following checking: - - Check dmesg. Run "dmesg > dmesg.log" and search "cxl" in dmesg.log file. - Check file "/sys/firmware/opal/msglog" to see whether there are link training failed messages. A successful message looks like this, which means this PCIe device has been scanned and recognized. The number followed "PHB#" is the PCIe device identifier in the format of "domain:bus:slot.func". You can see it by "lspci" also. - [ 63.403485191,5] PHB#0000:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..01 SLOT=CPU1 Slot2 (16x) + Tips to help you debug: + + Seeing 0477 by "lspci" is the most important milestone. If not, please check file "/sys/firmware/opal/msglog" to see whether there are link training failed messages. A successful message looks like this, which means this PCIe device has been scanned and recognized. The number followed "PHB#" is the PCIe device identifier in the format of "domain:bus:slot.func". You can see it by "lspci" also.: +[ 63.403485191,5] PHB#0000:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..01 SLOT=CPU1 Slot2 (16x) [ 63.403572553,5] PHB#0000:01:00.0 [EP ] 1014 0477 R:02 C:1200ff ( device) LOC_CODE=CPU1 Slot2 (16x) - - Check create_ip.tcl in capi2-bsp/[FPGACARD]/tcl and check the configuration of PCIHIP core. - + + Check dmesg. Run "dmesg > dmesg.log" and search "cxl" in dmesg.log file. A normal output should be look like this + [ 9.301403] cxl-pci 0000:01:00.0: Device uses a PSL9 +[ 9.301523] cxl-pci 0000:01:00.0: enabling device (0140 -> 0142) +[ 9.303327] cxl-pci 0000:01:00.0: PCI host bridge to bus 0006:00 +[ 9.306749] cxl afu0.0: Activating AFU directed mode + + Today most of the linux kernel versions already include cxl module. If you doubt this, please check by + modinfo cxl + + + Check create_ip.tcl in capi2-bsp/[FPGACARD]/tcl and check the configuration of PCIe core. If your PCIe device has been recognized as CAPI, do "ls /dev/cxl" and you can see "afu*" devices. Then your application software can open the device like operating an ordinary file. + ls /dev/cxl +afu0.0m afu0.0s Some other useful commands to check PCIe config (with the right PCIe identifier "domain:bus:slot.func") sudo lspci -s 0000:01:00.0 -vvv - For example, you can check the settings coded in Xilinx PCIHIP core, like SUBSYSTEM_ID: + For example, you can check the settings coded in Xilinx PCIe core, like SUBSYSTEM_ID: 0000:01:00.0 Processing accelerators: IBM Device 0477 (rev 02) (prog-if ff) Subsystem: IBM Device 0660 Link Speed @@ -322,6 +331,48 @@ End Capabilities: [400 v1] Vendor Specific Information: ID=1280 Rev=0 Len=080 <?> Kernel driver in use: cxl-pci Kernel modules: cxl + + + If nothing shows by "ls /dec/cxl", you should check PCIe config space by + sudo hexdump /sys/bus/pci/devices/0000\:00\:00.1/config + + Please change the PCIe device identifier (0000:00:00.1) accordingly. Make sure you have seen the VSEC is properly linked. If not, go back to check your VSEC address in capi_vsec.vhdl in last chapter. + 0000000 1014 0477 0146 0010 ff02 1200 0000 0000 +0000010 000c 0000 2200 0006 000c 1000 2200 0006 +0000020 000c 0000 0000 0002 0000 0000 1014 0668 +0000030 0000 0000 0040 0000 0000 0000 00ff 0000 +0000040 4801 0003 0008 0000 7005 0080 0000 0000 +0000050 0000 0000 0000 0000 0000 0000 0000 0000 +0000060 7011 0000 0000 0000 0000 0000 0000 0000 +0000070 b010 0002 8022 0000 2950 0000 f103 0043 +0000080 0000 1103 0000 0000 0000 0000 0000 0000 +0000090 0000 0000 0016 0000 0010 0000 000e 0000 +00000a0 0003 001e 0000 0000 0000 0000 0000 0000 +00000b0 0003 0000 2082 5300 0000 0000 0000 0000 +00000c0 0000 0000 0000 0000 0000 0000 0000 0000 +* +0000100 0001 1c01 0000 0000 0000 0044 2030 0046 --> next_ptr: 1c0 +0000110 0000 0000 e000 0000 0000 0000 0000 0000 +0000120 0000 0000 0000 0000 0000 0000 0000 0000 +* +00001c0 0019 1f01 0000 0000 0000 0000 0000 0000 --> next_ptr: 1f0 +00001d0 0000 0000 0000 0000 0000 0000 0000 0000 +* +00001f0 0002 e801 0000 0000 0000 3100 0000 0000 --> e80 (or 400) points to VSEC +0000200 0000 0000 00ff 8000 0000 0000 0000 0000 +0000210 0000 0000 0000 0000 0000 0000 0000 0000 +* +0000e80 000b 0001 1280 0800 0801 0021 0006 0200 --> VSEC; e80 matches here +0000e90 0000 b000 0000 0000 0000 0000 0000 0000 +0000ea0 0100 0000 0040 0000 0200 0000 0400 0000 +0000eb0 0000 0000 0000 0000 0000 0000 0000 0000 +* +0000ed0 0000 0000 0000 0000 0000 8000 0000 0000 +0000ee0 0000 0000 0000 0000 0000 0000 0000 0000 +* +0001000 + +
Stage 2: Verify Flash interface @@ -358,8 +409,12 @@ Kernel modules: cxl
Cleanup and submit Now a new FPGA card has been enabled to CAPI2.0 SNAP. Cleanup your workspace, check files and submit your work! - capi-utils is independent. Just create a pull request and assign a reviewer. It can only been merged into master branch after having been reviewed. - Submit the pull request of your "capi2-bsp fork" before "snap fork". After capi2-bsp is merged intohttps://github.com/open-power/capi2-bsp master branch, update the submodule pointer to the latest "open-power/capi2-bsp" master and then submit the pull request of your forked snap. + + capi-utils is independent. Just create a pull request and assign a reviewer. It can only been merged into master branch after having been reviewed. + Submit the pull request of your "capi2-bsp fork" before "snap fork". Assign the reviewer and wait capi2-bsp to be merged into https://github.com/open-power/capi2-bsp master branch + Update the submodule pointer to the latest "open-power/capi2-bsp" master and then submit the pull request of your forked snap. + +