<para> Choose an existing card as a base. Copy the folder to your new FPGACARD name. Then modify the contents according to the information collected from FPGA supplier.</para>
<important><para> Make sure the information in xdc/tcl files are permitted to be open-source. </para></important>
<para>There are some other modifications you should pay attention to:</para>
<itemizedlist>
<orderedlist>
<listitem> <para>Send email to OpenPower Acceleration Workgroup or contact your representative to apply for a <emphasis role="bold">subsystem device ID</emphasis> for the new card. For example, ADKU3 uses 0x0605. S241 uses 0x0660. The information needs to be filled in "[FPGACARD]/tcl/create_ip.tcl", <userinput>CONFIG.PF0_SUBSYSTEM_ID</userinput></para>
</listitem>
<listitem> <para> As a CAPI device, you need to make sure PF0 (physical function) has <userinput>PF0_DEVICE_ID=0477</userinput> and <userinput>PF0_SUBSYSTEM_VENDOR_ID=1014</userinput>. This is required by the linux kernel module cxl <link xlink:href="https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/tree/drivers/misc/cxl/pci.c">(pci.c)</link>, otherwise the card will not be recognized as a CAPI card by system. </para>
</listitem>
<listitem>
<para>If you are using Xilinx VU33P or VU37P who have HBM, this is actually a new FPGA family "virtexuplushbm". Or if you are using other new FPGA Production family, additional steps need to take:</para>
<orderedlist>
<itemizedlist>
<listitem><para>capi2-bsp/psl/create_ip.tcl: "set_property supported_families ...", add new family name like "virtexuplushbm Production"</para></listitem>
<listitem><para>capi2-bsp/common/tcl/create_capi_bsp.tcl: "set_property supported_families ...", do the same as above.</para></listitem>
<listitem><para>Add family support to PSL9 ZIP package: unzip the package, do the hacking, and zip them back again.</para>
@ -132,21 +132,30 @@
zip -r ibm.com_CAPI_PSL9_WRAP_2.00.zip component.xml src/ xgui/
rm -fr component.xml src/ xgui/</screen>
</listitem>
</orderedlist>
</itemizedlist>
</listitem>
<listitem><para><emphasis role="bold">VSEC starting address: </emphasis> VSEC (Vendor Specific Extended Capability Structure) is a part of PCIe capability list architecture. It needs to be properly linked in PCIe config space. File "capi2-bsp/[FPGACARD]/src/capi_vsec.vhdl, vsec_addr[21:32] defines the address for VSEC. It should be matched with PCIe core value <emphasis role="bold">PF0_SECONDARY_PCIE_CAP_NEXTPTR</emphasis>. Take card U200 for example, its vsec_addr[21:32] starts from 12'h400 (12'b0100_0000_0000), and there is a patch_ip.tcl to modify the pcie4 core's default value 12'h480 to 12'h400. (Actually this is a historical workaround, because Xilinx had modified its extended capability starting address from 12'h400 to 12'h480 on one version. The patch is "tcl/patch_ip.tcl" </para>
<para>About Xilinx PCIe code information for extended configuration space, you can find it on PG156 (for Ultrascale Device) or PG213 (for Ultrascale+ Device). </para>
<para>For Ultrascale+ HBM device, pcie4c core, the VSEC starts from 12'hE80. At this time vsec_addr must be changed in capi_vsec.vhdl. And the above two lines in patch_ip.tcl should be disabled.</para>
</listitem>
<listitem>
<para>Source files under "capi2-bsp/[FPGACARD]/src": capi_xilmltbt.vhdl. Edit the User image starting address "wbstart_addr". </para>
<screen>wbstart_addr <= "User_image_address" when (cpld_user_bs_req = '1') else "00000000000000000000000000000000";</screen>
<para>capi_xilmltbt.vhdl has a Xilinx multi-boot core. That means you can create two kinds of images: Factory image and User image. Factory images will be placed at address 0 of FPGA Flash, and User image will be placed at "User_image_address" on the flash. When power-on or the FPGA card is reset, the multiboot core knows where to load the image. Usually we put a Golden factory image on address 0 and never change it, and multiboot core always tries to load user image first. If the user image has something wrong, multiboot logic will tell the FPGA to "fallback" to factory image. You still see the card in the system and you can just program a new user image to try again. </para>
<para><emphasis role="bold">Vital Product Data: </emphasis>Source files under "capi2-bsp/[FPGACARD]/src": capi_vsec.vhdl. <emphasis role="bold">This step is optional.</emphasis> Edit the hardcoded "vpd44data_be" to add VPD (Vital Product Data) information. Ideally this information should be read from an I2C EEPROM. The FPGA supplier wrote the content of EEPROM before shipping. However, today we take the simpliest way to write some hard coded value. "capi2-bsp/common/script" has a script "gen_vsec.sh" to help you do this. </para>
</listitem>
<listitem>
<para>Source files under "capi2-bsp/[FPGACARD]/src": capi_vsec.vhdl. <emphasis role="bold">This step is optional.</emphasis> Edit the hardcoded "vpd44data_be" to add VPD (Vital Product Data) information. Ideally this information should be read from an I2C EEPROM. The FPGA supplier wrote the content of EEPROM before shipping. However, today we take the simpliest way to write some hard coded value. "capi2-bsp/common/script" has a script "gen_vsec.sh" to help you do this. </para>
<para><emphasis role="bold">User Image Address: </emphasis>Source files under "capi2-bsp/[FPGACARD]/src": capi_xilmltbt.vhdl. Edit the User image starting address "wbstart_addr". </para>
<screen>wbstart_addr <= "User_image_address" when (cpld_user_bs_req = '1') else "00000000000000000000000000000000";</screen>
<para>capi_xilmltbt.vhdl has a Xilinx multi-boot core. That means you can create two kinds of images: Factory image and User image. Factory images will be placed at address 0 of FPGA Flash, and User image will be placed at "User_image_address" on the flash. When power-on or the FPGA card is reset, the multiboot core knows where to load the image. Usually we put a Golden factory image on address 0 and never change it, and multiboot core always tries to load user image first. If the user image has something wrong, multiboot logic will tell the FPGA to "fallback" to factory image. You still see the card in the system and you can just program a new user image to try again. </para>
</listitem>
<listitem>
<para> Check Vivado Version. Make sure this version of Vivado tool supports the FPGA part name you have assigned in "capi2-bsp/[FPGACARD]/Makefile". For some very new FPGA chip types, in one Vivado version they may have a suffix of "es" (engineering sample), and in a newer Vivado version the "es" suffix is removed.</para>
<para>When you download and install <emphasis role="bold">Vivado Lab</emphasis>, please pick up as same version as the Vivado (SDx) that you are using to build images. </para>
</important>
<para></para>
<para>Seeing 0477 by "lspci" is the most important milestone. If not, please do following checking:</para>
<itemizedlist>
<listitem> <para> Check <emphasis role="bold">dmesg</emphasis>. Run "dmesg > dmesg.log" and search "cxl" in dmesg.log file.</para></listitem>
<listitem> <para> Check file "/sys/firmware/opal/msglog" to see whether there are link training failed messages. A successful message looks like this, which means this PCIe device has been scanned and recognized. The number followed "PHB#" is the PCIe device identifier in the format of "domain:bus:slot.func". You can see it by "lspci" also. </para>
<para> <emphasis role="bold">Tips to help you debug:</emphasis></para>
<orderedlist>
<listitem><para>Seeing 0477 by "lspci" is the most important milestone. If not, please check file "<emphasis role="bold">/sys/firmware/opal/msglog</emphasis>" to see whether there are link training failed messages. A successful message looks like this, which means this PCIe device has been scanned and recognized. The number followed "PHB#" is the PCIe device identifier in the format of "domain:bus:slot.func". You can see it by "lspci" also.:</para>
<listitem> <para> Check create_ip.tcl in capi2-bsp/[FPGACARD]/tcl and check the configuration of PCIHIP core.</para></listitem>
</itemizedlist>
<listitem> <para> Check <emphasis role="bold">dmesg</emphasis>. Run "dmesg > dmesg.log" and search "cxl" in dmesg.log file. A normal output should be look like this</para>
<screen>[ 9.301403] cxl-pci 0000:01:00.0: Device uses a PSL9
<listitem><para>Today most of the linux kernel versions already include cxl module. If you doubt this, please check by</para>
<screen>modinfo cxl</screen>
</listitem>
<listitem> <para> Check create_ip.tcl in capi2-bsp/[FPGACARD]/tcl and check the configuration of PCIe core.</para>
<para>If your PCIe device has been recognized as CAPI, do "ls /dev/cxl" and you can see "afu*" devices. Then your application software can open the device like operating an ordinary file.</para>
<screen>ls /dev/cxl
afu0.0m afu0.0s</screen>
<para>Some other useful commands to check PCIe config (with the right PCIe identifier "domain:bus:slot.func") </para>
<screen>sudo lspci -s 0000:01:00.0 -vvv</screen>
<para>For example, you can check the settings coded in Xilinx PCIHIP core, like SUBSYSTEM_ID:</para>
<para>For example, you can check the settings coded in Xilinx PCIe core, like SUBSYSTEM_ID:</para>
<screen>0000:01:00.0 Processing accelerators: IBM Device 0477 (rev 02) (prog-if ff)
Subsystem: IBM Device 0660</screen>
<para>Link Speed</para>
@ -322,6 +331,48 @@ End</screen>
<screen>Capabilities: [400 v1] Vendor Specific Information: ID=1280 Rev=0 Len=080 <?>
Kernel driver in use: cxl-pci
Kernel modules: cxl</screen>
</listitem>
<listitem>
<para>If nothing shows by "ls /dec/cxl", you should check PCIe config space by</para>
<para>Please change the PCIe device identifier (0000:00:00.1) accordingly. Make sure you have seen the VSEC is properly linked. If not, go back to check your VSEC address in capi_vsec.vhdl in last chapter.</para>
<para>Now a new FPGA card has been enabled to CAPI2.0 SNAP. Cleanup your workspace, check files and submit your work!</para>
<para>capi-utils is independent. Just create a pull request and assign a reviewer. It can only been merged into master branch after having been reviewed.</para>
<para>Submit the pull request of your "capi2-bsp fork" before "snap fork". After capi2-bsp is merged intohttps://github.com/open-power/capi2-bsp master branch, update the submodule pointer to the latest "open-power/capi2-bsp" master and then submit the pull request of your forked snap.</para>
<itemizedlist>
<listitem><para>capi-utils is independent. Just create a pull request and assign a reviewer. It can only been merged into master branch after having been reviewed.</para></listitem>
<listitem><para>Submit the pull request of your "capi2-bsp fork" before "snap fork". Assign the reviewer and wait capi2-bsp to be merged into https://github.com/open-power/capi2-bsp master branch</para></listitem>
<listitem><para>Update the submodule pointer to the latest "open-power/capi2-bsp" master and then submit the pull request of your forked snap.</para></listitem>
Yong Lu
5 years ago