Skip to content

gc2: es: Enhance VR fault handling with CPLD indicators - #2786

Closed
Joseph-Shih-ww wants to merge 1 commit into
facebook:mainfrom
Wiwynn:Liora/gc2-es/enhance_vr_fault_handler
Closed

Joseph-Shih-ww wants to merge 1 commit into
facebook:mainfrom
Wiwynn:Liora/gc2-es/enhance_vr_fault_handler

Conversation

@Joseph-Shih-ww

Copy link
Copy Markdown
Contributor

[Task Description]

  • Related to GC20T5T7-362
  • Enhance the existing VR power fault handling mechanism by using the new CPLD power fault indicator registers introduced in CPLD FW 0x00060303.
  • The new indicators are used when the BIC handles VR power fault events to identify the affected power rail.
  • This improves VR fault-source identification by using dedicated CPLD live fault status instead of relying on PWRGD status.

[Motivation]

  • CPLD FW 0x00060303 introduces BIC I2C register offsets 0x11 and 0x12 for power fault indicators.
  • The existing VR fault handling relies on PWRGD status, which does not directly represent the dedicated power fault indication provided by the CPLD.
  • Using the new CPLD fault indicators provides more direct and accurate fault-source identification.

[Design]

  • Add CPLD register 0x11 and 0x12 definitions for VR power fault status.
  • Update the VR fault mapping table to use the corresponding CPLD live fault bits for each supported power rail.
  • Keep the BIC and BMC VR fault source mappings aligned.
  • Use the cached VR module type during VR fault handling.
  • Remove the previous E1S power-enable filtering because the CPLD provides live fault status.

[Test Log]

  • Verified the CPLD 0x11 and 0x12 fault-bit mappings against the CPLD FW 0x00060303 register definition.
  • Confirmed the expected VR_FAULT SEL is generated with the correct fault source.
    • PMBus '''
      root@bmc-oob:/mnt/data# bic-util server 0x18 0x52 0x09 0xc0 0x00 0x00 0x01;bic-util server 0x18 0x52 0x09 0xc0 0x00 0x55 0x01 0x00

root@bmc-oob:/mnt/data# bic-util server 0x18 0x52 0x00 0x42 0x01 0x12 12
root@bmc-oob:/mnt/data# bic-util server --get_gpio |grep FORCE 31 FM_FORCE_ADR_N_R: 0
root@bmc-oob:/mnt/data# log-util all --print
2026 Oct 11 18:54:52 log-util: User cleared all logs
0 all 2026-10-11 18:54:54 healthd ASSERT: Verified boot failure (3,35)
0 all 2026-10-11 18:54:54 healthd Verified boot failure reason: U-Boot FIT did not contain the /keys node
1 server 2026-10-11 18:55:37 ipmid SEL Entry: FRU: 1, Record: Facebook Unified SEL (0xFB), GeneralInfo: POST(0x28), POST Failure Event: System HTTP boot fail, Fail Type: IPv6 fail, Error Code: 0x63
1 server 2026-10-11 18:56:41 ipmid SEL Entry: FRU: 1, Record: Facebook Unified SEL (0xFB), GeneralInfo: POST(0x28), POST Failure Event: System HTTP boot fail, Fail Type: IPv6 fail, Error Code: 0x63
1 server 2026-10-11 18:57:46 ipmid SEL Entry: FRU: 1, Record: Facebook Unified SEL (0xFB), GeneralInfo: POST(0x28), POST Failure Event: System PXE boot fail, Fail Type: IPv6 fail, Error Code: 0x10
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (003841) PVCCIN_CPU0 status: 0x3841 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007841) PVCCIN_CPU0 status: 0x7841 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007A00) PVCCIN_CPU0 status: 0x7a00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007B00) PVCCIN_CPU0 status: 0x7b00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007C80) PVCCIN_CPU0 status: 0x7c80 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007D00) PVCCIN_CPU0 status: 0x7d00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007E00) PVCCIN_CPU0 status: 0x7e00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007F00) PVCCIN_CPU0 status: 0x7f00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (008002) PVCCIN_CPU0 status: 0x8002 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (013841) PVCCFA_EHV_FIVRA_CPU0 status: 0x3841 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017841) PVCCFA_EHV_FIVRA_CPU0 status: 0x7841 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017A00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7a00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017B00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7b00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017C80) PVCCFA_EHV_FIVRA_CPU0 status: 0x7c80 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017D00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7d00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017E00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7e00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017F00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7f00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (018002) PVCCFA_EHV_FIVRA_CPU0 status: 0x8002 Assertion
1 server 2026-10-11 19:15:20 gpiod FRU: 1, Server is powered off
1 server 2026-10-11 19:15:24 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:24, Sensor: ME_POWER_STATE (0x16), Event Data: (000000) RUNNING Assertion
1 server 2026-10-11 19:15:33 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:33, Sensor: SYSTEM_STATUS (0x10), Event Data: (0AFFFF) SYS_VRWATCHDOG Assertion
root@bmc-oob:/mnt/data#
'''

  • Non-PMBus
root@bmc-oob:~#  bic-util server 0x18 0x52 0x00 0x42 0x00 0x18 0x02

root@bmc-oob:~#  bic-util server 0x18 0x52 0x00 0x42 0x01 0x11
02
root@bmc-oob:~# log-util all --print |grep -i 1v8
1    server   2026-10-11 20:27:12    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 20:27:12, Sensor: VR_FAULT (0xB1), Event Data: (06FFFF) P1V8_STBY status: 0xffff Assertion

[Task Description]
- Related to GC20T5T7-362
- Enhance the existing VR power fault handling mechanism by using the
  new CPLD power fault indicator registers introduced in CPLD FW
  0x00060303.
- The new indicators are used when the BIC handles VR power fault
  events to identify the affected power rail.
- This improves VR fault-source identification by using dedicated CPLD
  live fault status instead of relying on PWRGD status.

[Motivation]
- CPLD FW 0x00060303 introduces BIC I2C register offsets 0x11 and 0x12
  for power fault indicators.
- The existing VR fault handling relies on PWRGD status, which does not
  directly represent the dedicated power fault indication provided by
  the CPLD.
- Using the new CPLD fault indicators provides more direct and accurate
  fault-source identification.

[Design]
- Add CPLD register 0x11 and 0x12 definitions for VR power fault status.
- Update the VR fault mapping table to use the corresponding CPLD live
  fault bits for each supported power rail.
- Keep the BIC and BMC VR fault source mappings aligned.
- Use the cached VR module type during VR fault handling.
- Remove the previous E1S power-enable filtering because the CPLD
  provides live fault status.

[Test Log]
- Verified the CPLD 0x11 and 0x12 fault-bit mappings against the CPLD
  FW 0x00060303 register definition.
- Confirmed the expected VR_FAULT SEL is generated with the correct
  fault source.
  - PMBus
'''
root@bmc-oob:/mnt/data# bic-util server 0x18 0x52 0x09 0xc0 0x00 0x00 0x01;bic-util server 0x18 0x52 0x09 0xc0 0x00 0x55 0x01 0x00

root@bmc-oob:/mnt/data# bic-util server 0x18 0x52 0x00 0x42 0x01 0x12
12
root@bmc-oob:/mnt/data# bic-util server --get_gpio |grep FORCE
31 FM_FORCE_ADR_N_R: 0
root@bmc-oob:/mnt/data# log-util all --print
2026 Oct 11 18:54:52 log-util: User cleared all logs
0    all      2026-10-11 18:54:54    healthd          ASSERT: Verified boot failure (3,35)
0    all      2026-10-11 18:54:54    healthd          Verified boot failure reason: U-Boot FIT did not contain the /keys node
1    server   2026-10-11 18:55:37    ipmid            SEL Entry: FRU: 1, Record: Facebook Unified SEL (0xFB), GeneralInfo: POST(0x28), POST Failure Event: System HTTP boot fail, Fail Type: IPv6 fail, Error Code: 0x63
1    server   2026-10-11 18:56:41    ipmid            SEL Entry: FRU: 1, Record: Facebook Unified SEL (0xFB), GeneralInfo: POST(0x28), POST Failure Event: System HTTP boot fail, Fail Type: IPv6 fail, Error Code: 0x63
1    server   2026-10-11 18:57:46    ipmid            SEL Entry: FRU: 1, Record: Facebook Unified SEL (0xFB), GeneralInfo: POST(0x28), POST Failure Event: System PXE boot fail, Fail Type: IPv6 fail, Error Code: 0x10
1    server   2026-10-11 19:15:19    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (003841) PVCCIN_CPU0 status: 0x3841 Assertion
1    server   2026-10-11 19:15:19    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007841) PVCCIN_CPU0 status: 0x7841 Assertion
1    server   2026-10-11 19:15:19    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007A00) PVCCIN_CPU0 status: 0x7a00 Assertion
1    server   2026-10-11 19:15:19    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007B00) PVCCIN_CPU0 status: 0x7b00 Assertion
1    server   2026-10-11 19:15:19    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007C80) PVCCIN_CPU0 status: 0x7c80 Assertion
1    server   2026-10-11 19:15:19    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007D00) PVCCIN_CPU0 status: 0x7d00 Assertion
1    server   2026-10-11 19:15:19    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007E00) PVCCIN_CPU0 status: 0x7e00 Assertion
1    server   2026-10-11 19:15:19    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007F00) PVCCIN_CPU0 status: 0x7f00 Assertion
1    server   2026-10-11 19:15:19    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (008002) PVCCIN_CPU0 status: 0x8002 Assertion
1    server   2026-10-11 19:15:19    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (013841) PVCCFA_EHV_FIVRA_CPU0 status: 0x3841 Assertion
1    server   2026-10-11 19:15:19    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017841) PVCCFA_EHV_FIVRA_CPU0 status: 0x7841 Assertion
1    server   2026-10-11 19:15:19    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017A00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7a00 Assertion
1    server   2026-10-11 19:15:19    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017B00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7b00 Assertion
1    server   2026-10-11 19:15:19    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017C80) PVCCFA_EHV_FIVRA_CPU0 status: 0x7c80 Assertion
1    server   2026-10-11 19:15:19    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017D00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7d00 Assertion
1    server   2026-10-11 19:15:19    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017E00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7e00 Assertion
1    server   2026-10-11 19:15:19    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017F00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7f00 Assertion
1    server   2026-10-11 19:15:19    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (018002) PVCCFA_EHV_FIVRA_CPU0 status: 0x8002 Assertion
1    server   2026-10-11 19:15:20    gpiod            FRU: 1, Server is powered off
1    server   2026-10-11 19:15:24    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:24, Sensor: ME_POWER_STATE (0x16), Event Data: (000000) RUNNING Assertion
1    server   2026-10-11 19:15:33    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:33, Sensor: SYSTEM_STATUS (0x10), Event Data: (0AFFFF) SYS_VRWATCHDOG Assertion
root@bmc-oob:/mnt/data#
'''

  - Non-PMBus
```
root@bmc-oob:~#  bic-util server 0x18 0x52 0x00 0x42 0x00 0x18 0x02

root@bmc-oob:~#  bic-util server 0x18 0x52 0x00 0x42 0x01 0x11
02
root@bmc-oob:~# log-util all --print |grep -i 1v8
1    server   2026-10-11 20:27:12    ipmid            SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 20:27:12, Sensor: VR_FAULT (0xB1), Event Data: (06FFFF) P1V8_STBY status: 0xffff Assertion
```
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 12, 2026
@meta-codesync

meta-codesync Bot commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

This pull request has been imported. If you are a Meta employee, you can view this in D124729908. (Because this pull request was imported automatically, there will not be any future comments.)

@meta-codesync meta-codesync Bot closed this in ad02eee Oct 12, 2026
@meta-codesync meta-codesync Bot added the Merged label Oct 12, 2026
@meta-codesync

meta-codesync Bot commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

This pull request has been merged in ad02eee.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants