Repository navigation
gc2: es: Enhance VR fault handling with CPLD indicators - #2786
Closed
Joseph-Shih-ww wants to merge 1 commit into
Closed
Joseph-Shih-ww wants to merge 1 commit into
Joseph-Shih-ww wants to merge 1 commit into
Conversation
[Task Description] - Related to GC20T5T7-362 - Enhance the existing VR power fault handling mechanism by using the new CPLD power fault indicator registers introduced in CPLD FW 0x00060303. - The new indicators are used when the BIC handles VR power fault events to identify the affected power rail. - This improves VR fault-source identification by using dedicated CPLD live fault status instead of relying on PWRGD status. [Motivation] - CPLD FW 0x00060303 introduces BIC I2C register offsets 0x11 and 0x12 for power fault indicators. - The existing VR fault handling relies on PWRGD status, which does not directly represent the dedicated power fault indication provided by the CPLD. - Using the new CPLD fault indicators provides more direct and accurate fault-source identification. [Design] - Add CPLD register 0x11 and 0x12 definitions for VR power fault status. - Update the VR fault mapping table to use the corresponding CPLD live fault bits for each supported power rail. - Keep the BIC and BMC VR fault source mappings aligned. - Use the cached VR module type during VR fault handling. - Remove the previous E1S power-enable filtering because the CPLD provides live fault status. [Test Log] - Verified the CPLD 0x11 and 0x12 fault-bit mappings against the CPLD FW 0x00060303 register definition. - Confirmed the expected VR_FAULT SEL is generated with the correct fault source. - PMBus ''' root@bmc-oob:/mnt/data# bic-util server 0x18 0x52 0x09 0xc0 0x00 0x00 0x01;bic-util server 0x18 0x52 0x09 0xc0 0x00 0x55 0x01 0x00 root@bmc-oob:/mnt/data# bic-util server 0x18 0x52 0x00 0x42 0x01 0x12 12 root@bmc-oob:/mnt/data# bic-util server --get_gpio |grep FORCE 31 FM_FORCE_ADR_N_R: 0 root@bmc-oob:/mnt/data# log-util all --print 2026 Oct 11 18:54:52 log-util: User cleared all logs 0 all 2026-10-11 18:54:54 healthd ASSERT: Verified boot failure (3,35) 0 all 2026-10-11 18:54:54 healthd Verified boot failure reason: U-Boot FIT did not contain the /keys node 1 server 2026-10-11 18:55:37 ipmid SEL Entry: FRU: 1, Record: Facebook Unified SEL (0xFB), GeneralInfo: POST(0x28), POST Failure Event: System HTTP boot fail, Fail Type: IPv6 fail, Error Code: 0x63 1 server 2026-10-11 18:56:41 ipmid SEL Entry: FRU: 1, Record: Facebook Unified SEL (0xFB), GeneralInfo: POST(0x28), POST Failure Event: System HTTP boot fail, Fail Type: IPv6 fail, Error Code: 0x63 1 server 2026-10-11 18:57:46 ipmid SEL Entry: FRU: 1, Record: Facebook Unified SEL (0xFB), GeneralInfo: POST(0x28), POST Failure Event: System PXE boot fail, Fail Type: IPv6 fail, Error Code: 0x10 1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (003841) PVCCIN_CPU0 status: 0x3841 Assertion 1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007841) PVCCIN_CPU0 status: 0x7841 Assertion 1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007A00) PVCCIN_CPU0 status: 0x7a00 Assertion 1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007B00) PVCCIN_CPU0 status: 0x7b00 Assertion 1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007C80) PVCCIN_CPU0 status: 0x7c80 Assertion 1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007D00) PVCCIN_CPU0 status: 0x7d00 Assertion 1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007E00) PVCCIN_CPU0 status: 0x7e00 Assertion 1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007F00) PVCCIN_CPU0 status: 0x7f00 Assertion 1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (008002) PVCCIN_CPU0 status: 0x8002 Assertion 1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (013841) PVCCFA_EHV_FIVRA_CPU0 status: 0x3841 Assertion 1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017841) PVCCFA_EHV_FIVRA_CPU0 status: 0x7841 Assertion 1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017A00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7a00 Assertion 1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017B00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7b00 Assertion 1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017C80) PVCCFA_EHV_FIVRA_CPU0 status: 0x7c80 Assertion 1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017D00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7d00 Assertion 1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017E00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7e00 Assertion 1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017F00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7f00 Assertion 1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (018002) PVCCFA_EHV_FIVRA_CPU0 status: 0x8002 Assertion 1 server 2026-10-11 19:15:20 gpiod FRU: 1, Server is powered off 1 server 2026-10-11 19:15:24 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:24, Sensor: ME_POWER_STATE (0x16), Event Data: (000000) RUNNING Assertion 1 server 2026-10-11 19:15:33 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:33, Sensor: SYSTEM_STATUS (0x10), Event Data: (0AFFFF) SYS_VRWATCHDOG Assertion root@bmc-oob:/mnt/data# ''' - Non-PMBus ``` root@bmc-oob:~# bic-util server 0x18 0x52 0x00 0x42 0x00 0x18 0x02 root@bmc-oob:~# bic-util server 0x18 0x52 0x00 0x42 0x01 0x11 02 root@bmc-oob:~# log-util all --print |grep -i 1v8 1 server 2026-10-11 20:27:12 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 20:27:12, Sensor: VR_FAULT (0xB1), Event Data: (06FFFF) P1V8_STBY status: 0xffff Assertion ```
Contributor
|
This pull request has been imported. If you are a Meta employee, you can view this in D124729908. (Because this pull request was imported automatically, there will not be any future comments.) |
Contributor
|
This pull request has been merged in ad02eee. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[Task Description]
[Motivation]
[Design]
[Test Log]
root@bmc-oob:/mnt/data# bic-util server 0x18 0x52 0x09 0xc0 0x00 0x00 0x01;bic-util server 0x18 0x52 0x09 0xc0 0x00 0x55 0x01 0x00
root@bmc-oob:/mnt/data# bic-util server 0x18 0x52 0x00 0x42 0x01 0x12 12
root@bmc-oob:/mnt/data# bic-util server --get_gpio |grep FORCE 31 FM_FORCE_ADR_N_R: 0
root@bmc-oob:/mnt/data# log-util all --print
2026 Oct 11 18:54:52 log-util: User cleared all logs
0 all 2026-10-11 18:54:54 healthd ASSERT: Verified boot failure (3,35)
0 all 2026-10-11 18:54:54 healthd Verified boot failure reason: U-Boot FIT did not contain the /keys node
1 server 2026-10-11 18:55:37 ipmid SEL Entry: FRU: 1, Record: Facebook Unified SEL (0xFB), GeneralInfo: POST(0x28), POST Failure Event: System HTTP boot fail, Fail Type: IPv6 fail, Error Code: 0x63
1 server 2026-10-11 18:56:41 ipmid SEL Entry: FRU: 1, Record: Facebook Unified SEL (0xFB), GeneralInfo: POST(0x28), POST Failure Event: System HTTP boot fail, Fail Type: IPv6 fail, Error Code: 0x63
1 server 2026-10-11 18:57:46 ipmid SEL Entry: FRU: 1, Record: Facebook Unified SEL (0xFB), GeneralInfo: POST(0x28), POST Failure Event: System PXE boot fail, Fail Type: IPv6 fail, Error Code: 0x10
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (003841) PVCCIN_CPU0 status: 0x3841 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007841) PVCCIN_CPU0 status: 0x7841 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007A00) PVCCIN_CPU0 status: 0x7a00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007B00) PVCCIN_CPU0 status: 0x7b00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007C80) PVCCIN_CPU0 status: 0x7c80 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007D00) PVCCIN_CPU0 status: 0x7d00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007E00) PVCCIN_CPU0 status: 0x7e00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (007F00) PVCCIN_CPU0 status: 0x7f00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (008002) PVCCIN_CPU0 status: 0x8002 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (013841) PVCCFA_EHV_FIVRA_CPU0 status: 0x3841 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017841) PVCCFA_EHV_FIVRA_CPU0 status: 0x7841 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017A00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7a00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017B00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7b00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017C80) PVCCFA_EHV_FIVRA_CPU0 status: 0x7c80 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017D00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7d00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017E00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7e00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (017F00) PVCCFA_EHV_FIVRA_CPU0 status: 0x7f00 Assertion
1 server 2026-10-11 19:15:19 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:19, Sensor: VR_FAULT (0xB1), Event Data: (018002) PVCCFA_EHV_FIVRA_CPU0 status: 0x8002 Assertion
1 server 2026-10-11 19:15:20 gpiod FRU: 1, Server is powered off
1 server 2026-10-11 19:15:24 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:24, Sensor: ME_POWER_STATE (0x16), Event Data: (000000) RUNNING Assertion
1 server 2026-10-11 19:15:33 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-10-11 19:15:33, Sensor: SYSTEM_STATUS (0x10), Event Data: (0AFFFF) SYS_VRWATCHDOG Assertion
root@bmc-oob:/mnt/data#
'''