DM: Extend MySQL and MariaDB support for newer versions by dveeden · Pull Request #12628 · pingcap/tiflow

dveeden · 2026-05-07T09:40:53Z

Updated go-mysql

What problem does this PR solve?

Issue Number: closes #12629

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Support for newer versions of MySQL and MariaDB was extended

dveeden · 2026-05-07T09:41:31Z

/cc @OliverS929 @D3Hunter

gemini-code-assist

Code Review

This pull request updates the project to Go 1.25.9 and performs a significant upgrade of various dependencies, most notably go-mysql and tidb. The core changes involve refactoring GTID handling to align with the updated go-mysql library, including the transition from UUIDSet to MysqlGTIDSet and structural changes to MariaDB GTID sets. Feedback from the review highlights a potential side-effect in GTID incrementing that should be addressed by working on a copy of the data. Additionally, there are recommendations for improving code maintainability through the use of helper functions for repeated logic, removing redundant assignments, and simplifying loops where variables are unused.

gemini-code-assist · 2026-05-07T09:43:36Z

 		mariaGTID := singleGTID.(*gmysql.MariadbGTID)
 		mariaGTID.SequenceNumber++
 		gtidSet := new(gmysql.MariadbGTIDSet)
-		gtidSet.Sets = map[uint32]map[uint32]*gmysql.MariadbGTID{
-			mariaGTID.DomainID: {
-				mariaGTID.ServerID: mariaGTID,
-			},
+		gtidSet.Sets = map[uint32]*gmysql.MariadbGTID{
+			mariaGTID.DomainID: mariaGTID,
 		}
 		clone = gtidSet


In the MariaDB flavor case, mariaGTID is a pointer to an element within the original gSet. Modifying mariaGTID.SequenceNumber directly causes a side effect on the input gSet. To ensure GTIDIncrease is pure and doesn't modify its input, you should work on a copy of the GTID struct.

Suggested change

mariaGTID := singleGTID.(*gmysql.MariadbGTID)

mariaGTID.SequenceNumber++

gtidSet := new(gmysql.MariadbGTIDSet)

gtidSet.Sets = map[uint32]map[uint32]*gmysql.MariadbGTID{

mariaGTID.DomainID: {

mariaGTID.ServerID: mariaGTID,

},

gtidSet.Sets = map[uint32]*gmysql.MariadbGTID{

mariaGTID.DomainID: mariaGTID,

}

clone = gtidSet

mariaGTID := *singleGTID.(*gmysql.MariadbGTID)

mariaGTID.SequenceNumber++

gtidSet := new(gmysql.MariadbGTIDSet)

gtidSet.Sets = map[uint32]*gmysql.MariadbGTID{

mariaGTID.DomainID: &mariaGTID,

}

clone = gtidSet

gemini-code-assist · 2026-05-07T09:43:36Z

+		for u, tags := range *mSet {
+			sid = u
+			for _, intervals := range tags {
+				interval = intervals[0]
+			}
+		}


The logic for extracting the single SID and Interval from a MysqlGTIDSet is repeated here and in newGenerator (in generator.go). Since DM requires these sets to have exactly one entry (as enforced by verifySingleGTID), consider moving this extraction logic into a helper function to improve code reuse and maintainability.

gemini-code-assist · 2026-05-07T09:43:36Z

+			for tag, intervals := range tags {
+				intervals[0].Start++
+				intervals[0].Stop++
+				(*mSet)[u][tag] = intervals


This assignment is redundant because intervals is a slice, which is a reference type in Go. Modifying its elements on lines 167-168 already updates the data within the map. Removing this line would clarify that the modification is happening in-place on the cloned set.

gemini-code-assist · 2026-05-07T09:43:36Z

+		var sid uuid.UUID
+		var tags map[gmysql.Tag]gmysql.IntervalSlice
+		for sid, tags = range *mysqlGTIDs {
 		}
-		var uuidSet *gmysql.UUIDSet
-		for _, uuidSet = range mysqlGTIDs.Sets {
+		_ = sid


The variable sid is declared and assigned but not used, necessitating a blank assignment _ = sid to satisfy the compiler. This can be simplified by using a blank identifier directly in the range loop.

var tags map[gmysql.Tag]gmysql.IntervalSlice for _, tags = range *mysqlGTIDs { }

dveeden · 2026-05-08T08:52:00Z

/retest

#4578) ### What Update TiFlow latest CI images to the newer Go 1.25 image tag: - `ghcr.io/pingcap-qe/ci/jenkins:v2026.4.12-10-gc29110c-go1.25` for jenkins job - `ghcr.io/pingcap-qe/ci/base:v2026.4.12-10-gc29110c-go1.25` for the Prow unit test job ### Why TiFlow PR pingcap/tiflow#12628 updates `go.mod` to `go 1.25.9`. Some CI jobs were still running with older Go 1.25 images, causing errors like: ```text compile: version "go1.25.9" does not match go tool version "go1.25.8" ``` ### Test result I have run replay tests for all failed tasks. The Go version mismatch issue is now fully resolved. The remaining failures are either integration test failures inherent to the TiFlow PR itself, or runtime issues on individual Jenkins/K8s agents. These are not directly related to the current upgrade of the Go image. --------- Signed-off-by: lyb <yebin.li@pingcap.com>

wuhuizuo · 2026-05-09T02:22:19Z

/test pull-verify

dveeden · 2026-05-11T06:19:44Z

/retest

dveeden · 2026-05-11T10:49:50Z

/retest

dveeden · 2026-05-11T11:17:05Z

/retest

dveeden · 2026-05-11T15:52:04Z

/retest

dveeden · 2026-05-11T17:03:42Z

/retest pull-verify

ti-chi-bot · 2026-05-11T17:03:47Z

@dveeden: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

/test pull-build

/test pull-cdc-integration-kafka-test

/test pull-cdc-integration-mysql-test

/test pull-cdc-integration-pulsar-test

/test pull-cdc-integration-storage-test

/test pull-check

/test pull-dm-compatibility-test

/test pull-dm-integration-test

/test pull-error-log-review

/test pull-syncdiff-integration-test

/test pull-unit-test-cdc

/test pull-verify

/test wip-pull-unit-test-dm

/test wip-pull-unit-test-engine

The following commands are available to trigger optional jobs:

/test pull-dm-integration-test-next-gen

Use /test all to run the following jobs that were automatically triggered:

pingcap/tiflow/ghpr_verify

pingcap/tiflow/pull_cdc_integration_kafka_test

pingcap/tiflow/pull_cdc_integration_pulsar_test

pingcap/tiflow/pull_cdc_integration_storage_test

pingcap/tiflow/pull_cdc_integration_test

pingcap/tiflow/pull_dm_compatibility_test

pingcap/tiflow/pull_dm_integration_test

pingcap/tiflow/pull_dm_integration_test_next_gen

pingcap/tiflow/pull_syncdiff_integration_test

pull-build

pull-check

pull-error-log-review

pull-unit-test-cdc

Details

In response to this:

/retest pull-verify

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

dveeden · 2026-05-11T17:04:05Z

/test pull-verify

dveeden · 2026-05-11T19:42:58Z

/retest

dveeden · 2026-05-12T05:39:36Z

/retest

Updated go-mysql

go-mysql v1.15.0 (PR pingcap#1080) decodes upstream heartbeat events as a dedicated *replication.HeartbeatEvent type instead of GenericEvent. DM's switches on *replication.GenericEvent no longer match real heartbeats, so the validator never persists checkpoints, the syncer never flushes on idle, and the relay no longer marks heartbeats as ignored. This caused validator_basic to hang with non-zero pending rows. Accept both types since DM still injects its own heartbeats locally as GenericEvent. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The test asserted finished_bytes=59674 verbatim, but the tidb-pkg upgrade in this branch ships a slightly different lightning that reports 59754 bytes for the same dumped data. Replace the literal byte count with a backreference assertion that finished_bytes equals total_bytes (and is non-empty), so the test no longer breaks on benign size shifts from lightning. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

ti-chi-bot · 2026-06-25T06:56:39Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign lance6716 for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS
dm/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dveeden · 2026-06-26T09:12:35Z

/test pull-verify

dveeden · 2026-06-26T14:08:01Z

/retest

dveeden · 2026-07-01T07:15:24Z

/retest

ti-chi-bot Bot added do-not-merge/needs-triage-completed release-note Denotes a PR that will be considered when it comes time to generate release notes. area/dm Issues or PRs related to DM. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 7, 2026

ti-chi-bot Bot requested review from D3Hunter and OliverS929 May 7, 2026 09:41

ti-chi-bot Bot removed the do-not-merge/needs-triage-completed label May 7, 2026

gemini-code-assist Bot reviewed May 7, 2026

View reviewed changes

dveeden force-pushed the tagged_gtid branch 4 times, most recently from 59316e0 to a6afe34 Compare May 7, 2026 12:52

ti-chi-bot Bot added the do-not-merge/needs-linked-issue label May 7, 2026

wuhuizuo changed the title ~~DM: Extend MySQL and MariaDB support for newer versions~~ DM: Extend MySQL and MariaDB support for newer versions May 8, 2026

ti-chi-bot Bot removed the do-not-merge/needs-linked-issue label May 8, 2026

wuhuizuo changed the title ~~DM: Extend MySQL and MariaDB support for newer versions~~ DM: Extend MySQL and MariaDB support for newer versions May 8, 2026

lybcodes mentioned this pull request May 8, 2026

chore(pingcap/tiflow): update latest Go 1.25 CI images for tiflow jobs PingCAP-QE/ci#4578

Merged

ti-chi-bot Bot added the area/engine Issues or PRs related to Dataflow Engine. label May 11, 2026

dveeden and others added 5 commits June 24, 2026 16:58

DM: Support for Taggged GTIDs

f5e9faa

Updated go-mysql

Update errors_release.txt

aa2cc8f

Update TestAddGSetWithPurged

7d04ba9

dveeden force-pushed the tagged_gtid branch from e2c7533 to c4f6f36 Compare June 24, 2026 15:25

ti-chi-bot Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 24, 2026

dveeden force-pushed the tagged_gtid branch 2 times, most recently from 1c8f644 to c4f6f36 Compare June 25, 2026 06:56

Merge remote-tracking branch 'upstream/master' into tagged_gtid

6995e65

Conversation

dveeden commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Check List

Tests

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Uh oh!

dveeden commented May 7, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 7, 2026

Choose a reason for hiding this comment

Uh oh!

dveeden commented May 8, 2026

Uh oh!

wuhuizuo commented May 9, 2026

Uh oh!

dveeden commented May 11, 2026

Uh oh!

dveeden commented May 11, 2026

Uh oh!

dveeden commented May 11, 2026

Uh oh!

dveeden commented May 11, 2026

Uh oh!

dveeden commented May 11, 2026

Uh oh!

ti-chi-bot Bot commented May 11, 2026

Uh oh!

dveeden commented May 11, 2026

Uh oh!

dveeden commented May 11, 2026

Uh oh!

dveeden commented May 12, 2026

Uh oh!

ti-chi-bot Bot commented Jun 25, 2026

Uh oh!

dveeden commented Jun 26, 2026

Uh oh!

dveeden commented Jun 26, 2026

Uh oh!

dveeden commented Jul 1, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dveeden commented May 7, 2026 •

edited

Loading