When running this code on a single node or a cluster with one executors this works fine regardless of the cardinality of the categorical column. However, as soon as the number of executors are 2 or more and cardinality of the categorical column is more than 56 this will fail.
from synapse.ml.lightgbm import LightGBMRegressor
from pyspark.ml.feature import StringIndexer, VectorAssembler
# Generate synthetic data with 100,000 rows and 6 categorical columns
import numpy as np
import pandas as pd
num_rows = 100000
cat7_values = [i for i in range(200)] # 200 distinct values for cat7
df_pd = pd.DataFrame({
"cat1": np.random.choice(['A', 'B', 'C'], num_rows),
"cat2": np.random.choice(['X', 'Y', 'Z'], num_rows),
"cat3": np.random.choice(['foo', 'bar', 'baz'], num_rows),
"cat4": np.random.choice(['red', 'green', 'blue'], num_rows),
"cat5": np.random.choice(['dog', 'cat', 'mouse'], num_rows),
"cat6": np.random.choice(['apple', 'banana', 'cherry'], num_rows),
"cat7": np.random.choice(cat7_values, num_rows), # new high-cardinality column
"num1": np.random.randn(num_rows),
"num2": np.random.randn(num_rows),
"num3": np.random.randn(num_rows),
"num4": np.random.randn(num_rows),
"label": np.random.randn(num_rows)
})
df = spark.createDataFrame(df_pd)
# Index categorical columns
indexers = [
StringIndexer(inputCol=col, outputCol=f"{col}_idx")
for col in ["cat1", "cat2", "cat3", "cat4", "cat5", "cat6", "cat7"]
]
# Assemble features
assembler = VectorAssembler(
inputCols=["cat1_idx", "cat2_idx", "cat3_idx", "cat4_idx", "cat5_idx", "cat6_idx", "cat7_idx", "num1", "num2", "num3", "num4"],
outputCol="features"
)
# LightGBMRegressor
lgbm = LightGBMRegressor(
featuresCol="features",
labelCol="label",
categoricalSlotIndexes=[0, 1, 2, 3, 4, 5, 6],
useBarrierExecutionMode=True,
# numTasks=2
)
from pyspark.ml import Pipeline
pipeline = Pipeline(stages=indexers + [assembler, lgbm])
# df = df.repartition(2)
model = pipeline.fit(df)
SynapseML version
1.1.3
System information
Describe the problem
When running this code on a single node or a cluster with one executors this works fine regardless of the cardinality of the categorical column. However, as soon as the number of executors are 2 or more and cardinality of the categorical column is more than 56 this will fail.
We have compiled the SynapseML LGBM for scala 2.13 using
com.microsoft.ml.lightgbm:lightgbmlib:3.3.500instead ofcom.microsoft.ml.lightgbm:lightgbmlib:3.3.510and this resolved the issue.SynapseML213-success-Cluster-single-executor-high-cardinality.ipynb
SynapseML213-Success-Cluster-2+executors-56-cardinality.ipynb
SynapseML213-Failure-Cluster-2+executors-high-cardinality.ipynb
Code to reproduce issue
Other info / logs
No response
What component(s) does this bug affect?
area/cognitive: Cognitive projectarea/core: Core projectarea/deep-learning: DeepLearning projectarea/lightgbm: Lightgbm projectarea/opencv: Opencv projectarea/vw: VW projectarea/website: Websitearea/build: Project build systemarea/notebooks: Samples under notebooks folderarea/docker: Docker usagearea/models: models related issueWhat language(s) does this bug affect?
language/scala: Scala source codelanguage/python: Pyspark APIslanguage/r: R APIslanguage/csharp: .NET APIslanguage/new: Proposals for new client languagesWhat integration(s) does this bug affect?
integrations/synapse: Azure Synapse integrationsintegrations/azureml: Azure ML integrationsintegrations/databricks: Databricks integrations