Deploy/DataPipeline/Secor - Pod CrashLoopBackOff Error #662

Munishdev97 · 2023-12-05T14:39:41Z

Munishdev97
Dec 5, 2023

For Deploy/DataPipeline/Secor pipeline we are getting the following error in pod logs.
Please let us know if any fix available.

log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.
log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.DailyRollingFileAppender.
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /mount/secor/logs/partition.log (No such file or directory)
        at java.io.FileOutputStream.open0(Native Method)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
        at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
        at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
        at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
        at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
        at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
        at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
        at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
        at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
        at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
        at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66)
        at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72)
        at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45)
        at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
        at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
        at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417)
        at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
        at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
        at com.pinterest.secor.main.ConsumerMain.<clinit>(ConsumerMain.java:52)
log4j:ERROR Either File or DatePattern options are not set for appender [ROLLINGFILE].
2023-12-05 14:34:11,720 [main] (com.pinterest.secor.main.ConsumerMain:89) ERROR Consumer failed
java.lang.RuntimeException: Error loading configuration from /opt/secor/config/secor.partition.properties
        at com.pinterest.secor.common.SecorConfig$1.initialValue(SecorConfig.java:57)
        at com.pinterest.secor.common.SecorConfig$1.initialValue(SecorConfig.java:45)
        at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
        at java.lang.ThreadLocal.get(ThreadLocal.java:172)
        at com.pinterest.secor.common.SecorConfig.load(SecorConfig.java:72)
        at com.pinterest.secor.main.ConsumerMain.main(ConsumerMain.java:61)
Caused by: org.apache.commons.configuration.ConfigurationException: Cannot resolve include file secor.s3.properties
        at org.apache.commons.configuration.PropertiesConfiguration.loadIncludeFile(PropertiesConfiguration.java:1511)
        at org.apache.commons.configuration.PropertiesConfiguration.propertyLoaded(PropertiesConfiguration.java:606)
        at org.apache.commons.configuration.PropertiesConfigurationLayout.load(PropertiesConfigurationLayout.java:520)
        at org.apache.commons.configuration.PropertiesConfiguration.load(PropertiesConfiguration.java:513)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:395)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:361)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:324)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:261)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:238)
        at org.apache.commons.configuration.AbstractFileConfiguration.<init>(AbstractFileConfiguration.java:158)
        at org.apache.commons.configuration.PropertiesConfiguration.<init>(PropertiesConfiguration.java:252)
        at com.pinterest.secor.common.SecorConfig$1.initialValue(SecorConfig.java:55)

Answered by akulavasu

Dec 21, 2023

@sowmya-dixit @gandham-santhosh @santhosh-tg
Please review the PR that adds the kafka listener in server.properties during kafka installation in provision/datapipeline/kafka
Sunbird-Obsrv/sunbird-data-pipeline#1635

View full answer

gandham-santhosh · 2023-12-06T08:48:46Z

gandham-santhosh
Dec 6, 2023

@sowmya-dixit can you please help here. This is in OCI env

0 replies

Munishdev97 · 2023-12-07T05:07:07Z

Munishdev97
Dec 7, 2023
Author

@sowmya-dixit, any inputs from your end to resolve this

5 replies

sowmya-dixit Dec 7, 2023

@Munishdev97 Which branch is being used for secor deployment from https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/tree/master repo

Munishdev97 Dec 7, 2023
Author

@sowmya-dixit, We have used release-5.2.0 for the secor deployment pipeline.

sowmya-dixit Dec 7, 2023

@Munishdev97 - Can you retry now from same, we have pushed some fixes.

Munishdev97 Dec 7, 2023
Author

@sowmya-dixit, After rerun it's getting same error.

log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.
log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.DailyRollingFileAppender.
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /mount/secor/logs/partition.log (No such file or directory)
        at java.io.FileOutputStream.open0(Native Method)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
        at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
        at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
        at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
        at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
        at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
        at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
        at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
        at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
        at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
        at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66)
        at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72)
        at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45)
        at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
        at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
        at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417)
        at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
        at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
        at com.pinterest.secor.main.ConsumerMain.<clinit>(ConsumerMain.java:52)
log4j:ERROR Either File or DatePattern options are not set for appender [ROLLINGFILE].
2023-12-07 09:12:36,375 [main] (com.pinterest.secor.common.ShutdownHookRegistry:50) INFO  Shut down hook with priority 10 added to shut down hook registry
2023-12-07 09:12:36,408 [main] (com.pinterest.secor.main.ConsumerMain:78) INFO  starting 7 consumer threads
2023-12-07 09:12:36,493 [Thread-7] (org.apache.curator.framework.imps.CuratorFrameworkImpl:235) INFO  Starting
2023-12-07 09:12:36,493 [Thread-4] (org.apache.curator.framework.imps.CuratorFrameworkImpl:235) INFO  Starting
2023-12-07 09:12:36,493 [Thread-8] (org.apache.curator.framework.imps.CuratorFrameworkImpl:235) INFO  Starting
2023-12-07 09:12:36,493 [Thread-6] (org.apache.curator.framework.imps.CuratorFrameworkImpl:235) INFO  Starting
2023-12-07 09:12:36,493 [Thread-5] (org.apache.curator.framework.imps.CuratorFrameworkImpl:235) INFO  Starting
2023-12-07 09:12:36,493 [Thread-3] (org.apache.curator.framework.imps.CuratorFrameworkImpl:235) INFO  Starting
2023-12-07 09:12:36,494 [Thread-2] (org.apache.curator.framework.imps.CuratorFrameworkImpl:235) INFO  Starting
2023-12-07 09:12:36,500 [Thread-8] (org.apache.zookeeper.Environment:100) INFO  Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2023-12-07 09:12:36,500 [Thread-8] (org.apache.zookeeper.Environment:100) INFO  Client environment:host.name=assess-events-backup-0.secor.secor.svc.cluster.local
2023-12-07 09:12:36,501 [Thread-8] (org.apache.zookeeper.Environment:100) INFO  Client environment:java.version=1.8.0_342
2023-12-07 09:12:36,501 [Thread-8] (org.apache.zookeeper.Environment:100) INFO  Client environment:java.vendor=Oracle Corporation
2023-12-07 09:12:36,501 [Thread-8] (org.apache.zookeeper.Environment:100) INFO  Client environment:java.home=/usr/local/openjdk-8/jre
2023-12-07 09:12:36,501 [Thread-8] (org.apache.zookeeper.Environment:100) INFO  Client environment:java.class.path=secor-0.29-tests.jar:secor-0.29.jar:lib/HdrHistogram-2.1.12.jar:lib/LatencyUtils-2.0.3.jar:lib/accessors-smart-1.2.jar:lib/activation-1.1.1.jar:lib/aircompressor-0.15.jar:lib/animal-sniffer-annotations-1.14.jar:lib/annotations-17.0.0.jar:lib/aopalliance-1.0.jar:lib/apacheds-i18n-2.0.0-M15.jar:lib/apacheds-kerberos-codec-2.0.0-M15.jar:lib/api-asn1-api-1.0.0-M20.jar:lib/api-common-1.9.3.jar:lib/api-util-1.0.0-M20.jar:lib/asm-3.1.jar:lib/asm-5.0.4.jar:lib/asm-analysis-7.1.jar:lib/asm-commons-7.1.jar:lib/asm-tree-7.1.jar:lib/asm-util-7.1.jar:lib/audience-annotations-0.11.0.jar:lib/auto-value-annotations-1.7.2.jar:lib/avro-1.10.0.jar:lib/aws-java-sdk-core-1.11.821.jar:lib/aws-java-sdk-kms-1.11.821.jar:lib/aws-java-sdk-s3-1.11.821.jar:lib/aws-java-sdk-sts-1.11.821.jar:lib/azure-keyvault-core-1.0.0.jar:lib/azure-storage-8.6.5.jar:lib/base64-2.3.8.jar:lib/bcprov-jdk15on-1.52.jar:lib/byte-buddy-1.10.5.jar:lib/byte-buddy-agent-1.10.5.jar:lib/caffeine-2.3.4.jar:lib/checker-compat-qual-2.0.0.jar:lib/common-config-2.0.1.jar:lib/common-utils-2.0.1.jar:lib/commons-cli-1.2.jar:lib/commons-codec-1.9.jar:lib/commons-collections-3.2.2.jar:lib/commons-compress-1.20.jar:lib/commons-configuration-1.10.jar:lib/commons-httpclient-3.1.jar:lib/commons-io-2.4.jar:lib/commons-lang-2.6.jar:lib/commons-lang3-3.4.jar:lib/commons-logging-1.2.jar:lib/commons-math3-3.1.1.jar:lib/commons-net-3.1.jar:lib/commons-pool-1.6.jar:lib/curator-client-2.13.0.jar:lib/curator-framework-2.13.0.jar:lib/curator-recipes-2.7.1.jar:lib/elephant-bird-core-4.4.jar:lib/elephant-bird-hadoop-compat-4.4.jar:lib/elephant-bird-pig-4.4.jar:lib/error_prone_annotations-2.3.4.jar:lib/failureaccess-1.0.1.jar:lib/gax-1.57.1.jar:lib/gax-httpjson-0.74.1.jar:lib/google-api-client-1.30.9.jar:lib/google-api-services-storage-v1-rev20200611-1.30.9.jar:lib/google-auth-library-credentials-0.21.1.jar:lib/google-auth-library-oauth2-http-0.21.1.jar:lib/google-cloud-core-1.93.7.jar:lib/google-cloud-core-http-1.93.7.jar:lib/google-cloud-storage-1.111.2.jar:lib/google-http-client-1.36.0.jar:lib/google-http-client-appengine-1.36.0.jar:lib/google-http-client-jackson2-1.36.0.jar:lib/google-oauth-client-1.30.5.jar:lib/grpc-context-1.30.2.jar:lib/gson-2.8.6.jar:lib/guava-24.1.1-jre.jar:lib/guice-3.0.jar:lib/guice-servlet-3.0.jar:lib/hadoop-annotations-2.9.2.jar:lib/hadoop-auth-2.9.2.jar:lib/hadoop-aws-2.9.2.jar:lib/hadoop-common-2.9.2.jar:lib/hadoop-hdfs-client-2.9.2.jar:lib/hadoop-mapreduce-client-core-2.9.2.jar:lib/hadoop-openstack-2.9.2.jar:lib/hadoop-yarn-api-2.9.2.jar:lib/hadoop-yarn-client-2.9.2.jar:lib/hadoop-yarn-common-2.9.2.jar:lib/hamcrest-core-1.3.jar:lib/hive-storage-api-2.7.1.jar:lib/htrace-core4-4.1.0-incubating.jar:lib/httpclient-4.5.jar:lib/httpcore-4.4.1.jar:lib/ion-java-1.0.2.jar:lib/j2objc-annotations-1.1.jar:lib/jackson-annotations-2.11.1.jar:lib/jackson-core-2.11.1.jar:lib/jackson-core-asl-1.9.13.jar:lib/jackson-databind-2.11.0.jar:lib/jackson-dataformat-cbor-2.6.7.jar:lib/jackson-dataformat-msgpack-0.8.20.jar:lib/jackson-jaxrs-1.8.3.jar:lib/jackson-mapper-asl-1.9.13.jar:lib/jackson-module-paranamer-2.8.4.jar:lib/jackson-module-scala_2.11-2.8.4.jar:lib/jackson-xc-1.8.3.jar:lib/java-dogstatsd-client-2.10.2.jar:lib/java-xmlbuilder-1.1.jar:lib/javassist-3.27.0-GA.jar:lib/javax.annotation-api-1.3.2.jar:lib/javax.inject-1.jar:lib/jaxb-api-2.2.11.jar:lib/jaxb-impl-2.2.3-1.jar:lib/jcip-annotations-1.0-1.jar:lib/jcl-over-slf4j-1.7.30.jar:lib/jersey-client-1.9.jar:lib/jersey-core-1.9.jar:lib/jersey-guice-1.9.jar:lib/jersey-json-1.9.jar:lib/jersey-server-1.9.jar:lib/jets3t-0.9.4.jar:lib/jettison-1.1.jar:lib/jetty-6.1.26.jar:lib/jetty-sslengine-6.1.26.jar:lib/jetty-util-6.1.26.jar:lib/jffi-1.2.23-native.jar:lib/jffi-1.2.23.jar:lib/jmespath-java-1.11.821.jar:lib/jnr-a64asm-1.0.0.jar:lib/jnr-constants-0.9.15.jar:lib/jnr-enxio-0.25.jar:lib/jnr-ffi-2.1.12.jar:lib/jnr-posix-3.0.53.jar:lib/jnr-unixsocket-0.27.jar:lib/jnr-x86asm-1.0.2.jar:lib/joda-time-2.8.1.jar:lib/jopt-simple-5.0.4.jar:lib/jsch-0.1.54.jar:lib/json-path-2.4.0.jar:lib/json-simple-1.1.jar:lib/json-smart-2.3.jar:lib/jsp-api-2.1.jar:lib/jsr305-3.0.0.jar:lib/junit-4.12.jar:lib/kafka-avro-serializer-2.0.1.jar:lib/kafka-clients-2.3.1.jar:lib/kafka-schema-registry-client-2.0.1.jar:lib/kafka_2.11-1.0.0.jar:lib/libthrift-0.12.0.jar:lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:lib/log4j-1.2.17.jar:lib/lz4-java-1.6.0.jar:lib/metrics-core-2.2.0.jar:lib/metrics-core-4.0.7.jar:lib/metrics-jmx-4.0.7.jar:lib/micrometer-core-1.5.2.jar:lib/micrometer-registry-jmx-1.5.2.jar:lib/micrometer-registry-prometheus-1.5.2.jar:lib/micrometer-registry-statsd-1.5.2.jar:lib/mockito-core-3.3.3.jar:lib/msgpack-core-0.8.20.jar:lib/netty-3.6.2.Final.jar:lib/nimbus-jose-jwt-4.41.1.jar:lib/objenesis-2.6.jar:lib/okhttp-2.7.5.jar:lib/okio-1.6.0.jar:lib/opencensus-api-0.24.0.jar:lib/opencensus-contrib-http-util-0.24.0.jar:lib/orc-core-1.6.3.jar:lib/orc-shims-1.6.3.jar:lib/ostrich_2.11-9.27.0.jar:lib/paranamer-2.8.jar:lib/parquet-avro-1.11.0.jar:lib/parquet-column-1.11.0.jar:lib/parquet-common-1.11.0.jar:lib/parquet-encoding-1.11.0.jar:lib/parquet-format-structures-1.11.0.jar:lib/parquet-hadoop-1.11.0.jar:lib/parquet-jackson-1.11.0.jar:lib/parquet-pig-1.11.0.jar:lib/parquet-protobuf-1.11.0.jar:lib/parquet-thrift-1.11.0.jar:lib/powermock-api-mockito2-2.0.7.jar:lib/powermock-api-support-2.0.7.jar:lib/powermock-core-2.0.7.jar:lib/powermock-module-junit4-2.0.7.jar:lib/powermock-module-junit4-common-2.0.7.jar:lib/powermock-reflect-2.0.7.jar:lib/proto-google-common-protos-1.18.0.jar:lib/proto-google-iam-v1-0.13.0.jar:lib/protobuf-java-3.12.2.jar:lib/protobuf-java-util-3.12.2.jar:lib/scala-compiler-2.11.8.jar:lib/scala-library-2.11.8.jar:lib/scala-parser-combinators_2.11-1.0.4.jar:lib/scala-reflect-2.11.8.jar:lib/scala-xml_2.11-1.0.4.jar:lib/servlet-api-2.5.jar:lib/simpleclient-0.8.1.jar:lib/simpleclient_common-0.8.1.jar:lib/slf4j-api-1.7.30.jar:lib/slf4j-log4j12-1.7.30.jar:lib/snappy-java-1.1.7.3.jar:lib/stax2-api-3.1.4.jar:lib/threetenbp-1.4.4.jar:lib/threeten-extra-1.5.0.jar:lib/util-app_2.11-6.43.0.jar:lib/util-core_2.11-6.43.0.jar:lib/util-eval_2.11-6.43.0.jar:lib/util-function_2.11-6.43.0.jar:lib/util-jvm_2.11-6.43.0.jar:lib/util-lint_2.11-6.43.0.jar:lib/util-logging_2.11-6.43.0.jar:lib/util-registry_2.11-6.43.0.jar:lib/util-stats_2.11-6.43.0.jar:lib/woodstox-core-5.0.3.jar:lib/xmlenc-0.52.jar:lib/zkclient-0.10.jar:lib/zookeeper-3.4.6.jar:lib/zstd-jni-1.4.0-1.jar
2023-12-07 09:12:36,501 [Thread-8] (org.apache.zookeeper.Environment:100) INFO  Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2023-12-07 09:12:36,503 [Thread-8] (org.apache.zookeeper.Environment:100) INFO  Client environment:java.io.tmpdir=/tmp
2023-12-07 09:12:36,503 [Thread-8] (org.apache.zookeeper.Environment:100) INFO  Client environment:java.compiler=<NA>
2023-12-07 09:12:36,503 [Thread-8] (org.apache.zookeeper.Environment:100) INFO  Client environment:os.name=Linux
2023-12-07 09:12:36,503 [Thread-8] (org.apache.zookeeper.Environment:100) INFO  Client environment:os.arch=amd64
2023-12-07 09:12:36,503 [Thread-8] (org.apache.zookeeper.Environment:100) INFO  Client environment:os.version=5.4.17-2136.321.4.el7uek.x86_64
2023-12-07 09:12:36,503 [Thread-8] (org.apache.zookeeper.Environment:100) INFO  Client environment:user.name=secor
2023-12-07 09:12:36,504 [Thread-8] (org.apache.zookeeper.Environment:100) INFO  Client environment:user.home=/opt/secor
2023-12-07 09:12:36,504 [Thread-8] (org.apache.zookeeper.Environment:100) INFO  Client environment:user.dir=/opt/secor
2023-12-07 09:12:36,507 [Thread-8] (org.apache.zookeeper.ZooKeeper:438) INFO  Initiating client connection, connectString=10.0.6.64:2181,10.0.6.64:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@77718cdf
2023-12-07 09:12:36,507 [Thread-6] (org.apache.zookeeper.ZooKeeper:438) INFO  Initiating client connection, connectString=10.0.6.64:2181,10.0.6.64:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@2d4fa1ca
2023-12-07 09:12:36,507 [Thread-2] (org.apache.zookeeper.ZooKeeper:438) INFO  Initiating client connection, connectString=10.0.6.64:2181,10.0.6.64:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@1fb98e35
2023-12-07 09:12:36,507 [Thread-5] (org.apache.zookeeper.ZooKeeper:438) INFO  Initiating client connection, connectString=10.0.6.64:2181,10.0.6.64:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@4a21e255
2023-12-07 09:12:36,507 [Thread-3] (org.apache.zookeeper.ZooKeeper:438) INFO  Initiating client connection, connectString=10.0.6.64:2181,10.0.6.64:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@1080ea39
2023-12-07 09:12:36,507 [Thread-4] (org.apache.zookeeper.ZooKeeper:438) INFO  Initiating client connection, connectString=10.0.6.64:2181,10.0.6.64:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@7256234c
2023-12-07 09:12:36,507 [Thread-7] (org.apache.zookeeper.ZooKeeper:438) INFO  Initiating client connection, connectString=10.0.6.64:2181,10.0.6.64:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@2b3956bf
2023-12-07 09:12:36,524 [Thread-3-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:975) INFO  Opening socket connection to server 10.0.6.64/10.0.6.64:2181. Will not attempt to authenticate using SASL (unknown error)
2023-12-07 09:12:36,531 [Thread-3-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:852) INFO  Socket connection established to 10.0.6.64/10.0.6.64:2181, initiating session
2023-12-07 09:12:36,531 [Thread-8-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:975) INFO  Opening socket connection to server 10.0.6.64/10.0.6.64:2181. Will not attempt to authenticate using SASL (unknown error)
2023-12-07 09:12:36,532 [Thread-8-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:852) INFO  Socket connection established to 10.0.6.64/10.0.6.64:2181, initiating session
2023-12-07 09:12:36,533 [Thread-5-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:975) INFO  Opening socket connection to server 10.0.6.64/10.0.6.64:2181. Will not attempt to authenticate using SASL (unknown error)
2023-12-07 09:12:36,534 [Thread-5-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:852) INFO  Socket connection established to 10.0.6.64/10.0.6.64:2181, initiating session
2023-12-07 09:12:36,535 [Thread-4-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:975) INFO  Opening socket connection to server 10.0.6.64/10.0.6.64:2181. Will not attempt to authenticate using SASL (unknown error)
2023-12-07 09:12:36,535 [Thread-7-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:975) INFO  Opening socket connection to server 10.0.6.64/10.0.6.64:2181. Will not attempt to authenticate using SASL (unknown error)
2023-12-07 09:12:36,535 [Thread-4-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:852) INFO  Socket connection established to 10.0.6.64/10.0.6.64:2181, initiating session
2023-12-07 09:12:36,535 [Thread-7-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:852) INFO  Socket connection established to 10.0.6.64/10.0.6.64:2181, initiating session
2023-12-07 09:12:36,536 [Thread-6-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:975) INFO  Opening socket connection to server 10.0.6.64/10.0.6.64:2181. Will not attempt to authenticate using SASL (unknown error)
2023-12-07 09:12:36,536 [Thread-6-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:852) INFO  Socket connection established to 10.0.6.64/10.0.6.64:2181, initiating session
2023-12-07 09:12:36,537 [Thread-2-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:975) INFO  Opening socket connection to server 10.0.6.64/10.0.6.64:2181. Will not attempt to authenticate using SASL (unknown error)
2023-12-07 09:12:36,538 [Thread-6-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:1235) INFO  Session establishment complete on server 10.0.6.64/10.0.6.64:2181, sessionid = 0x10000002b480568, negotiated timeout = 40000
2023-12-07 09:12:36,538 [Thread-8-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:1235) INFO  Session establishment complete on server 10.0.6.64/10.0.6.64:2181, sessionid = 0x10000002b480564, negotiated timeout = 40000
2023-12-07 09:12:36,538 [Thread-5-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:1235) INFO  Session establishment complete on server 10.0.6.64/10.0.6.64:2181, sessionid = 0x10000002b480565, negotiated timeout = 40000
2023-12-07 09:12:36,538 [Thread-3-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:1235) INFO  Session establishment complete on server 10.0.6.64/10.0.6.64:2181, sessionid = 0x10000002b480563, negotiated timeout = 40000
2023-12-07 09:12:36,538 [Thread-2-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:852) INFO  Socket connection established to 10.0.6.64/10.0.6.64:2181, initiating session
2023-12-07 09:12:36,538 [Thread-4-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:1235) INFO  Session establishment complete on server 10.0.6.64/10.0.6.64:2181, sessionid = 0x10000002b480566, negotiated timeout = 40000
2023-12-07 09:12:36,538 [Thread-7-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:1235) INFO  Session establishment complete on server 10.0.6.64/10.0.6.64:2181, sessionid = 0x10000002b480567, negotiated timeout = 40000
2023-12-07 09:12:36,540 [Thread-2-SendThread(10.0.6.64:2181)] (org.apache.zookeeper.ClientCnxn$SendThread:1235) INFO  Session establishment complete on server 10.0.6.64/10.0.6.64:2181, sessionid = 0x10000002b480569, negotiated timeout = 40000
2023-12-07 09:12:36,544 [Thread-5-EventThread] (org.apache.curator.framework.state.ConnectionStateManager:228) INFO  State change: CONNECTED
2023-12-07 09:12:36,544 [Thread-3-EventThread] (org.apache.curator.framework.state.ConnectionStateManager:228) INFO  State change: CONNECTED
2023-12-07 09:12:36,544 [Thread-6-EventThread] (org.apache.curator.framework.state.ConnectionStateManager:228) INFO  State change: CONNECTED
2023-12-07 09:12:36,544 [Thread-4-EventThread] (org.apache.curator.framework.state.ConnectionStateManager:228) INFO  State change: CONNECTED
2023-12-07 09:12:36,545 [Thread-8-EventThread] (org.apache.curator.framework.state.ConnectionStateManager:228) INFO  State change: CONNECTED
2023-12-07 09:12:36,544 [Thread-2-EventThread] (org.apache.curator.framework.state.ConnectionStateManager:228) INFO  State change: CONNECTED
2023-12-07 09:12:36,547 [Thread-7-EventThread] (org.apache.curator.framework.state.ConnectionStateManager:228) INFO  State change: CONNECTED
2023-12-07 09:12:36,577 [Thread-4] (org.apache.kafka.common.config.AbstractConfig:347) INFO  ConsumerConfig values:
        allow.auto.create.topics = true
        auto.commit.interval.ms = 5000
        auto.offset.reset = earliest
        bootstrap.servers = [10.0.6.64:9092]
        check.crcs = true
        client.dns.lookup = default
        client.id = assess-events-backup-0.secor.secor.svc.cluster.local_1_12
        client.rack =
        connections.max.idle.ms = 540000
        default.api.timeout.ms = 60000
        enable.auto.commit = true
        exclude.internal.topics = true
        fetch.max.bytes = 52428800
        fetch.max.wait.ms = 500
        fetch.min.bytes = 1
        group.id = dev.telemetry.assess.events.backup
        group.instance.id = null
        heartbeat.interval.ms = 3000
        interceptor.classes = []
        internal.leave.group.on.close = true
        isolation.level = read_uncommitted
        key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
        max.partition.fetch.bytes = 1048576
        max.poll.interval.ms = 300000
        max.poll.records = 500
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
        receive.buffer.bytes = 65536
        reconnect.backoff.max.ms = 1000
        reconnect.backoff.ms = 50
        request.timeout.ms = 10000
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = null
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.mechanism = GSSAPI
        security.protocol = PLAINTEXT
        send.buffer.bytes = 131072
        session.timeout.ms = 10000
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
        ssl.endpoint.identification.algorithm = https
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLS
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS
        value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

2023-12-07 09:12:36,577 [Thread-3] (org.apache.kafka.common.config.AbstractConfig:347) INFO  ConsumerConfig values:
        allow.auto.create.topics = true
        auto.commit.interval.ms = 5000
        auto.offset.reset = earliest
        bootstrap.servers = [10.0.6.64:9092]
        check.crcs = true
        client.dns.lookup = default
        client.id = assess-events-backup-0.secor.secor.svc.cluster.local_1_11
        client.rack =
        connections.max.idle.ms = 540000
        default.api.timeout.ms = 60000
        enable.auto.commit = true
        exclude.internal.topics = true
        fetch.max.bytes = 52428800
        fetch.max.wait.ms = 500
        fetch.min.bytes = 1
        group.id = dev.telemetry.assess.events.backup
        group.instance.id = null
        heartbeat.interval.ms = 3000
        interceptor.classes = []
        internal.leave.group.on.close = true
        isolation.level = read_uncommitted
        key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
        max.partition.fetch.bytes = 1048576
        max.poll.interval.ms = 300000
        max.poll.records = 500
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
        receive.buffer.bytes = 65536
        reconnect.backoff.max.ms = 1000
        reconnect.backoff.ms = 50
        request.timeout.ms = 10000
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = null
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.mechanism = GSSAPI
        security.protocol = PLAINTEXT
        send.buffer.bytes = 131072
        session.timeout.ms = 10000
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
        ssl.endpoint.identification.algorithm = https
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLS
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS
        value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

2023-12-07 09:12:36,577 [Thread-5] (org.apache.kafka.common.config.AbstractConfig:347) INFO  ConsumerConfig values:
        allow.auto.create.topics = true
        auto.commit.interval.ms = 5000
        auto.offset.reset = earliest
        bootstrap.servers = [10.0.6.64:9092]
        check.crcs = true
        client.dns.lookup = default
        client.id = assess-events-backup-0.secor.secor.svc.cluster.local_1_13
        client.rack =
        connections.max.idle.ms = 540000
        default.api.timeout.ms = 60000
        enable.auto.commit = true
        exclude.internal.topics = true
        fetch.max.bytes = 52428800
        fetch.max.wait.ms = 500
        fetch.min.bytes = 1
        group.id = dev.telemetry.assess.events.backup
        group.instance.id = null
        heartbeat.interval.ms = 3000
        interceptor.classes = []
        internal.leave.group.on.close = true
        isolation.level = read_uncommitted
        key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
        max.partition.fetch.bytes = 1048576
        max.poll.interval.ms = 300000
        max.poll.records = 500
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
        receive.buffer.bytes = 65536
        reconnect.backoff.max.ms = 1000
        reconnect.backoff.ms = 50
        request.timeout.ms = 10000
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = null
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.mechanism = GSSAPI
        security.protocol = PLAINTEXT
        send.buffer.bytes = 131072
        session.timeout.ms = 10000
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
        ssl.endpoint.identification.algorithm = https
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLS
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS
        value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

2023-12-07 09:12:36,577 [Thread-6] (org.apache.kafka.common.config.AbstractConfig:347) INFO  ConsumerConfig values:
        allow.auto.create.topics = true
        auto.commit.interval.ms = 5000
        auto.offset.reset = earliest
        bootstrap.servers = [10.0.6.64:9092]
        check.crcs = true
        client.dns.lookup = default
        client.id = assess-events-backup-0.secor.secor.svc.cluster.local_1_14
        client.rack =
        connections.max.idle.ms = 540000
        default.api.timeout.ms = 60000
        enable.auto.commit = true
        exclude.internal.topics = true
        fetch.max.bytes = 52428800
        fetch.max.wait.ms = 500
        fetch.min.bytes = 1
        group.id = dev.telemetry.assess.events.backup
        group.instance.id = null
        heartbeat.interval.ms = 3000
        interceptor.classes = []
        internal.leave.group.on.close = true
        isolation.level = read_uncommitted
        key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
        max.partition.fetch.bytes = 1048576
        max.poll.interval.ms = 300000
        max.poll.records = 500
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
        receive.buffer.bytes = 65536
        reconnect.backoff.max.ms = 1000
        reconnect.backoff.ms = 50
        request.timeout.ms = 10000
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = null
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.mechanism = GSSAPI
        security.protocol = PLAINTEXT
        send.buffer.bytes = 131072
        session.timeout.ms = 10000
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
        ssl.endpoint.identification.algorithm = https
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLS
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS
        value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

2023-12-07 09:12:36,578 [Thread-7] (org.apache.kafka.common.config.AbstractConfig:347) INFO  ConsumerConfig values:
        allow.auto.create.topics = true
        auto.commit.interval.ms = 5000
        auto.offset.reset = earliest
        bootstrap.servers = [10.0.6.64:9092]
        check.crcs = true
        client.dns.lookup = default
        client.id = assess-events-backup-0.secor.secor.svc.cluster.local_1_15
        client.rack =
        connections.max.idle.ms = 540000
        default.api.timeout.ms = 60000
        enable.auto.commit = true
        exclude.internal.topics = true
        fetch.max.bytes = 52428800
        fetch.max.wait.ms = 500
        fetch.min.bytes = 1
        group.id = dev.telemetry.assess.events.backup
        group.instance.id = null
        heartbeat.interval.ms = 3000
        interceptor.classes = []
        internal.leave.group.on.close = true
        isolation.level = read_uncommitted
        key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
        max.partition.fetch.bytes = 1048576
        max.poll.interval.ms = 300000
        max.poll.records = 500
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
        receive.buffer.bytes = 65536
        reconnect.backoff.max.ms = 1000
        reconnect.backoff.ms = 50
        request.timeout.ms = 10000
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = null
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.mechanism = GSSAPI
        security.protocol = PLAINTEXT
        send.buffer.bytes = 131072
        session.timeout.ms = 10000
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
        ssl.endpoint.identification.algorithm = https
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLS
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS
        value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

2023-12-07 09:12:36,578 [Thread-8] (org.apache.kafka.common.config.AbstractConfig:347) INFO  ConsumerConfig values:
        allow.auto.create.topics = true
        auto.commit.interval.ms = 5000
        auto.offset.reset = earliest
        bootstrap.servers = [10.0.6.64:9092]
        check.crcs = true
        client.dns.lookup = default
        client.id = assess-events-backup-0.secor.secor.svc.cluster.local_1_16
        client.rack =
        connections.max.idle.ms = 540000
        default.api.timeout.ms = 60000
        enable.auto.commit = true
        exclude.internal.topics = true
        fetch.max.bytes = 52428800
        fetch.max.wait.ms = 500
        fetch.min.bytes = 1
        group.id = dev.telemetry.assess.events.backup
        group.instance.id = null
        heartbeat.interval.ms = 3000
        interceptor.classes = []
        internal.leave.group.on.close = true
        isolation.level = read_uncommitted
        key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
        max.partition.fetch.bytes = 1048576
        max.poll.interval.ms = 300000
        max.poll.records = 500
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
        receive.buffer.bytes = 65536
        reconnect.backoff.max.ms = 1000
        reconnect.backoff.ms = 50
        request.timeout.ms = 10000
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = null
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.mechanism = GSSAPI
        security.protocol = PLAINTEXT
        send.buffer.bytes = 131072
        session.timeout.ms = 10000
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
        ssl.endpoint.identification.algorithm = https
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLS
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS
        value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

2023-12-07 09:12:36,578 [Thread-2] (org.apache.kafka.common.config.AbstractConfig:347) INFO  ConsumerConfig values:
        allow.auto.create.topics = true
        auto.commit.interval.ms = 5000
        auto.offset.reset = earliest
        bootstrap.servers = [10.0.6.64:9092]
        check.crcs = true
        client.dns.lookup = default
        client.id = assess-events-backup-0.secor.secor.svc.cluster.local_1_10
        client.rack =
        connections.max.idle.ms = 540000
        default.api.timeout.ms = 60000
        enable.auto.commit = true
        exclude.internal.topics = true
        fetch.max.bytes = 52428800
        fetch.max.wait.ms = 500
        fetch.min.bytes = 1
        group.id = dev.telemetry.assess.events.backup
        group.instance.id = null
        heartbeat.interval.ms = 3000
        interceptor.classes = []
        internal.leave.group.on.close = true
        isolation.level = read_uncommitted
        key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
        max.partition.fetch.bytes = 1048576
        max.poll.interval.ms = 300000
        max.poll.records = 500
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
        receive.buffer.bytes = 65536
        reconnect.backoff.max.ms = 1000
        reconnect.backoff.ms = 50
        request.timeout.ms = 10000
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = null
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.mechanism = GSSAPI
        security.protocol = PLAINTEXT
        send.buffer.bytes = 131072
        session.timeout.ms = 10000
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
        ssl.endpoint.identification.algorithm = https
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLS
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS
        value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

2023-12-07 09:12:36,719 [Thread-2] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:117) INFO  Kafka version: 2.3.1
2023-12-07 09:12:36,720 [Thread-2] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:118) INFO  Kafka commitId: 18a913733fb71c01
2023-12-07 09:12:36,720 [Thread-2] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:119) INFO  Kafka startTimeMs: 1701940356717
2023-12-07 09:12:36,722 [Thread-5] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:117) INFO  Kafka version: 2.3.1
2023-12-07 09:12:36,722 [Thread-5] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:118) INFO  Kafka commitId: 18a913733fb71c01
2023-12-07 09:12:36,722 [Thread-5] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:119) INFO  Kafka startTimeMs: 1701940356720
2023-12-07 09:12:36,724 [Thread-8] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:117) INFO  Kafka version: 2.3.1
2023-12-07 09:12:36,724 [Thread-8] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:118) INFO  Kafka commitId: 18a913733fb71c01
2023-12-07 09:12:36,724 [Thread-8] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:119) INFO  Kafka startTimeMs: 1701940356723
2023-12-07 09:12:36,726 [Thread-7] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:117) INFO  Kafka version: 2.3.1
2023-12-07 09:12:36,726 [Thread-7] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:118) INFO  Kafka commitId: 18a913733fb71c01
2023-12-07 09:12:36,727 [Thread-7] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:119) INFO  Kafka startTimeMs: 1701940356726
2023-12-07 09:12:36,727 [Thread-6] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:117) INFO  Kafka version: 2.3.1
2023-12-07 09:12:36,727 [Thread-6] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:118) INFO  Kafka commitId: 18a913733fb71c01
2023-12-07 09:12:36,727 [Thread-6] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:119) INFO  Kafka startTimeMs: 1701940356726
2023-12-07 09:12:36,727 [Thread-4] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:117) INFO  Kafka version: 2.3.1
2023-12-07 09:12:36,728 [Thread-4] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:118) INFO  Kafka commitId: 18a913733fb71c01
2023-12-07 09:12:36,728 [Thread-4] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:119) INFO  Kafka startTimeMs: 1701940356725
2023-12-07 09:12:36,728 [Thread-3] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:117) INFO  Kafka version: 2.3.1
2023-12-07 09:12:36,728 [Thread-3] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:118) INFO  Kafka commitId: 18a913733fb71c01
2023-12-07 09:12:36,728 [Thread-3] (org.apache.kafka.common.utils.AppInfoParser$AppInfo:119) INFO  Kafka startTimeMs: 1701940356725
2023-12-07 09:12:36,859 [Thread-7] (com.pinterest.secor.consumer.Consumer:183) ERROR Thread failed
java.lang.RuntimeException: Failed to initialize the consumer
        at com.pinterest.secor.consumer.Consumer.run(Consumer.java:151)
Caused by: java.lang.ClassNotFoundException:
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at com.pinterest.secor.util.ReflectionUtil.createUploadManager(ReflectionUtil.java:58)
        at com.pinterest.secor.consumer.Consumer.init(Consumer.java:95)
        at com.pinterest.secor.consumer.Consumer.run(Consumer.java:148)
2023-12-07 09:12:36,859 [Thread-5] (com.pinterest.secor.consumer.Consumer:183) ERROR Thread failed
java.lang.RuntimeException: Failed to initialize the consumer
        at com.pinterest.secor.consumer.Consumer.run(Consumer.java:151)
Caused by: java.lang.ClassNotFoundException:
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at com.pinterest.secor.util.ReflectionUtil.createUploadManager(ReflectionUtil.java:58)
        at com.pinterest.secor.consumer.Consumer.init(Consumer.java:95)
        at com.pinterest.secor.consumer.Consumer.run(Consumer.java:148)
2023-12-07 09:12:36,859 [Thread-2] (com.pinterest.secor.consumer.Consumer:183) ERROR Thread failed
java.lang.RuntimeException: Failed to initialize the consumer
        at com.pinterest.secor.consumer.Consumer.run(Consumer.java:151)
Caused by: java.lang.ClassNotFoundException:
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at com.pinterest.secor.util.ReflectionUtil.createUploadManager(ReflectionUtil.java:58)
        at com.pinterest.secor.consumer.Consumer.init(Consumer.java:95)
        at com.pinterest.secor.consumer.Consumer.run(Consumer.java:148)
2023-12-07 09:12:36,859 [Thread-6] (com.pinterest.secor.consumer.Consumer:183) ERROR Thread failed
java.lang.RuntimeException: Failed to initialize the consumer
        at com.pinterest.secor.consumer.Consumer.run(Consumer.java:151)
Caused by: java.lang.ClassNotFoundException:
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at com.pinterest.secor.util.ReflectionUtil.createUploadManager(ReflectionUtil.java:58)
        at com.pinterest.secor.consumer.Consumer.init(Consumer.java:95)
        at com.pinterest.secor.consumer.Consumer.run(Consumer.java:148)
2023-12-07 09:12:36,859 [Thread-4] (com.pinterest.secor.consumer.Consumer:183) ERROR Thread failed
java.lang.RuntimeException: Failed to initialize the consumer
        at com.pinterest.secor.consumer.Consumer.run(Consumer.java:151)
Caused by: java.lang.ClassNotFoundException:
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at com.pinterest.secor.util.ReflectionUtil.createUploadManager(ReflectionUtil.java:58)
        at com.pinterest.secor.consumer.Consumer.init(Consumer.java:95)
        at com.pinterest.secor.consumer.Consumer.run(Consumer.java:148)
2023-12-07 09:12:36,859 [Thread-3] (com.pinterest.secor.consumer.Consumer:183) ERROR Thread failed
java.lang.RuntimeException: Failed to initialize the consumer
        at com.pinterest.secor.consumer.Consumer.run(Consumer.java:151)
Caused by: java.lang.ClassNotFoundException:
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at com.pinterest.secor.util.ReflectionUtil.createUploadManager(ReflectionUtil.java:58)
        at com.pinterest.secor.consumer.Consumer.init(Consumer.java:95)
        at com.pinterest.secor.consumer.Consumer.run(Consumer.java:148)
2023-12-07 09:12:36,859 [Thread-8] (com.pinterest.secor.consumer.Consumer:183) ERROR Thread failed
java.lang.RuntimeException: Failed to initialize the consumer
        at com.pinterest.secor.consumer.Consumer.run(Consumer.java:151)
Caused by: java.lang.ClassNotFoundException:
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at com.pinterest.secor.util.ReflectionUtil.createUploadManager(ReflectionUtil.java:58)
        at com.pinterest.secor.consumer.Consumer.init(Consumer.java:95)
        at com.pinterest.secor.consumer.Consumer.run(Consumer.java:148)
2023-12-07 09:12:36,865 [Thread-0] (com.pinterest.secor.common.ShutdownHookRegistry:55) INFO  Running hooks for priority 10

Munishdev97 Dec 7, 2023
Author

@sowmya-dixit, Can you share the release tag used in build pipeline.

Munishdev97 · 2023-12-08T05:59:48Z

Munishdev97
Dec 8, 2023
Author

@sowmya-dixit Any update about the release tag for build pipeline.

29 replies

sowmya-dixit Dec 15, 2023

@vj4a Great.
You can verify it in 2 ways

Push some sample data manually to the respective kafka topic to quickly verify.
Use portal to generate such events e.g., taking some assessments to generate assess events, but this needs complete setup to be working.

Note: upload policy depends on these 2 property secor.max.file.age.seconds & secor.max.file.size.bytes under secor.properties file, it uploads data after max.file.age.seconds or if data size reaches max.file.size.bytes.

vj4a Dec 18, 2023

@sowmya-dixit , we tried pushing data into the topic and it is getting uploaded to the mentioned bucket. Will add the steps taken for fix here.

mohitga Dec 18, 2023
Maintainer

@vj4a great

akulavasu Dec 21, 2023

@sowmya-dixit @gandham-santhosh @santhosh-tg
Please review the PR that adds the kafka listener in server.properties during kafka installation in provision/datapipeline/kafka
Sunbird-Obsrv/sunbird-data-pipeline#1635

Answer selected by Munishdev97

gandham-santhosh Jan 16, 2024

@sowmya-dixit can you please review the PR and merge so that we can close this thread

mohitga Jan 19, 2024
Maintainer

@sowmya-dixit please conform is the PR is merged?

mohitga · 2024-01-29T05:10:47Z

mohitga
Jan 29, 2024
Maintainer

thanks @sowmya-dixit,

@gandham-santhosh the PR is merged can you take is forward?

1 reply

gandham-santhosh Jan 29, 2024

@Munishdev97 please mark the thread and close as PR is merged

Deploy/DataPipeline/Secor - Pod CrashLoopBackOff Error #662

Uh oh!

Replies: 4 comments · 35 replies

Uh oh!

Uh oh!

Munishdev97 Dec 7, 2023 Author

Uh oh!

Uh oh!

Munishdev97 Dec 7, 2023 Author

Uh oh!

Uh oh!

Munishdev97 Dec 7, 2023 Author

Uh oh!

Munishdev97 Dec 7, 2023 Author

Uh oh!

Munishdev97 Dec 8, 2023 Author

Uh oh!

Uh oh!

Uh oh!

mohitga Dec 18, 2023 Maintainer

Uh oh!

Uh oh!

Uh oh!

mohitga Jan 19, 2024 Maintainer

Uh oh!

mohitga Jan 29, 2024 Maintainer

Uh oh!

Replies: 4 comments 35 replies

Munishdev97
Dec 7, 2023
Author

Munishdev97 Dec 7, 2023
Author

Munishdev97 Dec 7, 2023
Author

Munishdev97 Dec 7, 2023
Author

Munishdev97
Dec 8, 2023
Author

mohitga Dec 18, 2023
Maintainer

mohitga Jan 19, 2024
Maintainer

mohitga
Jan 29, 2024
Maintainer