V3.0.10.6

StreamNative Weekly Release Notes v3.0.10.6

General Changes

Apache Pulsar

(#23594) [fix] [broker] No longer allow creating subscription that contains slash

232f0ef492 remove unnecessary codes

(#24366) [fix][broker]Fix deadlock when compaction and topic deletion execute concurrently

(#24350) [fix][broker] Fix issue that topic policies was deleted after a sub topic deleted, even if the partitioned topic still exists

(#24384) [fix][ml]Revert a behavior change of releasing idle offloaded ledger handle: only release idle BlobStoreBackedReadHandle

(#24397) [improve][misc] Upgrade Netty to 4.1.122.Final and tcnative to 2.0.72.Final

(#24391) [improve][broker] Add managedCursor/LedgerInfoCompressionType settings to broker.conf

(#24392) [improve][broker] Make maxBatchDeletedIndexToPersist configurable and document other related configs

(#24386) [improve][broker] Added synchronized for sendMessages in Non-Persistent message dispatchers

(#24381) [improve][ml]Release idle offloaded read handle only the ref count is 0

(#19783) [improve][offloaders] Automatically evict Offloaded Ledgers from memory

(#24360) [fix][broker] expose consumer name for partitioned topic stats

(#24359) [improve][broker]Improve the log when encountered in-flight read limitation

(#24354) [fix][io] Acknowledge RabbitMQ message after processing the message successfully

(#24352) [fix][broker] Ignore metadata changes when broker is not in the Started state

(#24190) [fix][broker] Resolve the issue of frequent updates in message expiration deletion rate

(#24338) [fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent issue

(#24331) [fix][offload] Complete the future outside of the reading loop in BlobStoreBackedReadHandleImplV2.readAsync

(#24324) [fix][test] Fix flaky AutoScaledReceiverQueueSizeTest.testNegativeClientMemory

(#24316) [fix][io] Fix kinesis avro bytes handling

a75d16fa6e Fix checkstyle errors after previous cherry-picks

(#24344) [improve][ml] Offload ledgers without check ledger length

(#24286) [fix][broker]Non-global topic policies and global topic policies overwrite each other

(#24279) [fix][broker]Global topic policies do not affect after unloading topic and persistence global topic policies never affect

(#24349) [fix][io][branch-3.0] Backport Kinesis Sink custom native executable support #23762

(#24317) [fix][io][branch-3.0]Pulsar-SQL: Fix classcast ex when decode decimal value

(#24313) [fix][broker] Fix potential deadlock when creating partitioned topic

(#24293) [fix][broker] fix wrong method name checkTopicExists.

(#24307) [fix][build] Ensure that buildtools is Java 8 compatible and fix remaining compatibility issue

(#24304) [fix][test] Simplify BetweenTestClassesListenerAdapter and fix issue with BeforeTest/AfterTest annotations

(#24289) [improve][io] Add configuration parameter for disabling aggregation for Kinesis Producers

(#24302) [improve] Upgrade pulsar-client-python to 3.7.0 in Docker image

(#24299) [fix][test] Fix more Netty ByteBuf leaks in tests

(#24297) [fix][io] Fix SyntaxWarning in Pulsar Python functions

(#24282) [fix][client] Fix producer publishing getting stuck after message with incompatible schema is discarded

(#24283) [cleanup][test] Remove unused parameter from deleteNamespaceWithRetry method in MockedPulsarServiceBaseTest

(#24263) [improve][build] Upgrade zstd version from 1.5.2-3 to 1.5.7-3

(#24281) [fix][test] Fix multiple ByteBuf leaks in tests

(#24275) [fix][broker] Fix HashedWheelTimer leak in PulsarService by stopping it in shutdown

(#24274) [fix][misc] Fix ByteBuf leak in SchemaUtils

(#24254) [fix][broker]Fix incorrect priority between topic policies and global topic policies

(#24266) [improve][ci] Disable detailed console logging for integration tests in CI

(#24261) [fix][test] Fix flaky ManagedCursorTest.testLastActiveAfterResetCursor and disable failing SchemaTest

(#24244) [fix][test] Fix flaky ManagedCursorTest.testSkipEntriesWithIndividualDeletedMessages

(#24248) [improve][io][kca] support fully-qualified topic names in source records

(#24260) [improve][build] Upgrade Gradle Develocity Maven Extension dependencies

(#24258) [fix][test] Fix TestNG BetweenTestClassesListenerAdapter listener

(#24257) [fix][broker] Unregister non-static metrics collectors registered in Prometheus default registry

bebc3b0d6a Fix checkstyle issue in previous cherry-pick c2d33cc

(#24178) [fix][broker]fix memory leak, messages lost, incorrect replication state if using multiple schema versions(auto_produce)

(#24219) [improve][broker]Improve the feature “Optimize subscription seek (cursor reset) by timestamp”: search less entries

(#23919) [fix][broker] Fix seeking by timestamp can be reset the cursor position to earliest

(#22792) [improve][broker] Optimize subscription seek (cursor reset) by timestamp

(#24243) [improve][build] Upgrade SpotBugs to 4.9.x

(#24240) [improve][build] Upgrade to jacoco 0.8.13

(#24237) [improve][build] Upgrade Lombok to 1.18.38 to support JDK 24

(#24221) [improve][io] support kafka connect transforms and predicates

(#24230) [improve][client]Improve transaction log when a TXN command timeout

(#24223) [fix][broker] Orphan schema after disabled a cluster for a namespace

(#24228) [fix][broker] Fix ByteBuf memory leak in REST API for publishing messages

b4e1c93d2c Fix presto-distribution/LICENSE

(#24184) [fix][client] Fix incorrect producer.getPendingQueueSize due to incomplete queue implementation

(#24214) [improve] Upgrade Netty to 4.1.121.Final

(#24212) [fix][test] Fix flaky BatchMessageWithBatchIndexLevelTest.testBatchMessageAck

(#24218) [fix][test] Fix multiple resource leaks in tests

(#24187) [improve][client] validate ClientConfigurationData earlier to avoid resource leaks

(#24216) [fix][broker] Fix HealthChecker deadlock in shutdown

(#24209) [fix][broker] Fix tenant creation and update with null value

(#24192) [fix][admin] Backlog quota’s policy is null which causes a NPE

(#24210) [fix][broker] Fix broker shutdown delay by resolving hanging health checks

(#24207) [fix][broker] Fix compaction service log’s wrong condition

(#24204) [fix][test] Fix resource leaks in ProxyTest and fix invalid tests

(#24201) [improve][io] Upgrade Kafka client and compatible Confluent platform version

(#24118))) Revert “[fix][broker] Add topic consistency check

(#24154))) Revert “[fix][broker] Directly query single topic existence when the topic is partitioned

(#24032) [fix][broker] Fix missing validation when setting retention policy on topic level

(#24098) [fix][ml] Skip deleting cursor if it was already deleted before calling unsubscribe

(#24181) [fix][proxy] Fix incorrect client error when calling get topic metadata

(#24158) [fix][proxy] Propagate client connection feature flags through Pulsar Proxy to Broker

(#24103) [fix][schema] Reject unsupported Avro schema types during schema registration

(#24091) [fix][broker] Fix some problems in calculate totalAvailableBookies in method getExcludedBookiesWithIsolationGroups when some bookies belongs to multiple isolation groups.

(#21320) [fix][bk] Fix the var name for IsolationGroups

(#24171) [improve][test] Use configured session timeout for MockZooKeeper and TestZKServer in PulsarTestContext

(#24172) [fix][test] Improve reliability of IncrementPartitionsTest

(#24170) [fix][test]flaky-test:ManagedLedgerInterceptorImplTest.testManagedLedgerPayloadInputProcessorFailure

(#23980) [fix][broker] Consumer stuck when delete subscription __compaction failed

(#24167) [fix][ml] Fix ML thread blocking issue in internalGetPartitionedStats API

(#24166) [fix][test] Fix invalid test CompactionTest.testDeleteCompactedLedgerWithSlowAck

(#24150) [fix][broker] The feature brokerDeleteInactivePartitionedTopicMetadataEnabled leaves orphan topic policies and topic schemas

(#24154) [fix][broker] Directly query single topic existence when the topic is partitioned

(#24118) [fix][broker] Add topic consistency check

(#24056) [fix][test] Update partitioned topic subscription assertions in IncrementPartitionsTest

(#24033) [cleanup][misc] Add override annotation

(#24161) [fix][test] Fix flaky BrokerServiceChaosTest.testFetchPartitionedTopicMetadataWithCacheRefresh

(#24162) [fix][test] Fix flaky BrokerServiceChaosTest

1035accffd Bump version to next snapshot version

(#24097) [fix] [broker] topics infinitely failed to delete after remove cluster from replicated clusters modifying when using partitioned system topic

(#22261) [fix] Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 in /pulsar-function-go

(#24132) [fix][io] Fix KinesisSink json flattening for AVRO’s SchemaType.BYTES

(#20984) [fix][broker] Fix get outdated compactedTopicContext after compactionHorizon has been updated

(#20697) [improve][broker] Improve CompactedTopicImpl lock

(#24131) [fix][ml] Return 1 when bytes size is 0 or negative for entry count estimation

(#24128) [improve][io] Enhance Kafka connector logging with focused bootstrap server information

(#24125) [fix][ml] Don’t estimate number of entries when ledgers are empty, return 1 instead

(#24123) [improve][client] Prevent NullPointException when closing ClientCredentialsFlow

(#24124) [improve][io] Remove sleep when sourceTask.poll of kafka return null

(#24116) [improve][broker] Change topic exists log to warn

(#24104) [fix][client] Pattern subscription regression when broker-side evaluation is disabled

(#24100) [fix][client] Fix consumer leak when thread is interrupted before subscribe completes

(#24089) [fix][ml] Fix issues in estimateEntryCountBySize

(#24073) [improve][broker] Optimize message expiration rate repeated update issues

(#24087) [fix][broker] Avoid IllegalStateException when marker_type field is not set in publishing

(#24083) [fix][ci] Bump dependency-check to 12.1.0 to fix OWASP Dependency Check job

(#24082) [clean][client] Clean code for the construction of retry/dead letter topic name

(#24079) [fix][broker] Fix NPE while publishing Metadata-Event with not init producer

(#24080) [fix][broker] Fix Metadata event synchronizer should not fail with bad version

(#24081) [fix][broker] Fix Metadata Event Synchronizer producer creation retry so that the producer gets created eventually

(#24048) [fix][broker] Fix UnsupportedOperationException while setting subscription level dispatch rate policy

(#24054) [fix][ml] Corrected pulsar_storage_size metric to not multiply offloaded storage by the write quorum

(#24067) [fix][broker] http metric endpoint get compaction latency stats always be 0

(#24064) [improve][broker] Optimize ThresholdShedder with improved boundary checks and parameter reuse

(#24055) [fix] Avoid negative estimated entry count

(#24060) [improve][monitor] Add version=0.0.4 to /metrics content type for Prometheus 3.x compatibility

(#24059) [fix][client] Copy eventTime to retry letter topic and DLQ messages

(#24061) [fix][client] Fix building broken batched message when publishing

(#24063) [fix][broker]Fix failed consumption after loaded up a terminated topic

(#24072) [fix][broker] Pattern subscription doesn’t work when the pattern excludes the topic domain.

ebce3b07ed Fix presto LICENSE after Netty 4.1.119.Final upgrade

(#24049) [improve] Upgrade Netty to 4.1.119.Final

(#23975) [fix][broker] Add expire check for replicator

(#24023) [fix][doc] fix doc related to chunk message feature.

8437af98eb Bump version to next snapshot version

(#23962) [improve][ml] Use lock-free queue in InflightReadsLimiter since there’s no concurrent access

(#23978) [improve][cli] Support additional msg metadata for V1 topic on peek message cmd

(#24014) [fix][broker] Fix BucketDelayedDeliveryTracker thread safety

(#24019) [fix][test]Fix flaky test V1_ProducerConsumerTest.testConcurrentConsumerReceiveWhileReconnect

(#24011) [fix][test] Fix flaky test OneWayReplicatorUsingGlobalZKTest.testConfigReplicationStartAt

(#23931) [improve] [broker] Make the estimated entry size more accurate

(#24004) [improve][ci] Upgrade Gradle Develocity Maven Extension to 1.23.1

(#23697) [fix][broker] Geo Replication lost messages or frequently fails due to Deduplication is not appropriate for Geo-Replication

(#24006) [fix][broker] fix broker identifying incorrect stuck topic

(#23286) [improve][broker] Fix non-persistent system topic schema compatibility

(#23881) [improve][fn] Set default tenant and namespace for ListFunctions cmd

(#23730) [fix][admin] Verify is policies read only before revoke permissions on topic

(#24003) [improve][test] Upgrade Testcontainers to 1.20.4 and docker-java to 3.4.0

KoP

[refactor] Create PartitionLog only after ProducerStateManager#recover is done

StreamNative Pulsar Plugins

Add test to verify the sts module

Fix the backup tool can not use sts to authenticate

pulsarctl

e7d5e82 Use snstage docker image

(#1778) Fix jwt cve

(#1711) fix code check

(#1704) feat: Subscription get message by id json output

(#1699) Update subscription get message by id typo lederId to ledgerId

(#1585) fix: upgrade golang version to fix CVE

(#1587) Setup go version to 1.22 fix cve

(#1549) update pulsar-client-go to master latest commit

2af1258 fix ci

(#1537) Bump the pulsar-client-go to the master version

6f25051 Fix TestDeleteNonExistPartitionedTopic

(#1509) Fix json marshal error for Secrets and UserConfigs when creating/updating functions

(#1455) Support create token with headers

(#1478) Upgrade the dependency version to fix vulnerabilities

(#1447) Add trivy scan workflow to avoid vulnerabilities

(#1451) [fix] Upgrade go version to 1.21 to fix CVE-2023-24538

(#1362) fix source test typo

(#1360) fix source test

(#1419) Auth SN docker hub

(#1405) Support no auth context

(#1393) fix token

(#1402) Add docker hub login

(#1398) Auth SN docker hub

5cb0593 Disable bk unit test and fix it later ---

(#1257) Add method to mark bookie readonly

(#1328) Build arm64 linux executable binary artifact

(#1305) Update jose2go to fix GHSA-mhpq-9638-x6pw

(#1268) Update golang.org/x/net

(#1205) Replace apache pulsar client go repo on 3.0 branch

(#1084) Fixed remove auth plugin suffix

(#1072) Removed error char

(#1065) Bump pulsar version to 3.0.0.1

(#1067) fix: Fix TestUpdateTopicNotExist and TestUpdateNonPartitionedTopic

Function Mesh Worker Service

Support invalid name