[CDC] Tserver crash seen (on 2.17.2.0-b23) with CDC and tablet splitting enabled

Jira Link: DB-5060

Description

Run : test_cdc_with_tablet_splitting_with_debezium

Logs: All_Logs_Downloaded.zip

Stacktrace: core_yb-tserver.20355.backtrace.txt

  thread #10, stop reason = signal 0
    frame #0: 0x00007f1a9cbf800f libpthread.so.0`[email protected]@GLIBC_2.3.2 at pthread_cond_wait.S:185
    frame #1: 0x0000558d1c192445 yb-tserver`yb::Synchronizer::WaitUntil(std::__1::chrono::time_point<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>> const&) [inlined] void std::__1::condition_variable::wait<yb::Synchronizer::WaitUntil(std::__1::chrono::time_point<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>> const&)::$_0>(this=0x0000558d341d1db8, __lk=<unavailable>, __pred=(unnamed class) @ rbx)::$_0) at __mutex_base:398:9
    frame #2: 0x0000558d1c192422 yb-tserver`yb::Synchronizer::WaitUntil(this=0x0000558d341d1d90, time=0x00007f1a931b5aa0) at async_util.cc:75:11
    frame #3: 0x0000558d1b0c085d yb-tserver`yb::Status yb::client::YBClient::Data::SyncLeaderMasterRpc<yb::master::MasterClientProxy, yb::master::GetTableLocationsRequestPB, yb::master::GetTableLocationsResponsePB>(std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::master::GetTableLocationsRequestPB const&, yb::master::GetTableLocationsResponsePB*, char const*, void (yb::master::MasterClientProxy::* const&)(yb::master::GetTableLocationsRequestPB const&, yb::master::GetTableLocationsResponsePB*, yb::rpc::RpcController*, std::__1::function<void ()>) const, int*) [inlined] yb::Synchronizer::Wait(this=<unavailable>) at async_util.h:82:12
    frame #4: 0x0000558d1b0c0849 yb-tserver`yb::Status yb::client::YBClient::Data::SyncLeaderMasterRpc<yb::master::MasterClientProxy, yb::master::GetTableLocationsRequestPB, yb::master::GetTableLocationsResponsePB>(this=0x0000558d31f9c840, deadline=<unavailable>, req=<unavailable>, resp=<unavailable>, func_name=<unavailable>, func=<unavailable>, attempts=0x0000000000000000)(yb::master::GetTableLocationsRequestPB const&, yb::master::GetTableLocationsResponsePB*, yb::rpc::RpcController*, std::__1::function<void ()>) const, int*) at client-internal.cc:199:37
    frame #5: 0x0000558d1b0f8905 yb-tserver`yb::client::YBClient::GetTablets(this=0x0000558d31d3a628, table_name=<unavailable>, max_tablets=0, tablets=0x00007f1a931b5c80, partition_list_version=0x0000000000000000, require_tablets_running=(value_ = false), include_inactive=(value_ = false)) at client.cc:1810:3
    frame #6: 0x0000558d1b14649c yb-tserver`yb::client::TableIterator::TableIterator(this=0x00007f1a931b5ff0, table=0x0000558d3427a018, options=0x00007f1a931b61a0) at table_handle.cc:212:3
    frame #7: 0x0000558d1afd6514 yb-tserver`yb::cdc::CDCServiceImpl::UpdateCDCMetrics() [inlined] yb::client::TableRange::begin(this=0x00007f1a931b6190) const at table_handle.h:220:41
    frame #8: 0x0000558d1afd64fa yb-tserver`yb::cdc::CDCServiceImpl::UpdateCDCMetrics(this=0x0000558d31d80a20) at cdc_service.cc:1848:24
    frame #9: 0x0000558d1afd3128 yb-tserver`yb::cdc::CDCServiceImpl::UpdatePeersAndMetrics(this=0x0000558d31d80a20) at cdc_service.cc:2554:7
    frame #10: 0x0000558d1afd535c yb-tserver`void* std::__1::__thread_proxy[abi:v15003]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (yb::cdc::CDCServiceImpl::*)(), yb::cdc::CDCServiceImpl*>>(void*) [inlined] decltype(__f=0x0000558d320d3b48, __a0=0x0000558d320d3b58).*std::declval<void (yb::cdc::CDCServiceImpl::*)()>()()) std::__1::__invoke[abi:v15003]<void (yb::cdc::CDCServiceImpl::*)(), yb::cdc::CDCServiceImpl*, void>(void (yb::cdc::CDCServiceImpl::*&&)(), yb::cdc::CDCServiceImpl*&&) at invoke.h:359:23
    frame #11: 0x0000558d1afd5342 yb-tserver`void* std::__1::__thread_proxy[abi:v15003]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (yb::cdc::CDCServiceImpl::*)(), yb::cdc::CDCServiceImpl*>>(void*) [inlined] void std::__1::__thread_execute[abi:v15003]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (yb::cdc::CDCServiceImpl::*)(), yb::cdc::CDCServiceImpl*, 2ul>(__t=0x0000558d320d3b40, (null)=<unavailable>)(), yb::cdc::CDCServiceImpl*>&, std::__1::__tuple_indices<2ul>) at thread:284:5
    frame #12: 0x0000558d1afd5342 yb-tserver`void* std::__1::__thread_proxy[abi:v15003]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (yb::cdc::CDCServiceImpl::*)(), yb::cdc::CDCServiceImpl*>>(__vp=0x0000558d320d3b40) at thread:295:5
    frame #13: 0x00007f1a9cbf3694 libpthread.so.0`start_thread(arg=0x00007f1a931be700) at pthread_create.c:333
    frame #14: 0x00007f1a9d0f541d libc.so.6`__clone at clone.S:109

thread #118, stop reason = signal 0
    frame #0: 0x0000558d1bc80180 yb-tserver`rocksdb::RandomAccessFileReader::ReadAndValidate(this=0x0000558d8c9bc750, offset=1055564, n=3375, result=0x00007f1a8619a5b0, scratch="", validator=0x0000558d7f50c298) at file_reader_writer.cc:77
    frame #1: 0x0000558d1bc53ef4 yb-tserver`rocksdb::ReadBlockContents(rocksdb::RandomAccessFileReader*, rocksdb::Footer const&, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockContents*, rocksdb::Env*, std::__1::shared_ptr<yb::MemTracker> const&, bool) at format.cc:389:15
    frame #2: 0x0000558d1bc53dd7 yb-tserver`rocksdb::ReadBlockContents(file=0x0000558d8c9bc750, footer=0x0000558d7f50c1d8, options=0x0000558e476906a8, handle=0x00007f1a8619a790, contents=0x00007f1a8619a6b8, env=<unavailable>, mem_tracker=std::__1::shared_ptr<yb::MemTracker>::element_type @ 0x0000558d4179e720, decompression_requested=<unavailable>) at format.cc:461:12
    frame #3: 0x0000558d1bc49abb yb-tserver`rocksdb::block_based_table::ReadBlockFromFile(file=<unavailable>, footer=<unavailable>, options=<unavailable>, handle=<unavailable>, result=0x00007f1a8619a7e8, env=<unavailable>, mem_tracker=std::__1::shared_ptr<yb::MemTracker>::element_type @ 0x0000558d4179e720, do_uncompress=<unavailable>) at block_based_table_internal.h:40:14
    frame #4: 0x0000558d1bc48bed yb-tserver`rocksdb::BlockBasedTable::RetrieveBlock(this=0x0000558e10534630, ro=0x0000558e476906a8, index_value=<unavailable>, block_type=<unavailable>, use_cache=<unavailable>) at block_based_table_reader.cc:1212:9
    frame #5: 0x0000558d1bc4b854 yb-tserver`rocksdb::BlockBasedTable::NewDataBlockIterator(this=0x0000558e10534630, ro=0x0000558e476906a8, index_value=0x00007f1a8619a960, block_type=kData, input_iter=0x0000000000000000) at block_based_table_reader.cc:1258:16
    frame #6: 0x0000558d1bc72326 yb-tserver`rocksdb::(anonymous namespace)::TwoLevelIterator::InitDataBlock(this=0x0000558d3edcb0e8) at two_level_iterator.cc:233:40
    frame #7: 0x0000558d1bc719d9 yb-tserver`rocksdb::(anonymous namespace)::TwoLevelIterator::Seek(this=0x0000558d3edcb0e8, target=0x00007f1a8619aab8) at two_level_iterator.cc:122:3
    frame #8: 0x0000558d1bc60614 yb-tserver`rocksdb::MergingIterator::Seek(yb::Slice const&) [inlined] rocksdb::IteratorWrapper::Seek(this=0x0000558d3edcaf20, k=0x00007f1a8619aab8) at iterator_wrapper.h:126:53
    frame #9: 0x0000558d1bc60607 yb-tserver`rocksdb::MergingIterator::Seek(this=0x0000558d3edcaea0, target=0x00007f1a8619aab8) at merger.cc:137:21
    frame #10: 0x0000558d1bbbe98d yb-tserver`rocksdb::DBIter::Seek(this=0x0000558d3edcad38, target=0x00007f1a8619add0) at db_iter.cc:762:12
    frame #11: 0x0000558d1b0341a0 yb-tserver`yb::cdc::ProcessIntents(yb::OpId const&, yb::StronglyTypedUuid<yb::TransactionId_Tag> const&, yb::cdc::StreamMetadata const&, std::__1::unordered_map<unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>> const&, std::__1::unordered_map<unsigned int, std::__1::vector<yb::master::PgAttributePB, std::__1::allocator<yb::master::PgAttributePB>>, std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, std::__1::vector<yb::master::PgAttributePB, std::__1::allocator<yb::master::PgAttributePB>>>>> const&, yb::cdc::GetChangesResponsePB*, yb::ScopedTrackedConsumption*, yb::cdc::CDCSDKCheckpointPB*, std::__1::shared_ptr<yb::tablet::TabletPeer> const&, std::__1::vector<yb::docdb::IntentKeyValueForCDC, std::__1::allocator<yb::docdb::IntentKeyValueForCDC>>*, yb::docdb::ApplyTransactionState*, yb::client::YBClient*, std::__1::shared_ptr<yb::Schema>*, unsigned int*, unsigned long const&) at docdb.cc:839:23
    frame #12: 0x0000558d1b033e37 yb-tserver`yb::cdc::ProcessIntents(yb::OpId const&, yb::StronglyTypedUuid<yb::TransactionId_Tag> const&, yb::cdc::StreamMetadata const&, std::__1::unordered_map<unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>> const&, std::__1::unordered_map<unsigned int, std::__1::vector<yb::master::PgAttributePB, std::__1::allocator<yb::master::PgAttributePB>>, std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, std::__1::vector<yb::master::PgAttributePB, std::__1::allocator<yb::master::PgAttributePB>>>>> const&, yb::cdc::GetChangesResponsePB*, yb::ScopedTrackedConsumption*, yb::cdc::CDCSDKCheckpointPB*, std::__1::shared_ptr<yb::tablet::TabletPeer> const&, std::__1::vector<yb::docdb::IntentKeyValueForCDC, std::__1::allocator<yb::docdb::IntentKeyValueForCDC>>*, yb::docdb::ApplyTransactionState*, yb::client::YBClient*, std::__1::shared_ptr<yb::Schema>*, unsigned int*, unsigned long const&) at tablet.cc:1941:22
    frame #13: 0x0000558d1b033c3f yb-tserver`yb::cdc::ProcessIntents(op_id=0x00007f1a8619b240, transaction_id=0x00007f1a8619b3b0, metadata=0x00007f1a8619bd50, enum_oid_label_map=0x00007f1a8619bca8, composite_atts_map=0x00007f1a8619bce8, resp=0x0000558e67d4fe70, consumption=0x00007f1a8619b7f0, checkpoint=0x00007f1a8619b848, tablet_peer=std::__1::shared_ptr<yb::tablet::TabletPeer>::element_type @ 0x0000558d61299800, keyValueIntents=0x00007f1a8619b530, stream_state=0x00007f1a8619b910, client=0x0000558d31d3a628, cached_schema=0x00007f1a8619bc90, cached_schema_version=0x00007f1a8619bc88, commit_time=0x00007f1a8619b810) at cdcsdk_producer.cc:1031:3
    frame #14: 0x0000558d1b03b5a3 yb-tserver`yb::cdc::GetChangesForCDCSDK(stream_id=<unavailable>, tablet_id="62e7f38ba5fa468d99720ee0a78d7bd4", from_op_id=0x00007f1a8619bbe0, stream_metadata=0x00007f1a8619bd50, tablet_peer=std::__1::shared_ptr<yb::tablet::TabletPeer>::element_type @ 0x0000558d61299800, mem_tracker=std::__1::shared_ptr<yb::MemTracker>::element_type @ 0x0000558d35c3b360, enum_oid_label_map=0x00007f1a8619bca8, composite_atts_map=0x00007f1a8619bce8, client=0x0000558d31d3a628, msgs_holder=0x00007f1a8619bb70, resp=0x0000558e67d4fe70, commit_timestamp="", cached_schema=0x00007f1a8619bc90, cached_schema_version=0x00007f1a8619bc88, last_streamed_op_id=0x00007f1a8619ba80, last_readable_opid_index=0x00007f1a8619bbb0, deadline=yb::CoarseTimePoint @ 0x00007f1a8619ba48) at cdcsdk_producer.cc:1394:5
    frame #15: 0x0000558d1aff2678 yb-tserver`yb::cdc::CDCServiceImpl::GetChanges(this=0x0000558d31d80a20, req=0x0000558e67d4fe20, resp=0x0000558e67d4fe70, context=RpcContext @ 0x00007f1a8619bfe0) at cdc_service.cc:1629:14
    frame #16: 0x0000558d1b092926 yb-tserver`std::__1::__function::__func<yb::cdc::CDCServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_3, std::__1::allocator<yb::cdc::CDCServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_3>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) [inlined] yb::cdc::CDCServiceIf::InitMethods(this=<unavailable>, req=<unavailable>, resp=<unavailable>, rpc_context=RpcContext @ 0x00007f1a8619bfa0)::$_3::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::cdc::GetChangesRequestPB const*, yb::cdc::GetChangesResponsePB*, yb::rpc::RpcContext)::operator()(yb::cdc::GetChangesRequestPB const*, yb::cdc::GetChangesResponsePB*, yb::rpc::RpcContext) const at cdc_service.service.cc:355:9
    frame #17: 0x0000558d1b0928e8 yb-tserver`std::__1::__function::__func<yb::cdc::CDCServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_3, std::__1::allocator<yb::cdc::CDCServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_3>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) at local_call.h:115:7
    frame #18: 0x0000558d1b0925d2 yb-tserver`std::__1::__function::__func<yb::cdc::CDCServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_3, std::__1::allocator<yb::cdc::CDCServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_3>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) [inlined] yb::cdc::CDCServiceIf::InitMethods(this=<unavailable>, call=nullptr)::$_3::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>) const at cdc_service.service.cc:353:7
    frame #19: 0x0000558d1b092555 yb-tserver`std::__1::__function::__func<yb::cdc::CDCServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_3, std::__1::allocator<yb::cdc::CDCServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_3>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) [inlined] decltype(__f=<unavailable>, __args=<unavailable>)::$_3&>()(std::declval<std::__1::shared_ptr<yb::rpc::InboundCall>>())) std::__1::__invoke[abi:v15003]<yb::cdc::CDCServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_3&, std::__1::shared_ptr<yb::rpc::InboundCall>>(yb::cdc::CDCServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_3&, std::__1::shared_ptr<yb::rpc::InboundCall>&&) at invoke.h:394:23
    frame #20: 0x0000558d1b092534 yb-tserver`std::__1::__function::__func<yb::cdc::CDCServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_3, std::__1::allocator<yb::cdc::CDCServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_3>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) [inlined] void std::__1::__invoke_void_return_wrapper<void, true>::__call<yb::cdc::CDCServiceIf::InitMethods(__args=<unavailable>, __args=<unavailable>)::$_3&, std::__1::shared_ptr<yb::rpc::InboundCall>>(yb::cdc::CDCServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_3&, std::__1::shared_ptr<yb::rpc::InboundCall>&&) at invoke.h:479:9
    frame #21: 0x0000558d1b092534 yb-tserver`std::__1::__function::__func<yb::cdc::CDCServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_3, std::__1::allocator<yb::cdc::CDCServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_3>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) [inlined] std::__1::__function::__alloc_func<yb::cdc::CDCServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_3, std::__1::allocator<yb::cdc::CDCServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_3>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator(this=<unavailable>, __arg=<unavailable>)[abi:v15003](std::__1::shared_ptr<yb::rpc::InboundCall>&&) at function.h:185:16
    frame #22: 0x0000558d1b092534 yb-tserver`std::__1::__function::__func<yb::cdc::CDCServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_3, std::__1::allocator<yb::cdc::CDCServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_3>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator(this=<unavailable>, __arg=<unavailable>)(std::__1::shared_ptr<yb::rpc::InboundCall>&&) at function.h:359:12
    frame #23: 0x0000558d1b094d8f yb-tserver`yb::cdc::CDCServiceIf::Handle(std::__1::shared_ptr<yb::rpc::InboundCall>) [inlined] std::__1::__function::__value_func<void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator(this=<unavailable>, __args=nullptr)[abi:v15003](std::__1::shared_ptr<yb::rpc::InboundCall>&&) const at function.h:512:16
    frame #24: 0x0000558d1b094d70 yb-tserver`yb::cdc::CDCServiceIf::Handle(std::__1::shared_ptr<yb::rpc::InboundCall>) [inlined] std::__1::function<void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator(this=<unavailable>, __arg=nullptr)(std::__1::shared_ptr<yb::rpc::InboundCall>) const at function.h:1197:12
    frame #25: 0x0000558d1b094d70 yb-tserver`yb::cdc::CDCServiceIf::Handle(this=<unavailable>, call=<unavailable>) at cdc_service.service.cc:293:3
    frame #26: 0x0000558d1bd9479e yb-tserver`yb::rpc::ServicePoolImpl::Handle(this=0x0000558d31d94000, incoming=nullptr) at service_pool.cc:263:19
    frame #27: 0x0000558d1bcd5b0f yb-tserver`yb::rpc::InboundCall::InboundCallTask::Run(this=<unavailable>) at inbound_call.cc:236:13
    frame #28: 0x0000558d1bda3113 yb-tserver`yb::rpc::(anonymous namespace)::Worker::Execute(this=0x0000558d31d49030) at thread_pool.cc:104:15
    frame #29: 0x0000558d1c4117ff yb-tserver`yb::Thread::SuperviseThread(void*) [inlined] std::__1::__function::__value_func<void ()>::operator(this=0x0000558d3278eb90)[abi:v15003]() const at function.h:512:16
    frame #30: 0x0000558d1c4117ec yb-tserver`yb::Thread::SuperviseThread(void*) [inlined] std::__1::function<void ()>::operator(this=0x0000558d3278eb90)() const at function.h:1197:12
    frame #31: 0x0000558d1c4117ec yb-tserver`yb::Thread::SuperviseThread(arg=0x0000558d3278eb40) at thread.cc:800:3
    frame #32: 0x00007f1a9cbf3694 libpthread.so.0`start_thread(arg=0x00007f1a861a4700) at pthread_create.c:333
    frame #33: 0x00007f1a9d0f541d libc.so.6`__clone at clone.S:109