This bug touches multiple components: Please also see https://github.com/risingwavelabs/risingwave-operator/issues/257
If you misconfigure the bucket your bucket endpoint in your risingwave instance, you will crash the compactors and compute. IMHO we should try to be graceful about this and give the user a meaningful error
Reproduce
apiVersion: v1
kind: Secret
metadata:
name: cos-credentials
stringData:
AccessKeyID: SOME-VALUE-HERE
SecretAccessKey: SOME-OTHER-VALUE-HERE
---
apiVersion: risingwave.risingwavelabs.com/v1alpha1
kind: RisingWave
metadata:
name: risingwave-s3-compatible
spec:
storages:
meta:
memory: true
object:
s3:
secret: cos-credentials
region: eu-central-1
bucket: somebucket
endpoint: somebucket.${REGION}.amazonaws.com # Incorrect endpoint
virtualHostedStyle: true
global:
image: ghcr.io/risingwavelabs/risingwave:nightly-20221107
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 100m
memory: 100Mi
replicas:
meta: 1
frontend: 1
compute: 1
compactor: 1
Output
risingwave-s3-compatible-compactor-6fb6c6db69-vztdd 0/1 CrashLoopBackOff 7 (29s ago) 8m4s
risingwave-s3-compatible-compute-0 0/1 CrashLoopBackOff 6 (2m44s ago) 8m4s
risingwave-s3-compatible-frontend-8558f5c54f-2vp5m 1/1 Running 0 8m4s
risingwave-s3-compatible-meta-55d979986d-dch5r 1/1 Running 0 8m4s
logs from risingwave-s3-compatible-compute-0
k logs risingwave-s3-compatible-compute-0
launching compute node
2022-11-08T14:19:12.162902Z INFO risingwave_rt: setting tokio worker threads to 1
2022-11-08T14:19:12.163039Z INFO risingwave_compute: Compute node options: ComputeNodeOpts { host: "10.244.0.63:5688", client_address: Some("risingwave-s3-compatible-compute-0.risingwave-s3-compatible-compute:5688"), state_store: "hummock+s3-compatible://hummock001test", prometheus_listener_addr: "0.0.0.0:1222", metrics_level: 1, meta_address: "http://risingwave-s3-compatible-meta:5690", config_path: "/risingwave/config/risingwave.toml", enable_jaeger_tracing: false, async_stack_trace: Off, file_cache_dir: "", enable_managed_cache: false }
2022-11-08T14:19:12.163084Z INFO risingwave_compute: Server Listening at 10.244.0.63:5688
2022-11-08T14:19:12.163150Z INFO risingwave_compute: Client address is risingwave-s3-compatible-compute-0.risingwave-s3-compatible-compute:5688
2022-11-08T14:19:12.163390Z INFO risingwave_compute::server: Starting compute node with config ComputeNodeConfig { server: ServerConfig { heartbeat_interval_ms: 1000, connection_pool_size: 16 }, batch: BatchConfig { worker_threads_num: None, developer: DeveloperConfig { batch_output_channel_size: 64, batch_chunk_size: 1024, stream_enable_executor_row_count: false, stream_connector_message_buffer_size: 16, unsafe_stream_hash_agg_cache_size: 65536, unsafe_stream_join_cache_size: 65536, unsafe_stream_extreme_cache_size: 1024, stream_chunk_size: 1024 } }, streaming: StreamingConfig { barrier_interval_ms: 250, in_flight_barrier_nums: 40, checkpoint_frequency: 10, minimal_scheduling: false, worker_node_parallelism: 1, actor_runtime_worker_threads_num: None, total_memory_available_bytes: 8232747008, developer: DeveloperConfig { batch_output_channel_size: 64, batch_chunk_size: 1024, stream_enable_executor_row_count: false, stream_connector_message_buffer_size: 16, unsafe_stream_hash_agg_cache_size: 65536, unsafe_stream_join_cache_size: 65536, unsafe_stream_extreme_cache_size: 1024, stream_chunk_size: 1024 } }, storage: StorageConfig { sstable_size_mb: 256, block_size_kb: 1024, bloom_false_positive: 0.01, share_buffers_sync_parallelism: 1, share_buffer_compaction_worker_threads_number: 4, shared_buffer_capacity_mb: 1024, data_directory: "hummock_001", write_conflict_detection_enabled: false, block_cache_capacity_mb: 256, meta_cache_capacity_mb: 64, disable_remote_compactor: false, enable_local_spill: true, local_object_store: "tempdisk", share_buffer_upload_concurrency: 8, compactor_memory_limit_mb: 512, sstable_id_remote_fetch_number: 10, file_cache: FileCacheConfig { capacity_mb: 1024, total_buffer_capacity_mb: 128, cache_file_fallocate_unit_mb: 512, cache_meta_fallocate_unit_mb: 16, cache_file_max_write_size_mb: 4 }, min_sst_size_for_streaming_upload: 33554432, max_sub_compaction: 4, object_store_use_batch_delete: true, enable_state_store_v1: false } } with debug assertions off
2022-11-08T14:19:12.166027Z INFO risingwave_compute::server: Assigned worker node id 3
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: io error: error trying to connect: dns error: failed to lookup address information: Name or service not known
backtrace of `ObjectError`:
0: std::backtrace_rs::backtrace::libunwind::trace
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
1: std::backtrace_rs::backtrace::trace_unsynchronized
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: std::backtrace::Backtrace::create
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/backtrace.rs:333:13
3: <risingwave_object_store::object::error::ObjectError as core::convert::From<risingwave_object_store::object::error::ObjectErrorInner>>::from
at ./risingwave/src/object_store/src/object/error.rs:38:10
4: <T as core::convert::Into<U>>::into
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/convert/mod.rs:726:9
5: <risingwave_object_store::object::error::ObjectError as core::convert::From<aws_smithy_http::result::SdkError<E>>>::from
at ./risingwave/src/object_store/src/object/error.rs:81:9
6: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/result.rs:2108:27
7: risingwave_object_store::object::s3::S3ObjectStore::configure_bucket_lifecycle::{{closure}}
at ./risingwave/src/object_store/src/object/s3.rs:701:13
8: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/future/mod.rs:91:19
9: risingwave_object_store::object::s3::S3ObjectStore::new_s3_compatible::{{closure}}
at ./risingwave/src/object_store/src/object/s3.rs:572:13
10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/future/mod.rs:91:19
11: risingwave_object_store::object::parse_remote_object_store::{{closure}}
at ./risingwave/src/object_store/src/object/mod.rs:815:17
12: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/future/mod.rs:91:19
13: risingwave_storage::store_impl::StateStoreImpl::new::{{closure}}
at ./risingwave/src/storage/src/store_impl.rs:152:17
14: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/future/mod.rs:91:19
15: risingwave_compute::server::compute_node_serve::{{closure}}
at ./risingwave/src/compute/src/server.rs:118:5
16: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/future/mod.rs:91:19
17: risingwave_compute::start::{{closure}}
at ./risingwave/src/compute/src/lib.rs:116:69
18: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/future/mod.rs:91:19
19: <core::pin::Pin<P> as core::future::future::Future>::poll
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/future/future.rs:124:9
20: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tracing-0.1.36/src/instrument.rs:272:9
21: tokio::park::thread::CachedParkThread::block_on::{{closure}}
at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/park/thread.rs:267:54
22: tokio::coop::with_budget::{{closure}}
at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/coop.rs:102:9
23: std::thread::local::LocalKey<T>::try_with
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/thread/local.rs:446:16
24: std::thread::local::LocalKey<T>::with
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/thread/local.rs:422:9
25: tokio::coop::with_budget
at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/park/thread.rs:267:31
26: tokio::coop::budget
at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/coop.rs:72:5
27: tokio::park::thread::CachedParkThread::block_on
at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/park/thread.rs:267:31
28: tokio::runtime::enter::Enter::block_on
at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/runtime/enter.rs:152:13
29: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/runtime/scheduler/multi_thread/mod.rs:79:9
30: tokio::runtime::Runtime::block_on
at ./root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/runtime/mod.rs:492:44
31: risingwave_rt::main_okk
at ./risingwave/src/utils/runtime/src/lib.rs:264:5
32: risingwave::main::{{closure}}
at ./risingwave/src/cmd_all/src/bin/risingwave.rs:50:17
33: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/alloc/src/boxed.rs:1952:9
34: risingwave::main
at ./risingwave/src/cmd_all/src/bin/risingwave.rs:163:13
35: core::ops::function::FnOnce::call_once
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/ops/function.rs:251:5
36: std::sys_common::backtrace::__rust_begin_short_backtrace
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/sys_common/backtrace.rs:122:18
37: std::rt::lang_start::{{closure}}
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/rt.rs:166:18
38: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/ops/function.rs:286:13
39: std::panicking::try::do_call
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/panicking.rs:483:40
40: std::panicking::try
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/panicking.rs:447:19
41: std::panic::catch_unwind
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/panic.rs:137:14
42: std::rt::lang_start_internal::{{closure}}
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/rt.rs:148:48
43: std::panicking::try::do_call
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/panicking.rs:483:40
44: std::panicking::try
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/panicking.rs:447:19
45: std::panic::catch_unwind
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/panic.rs:137:14
46: std::rt::lang_start_internal
at ./rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/rt.rs:148:20
47: main
48: <unknown>
49: __libc_start_main
50: _start
', /risingwave/src/object_store/src/object/s3.rs:573:14
stack backtrace:
0: 0xaaaae9443de0 - std::backtrace_rs::backtrace::libunwind::trace::h04a18e7ccbcb06b1
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
1: 0xaaaae9443de0 - std::backtrace_rs::backtrace::trace_unsynchronized::he3023a106d1e983b
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0xaaaae9443de0 - std::sys_common::backtrace::_print_fmt::heef32bbe596930aa
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/sys_common/backtrace.rs:66:5
3: 0xaaaae9443de0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h188116f30beafc25
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/sys_common/backtrace.rs:45:22
4: 0xaaaae946d77c - core::fmt::write::h5882608e3fe006e9
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/fmt/mod.rs:1209:17
5: 0xaaaae943c6d8 - std::io::Write::write_fmt::hf5f129ed0af94b2d
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/io/mod.rs:1682:15
6: 0xaaaae94453f0 - std::sys_common::backtrace::_print::h5b7f4c819c680459
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/sys_common/backtrace.rs:48:5
7: 0xaaaae94453f0 - std::sys_common::backtrace::print::h52ba7378eac337d4
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/sys_common/backtrace.rs:35:9
8: 0xaaaae94453f0 - std::panicking::default_hook::{{closure}}::h6e4f5e51d086175e
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/panicking.rs:267:22
9: 0xaaaae94450e8 - std::panicking::default_hook::h921f654ba63386e3
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/panicking.rs:286:9
10: 0xaaaae5dd8910 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h3244ff998172224a
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/alloc/src/boxed.rs:1952:9
11: 0xaaaae5dd8910 - risingwave_rt::set_panic_hook::{{closure}}::h5440d81392049248
at /risingwave/src/utils/runtime/src/lib.rs:81:9
12: 0xaaaae5dd8910 - std::panicking::update_hook::{{closure}}::h4122e5117ab475a5
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/panicking.rs:232:47
13: 0xaaaae9445c20 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h931855a755dd854d
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/alloc/src/boxed.rs:1952:9
14: 0xaaaae9445c20 - std::panicking::rust_panic_with_hook::ha40c5feea6f8040a
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/panicking.rs:692:13
15: 0xaaaae94459dc - std::panicking::begin_panic_handler::{{closure}}::hc4856f3b1de015c8
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/panicking.rs:579:13
16: 0xaaaae9444270 - std::sys_common::backtrace::__rust_end_short_backtrace::hceabb7b4314cd733
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/sys_common/backtrace.rs:138:18
17: 0xaaaae9445720 - rust_begin_unwind
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/panicking.rs:575:5
18: 0xaaaae946af30 - core::panicking::panic_fmt::hab281a5433b6cc39
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/panicking.rs:65:14
19: 0xaaaae946b2bc - core::result::unwrap_failed::h8fb7a5daf92b5ab3
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/result.rs:1791:5
20: 0xaaaae6856cd4 - core::result::Result<T,E>::unwrap::h3ef19e5cfac6c233
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/result.rs:1113:23
21: 0xaaaae6856cd4 - risingwave_object_store::object::s3::S3ObjectStore::new_s3_compatible::{{closure}}::h259176a6cf9b2dc1
at /risingwave/src/object_store/src/object/s3.rs:571:9
22: 0xaaaae6856cd4 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h7630ca0368cdc33d
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/future/mod.rs:91:19
23: 0xaaaae6856cd4 - risingwave_object_store::object::parse_remote_object_store::{{closure}}::h9f51c38f3c13c856
at /risingwave/src/object_store/src/object/mod.rs:815:17
24: 0xaaaae6856cd4 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h9e57b7bbae11bc2e
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/future/mod.rs:91:19
25: 0xaaaae6856cd4 - risingwave_storage::store_impl::StateStoreImpl::new::{{closure}}::ha5d2cde567dff398
at /risingwave/src/storage/src/store_impl.rs:152:17
26: 0xaaaae68de9d8 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h6826ba93c6581cc9
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/future/mod.rs:91:19
27: 0xaaaae68de9d8 - risingwave_compute::server::compute_node_serve::{{closure}}::h83e04e0d4181aecb
at /risingwave/src/compute/src/server.rs:118:5
28: 0xaaaae68ac3cc - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h473ff048fd219e3c
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/future/mod.rs:91:19
29: 0xaaaae68ac3cc - risingwave_compute::start::{{closure}}::h3f4f28b1acd7789c
at /risingwave/src/compute/src/lib.rs:116:69
30: 0xaaaae68ac3cc - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h2654e9b5d3c79f44
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/future/mod.rs:91:19
31: 0xaaaae5b036d0 - <core::pin::Pin<P> as core::future::future::Future>::poll::h38ffa97c1ed3d0a3
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/future/future.rs:124:9
32: 0xaaaae5b036d0 - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h9d5ccbbf351b1d46
at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tracing-0.1.36/src/instrument.rs:272:9
33: 0xaaaae5b036d0 - tokio::park::thread::CachedParkThread::block_on::{{closure}}::hbc54adc2e03e3e0a
at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/park/thread.rs:267:54
34: 0xaaaae5b036d0 - tokio::coop::with_budget::{{closure}}::h329aa459a32fb5cf
at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/coop.rs:102:9
35: 0xaaaae5b036d0 - std::thread::local::LocalKey<T>::try_with::hfe5e8a113fce0ce7
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/thread/local.rs:446:16
36: 0xaaaae5b036d0 - std::thread::local::LocalKey<T>::with::h581feed27a449fbd
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/thread/local.rs:422:9
37: 0xaaaae5abe664 - tokio::coop::with_budget::hca7d6d16d406e847
at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/park/thread.rs:267:31
38: 0xaaaae5abe664 - tokio::coop::budget::hbcffd4cf2b1f04a8
at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/coop.rs:72:5
39: 0xaaaae5abe664 - tokio::park::thread::CachedParkThread::block_on::h9e678d4a194eb4b1
at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/park/thread.rs:267:31
40: 0xaaaae5cc1a28 - tokio::runtime::enter::Enter::block_on::h3c9fbf9e60c4dad0
at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/runtime/enter.rs:152:13
41: 0xaaaae5cc1a28 - tokio::runtime::scheduler::multi_thread::MultiThread::block_on::hc840d80997296e77
at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/runtime/scheduler/multi_thread/mod.rs:79:9
42: 0xaaaae5c03ff4 - tokio::runtime::Runtime::block_on::h2f6230e65bedabf3
at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/runtime/mod.rs:492:44
43: 0xaaaae5c60420 - risingwave_rt::main_okk::h38d642d91f68cb7f
at /risingwave/src/utils/runtime/src/lib.rs:264:5
44: 0xaaaae5d29cb0 - risingwave::main::{{closure}}::h3af9da7f6a6ce4be
at /risingwave/src/cmd_all/src/bin/risingwave.rs:50:17
45: 0xaaaae5d298c0 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hb5de24c8dc08bc80
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/alloc/src/boxed.rs:1952:9
46: 0xaaaae5d298c0 - risingwave::main::h5e9a1cd14354fb12
at /risingwave/src/cmd_all/src/bin/risingwave.rs:163:13
47: 0xaaaae5a9759c - core::ops::function::FnOnce::call_once::h221cc6e932ed01d4
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/ops/function.rs:251:5
48: 0xaaaae5a9759c - std::sys_common::backtrace::__rust_begin_short_backtrace::hf9b2cb7cc0a6c243
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/sys_common/backtrace.rs:122:18
49: 0xaaaae5b6fae8 - std::rt::lang_start::{{closure}}::h8f8c9fcd958b2d02
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/rt.rs:166:18
50: 0xaaaae94347bc - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h2595f7476fadd250
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/core/src/ops/function.rs:286:13
51: 0xaaaae94347bc - std::panicking::try::do_call::ha6ea34184de7bcc3
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/panicking.rs:483:40
52: 0xaaaae94347bc - std::panicking::try::h678699e0d7da6047
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/panicking.rs:447:19
53: 0xaaaae94347bc - std::panic::catch_unwind::h2c028ce780041345
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/panic.rs:137:14
54: 0xaaaae94347bc - std::rt::lang_start_internal::{{closure}}::h1b79246b935419bb
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/rt.rs:148:48
55: 0xaaaae94347bc - std::panicking::try::do_call::hafc072d0fb06af36
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/panicking.rs:483:40
56: 0xaaaae94347bc - std::panicking::try::hba7529a38a9b722a
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/panicking.rs:447:19
57: 0xaaaae94347bc - std::panic::catch_unwind::h829b46919a3bf31e
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/panic.rs:137:14
58: 0xaaaae94347bc - std::rt::lang_start_internal::hf57a3cc7ad699fc4
at /rustc/b8c35ca26b191bb9a9ac669a4b3f4d3d52d97fb1/library/std/src/rt.rs:148:20
59: 0xaaaae5d2a264 - main
60: 0xffffb15d73fc - <unknown>
61: 0xffffb15d74cc - __libc_start_main
62: 0xaaaae5a69030 - _start
63: 0x0 - <unknown>
It is generally hard for kernel to understand what causes remote cloud storage connection error. I think it makes sense to panic and return whatever error message the remote cloud storage gives back to us. In other words, if the remote cloud storage doesn't give back sufficient error message to us, we cannot know whether the problem is due to invalid endpoint, bucket not exist, or other errors.
bucket is misconfigured, you will receive an error message NoSuchBucket: Error { code: "NoSuchBucket", message: "The specified bucket does not exist.", request_id: "NjM2Y2I4ZWVfOGEzNTY4MDlfOTljZV8yZTc0ZTU0" }
ak/sk is misconfigured, you will receive an error message SignatureDoesNotMatch:code: "SignatureDoesNotMatch", message: "The Signature you specified is invalid.", request_id: "NjM2Y2JhMWFfNzI0NTIyMDlfYjY4MF8xZmU1OWMw
region is misconfigured, it's hard for kernel to get a specific error message.