To support Meta's high availability, we need to do some refactor in Meta service:

  1. Leader election
  2. Service discovery
  3. State persistence
  • [ ] support leader and standby status in Meta, and do leader election inside etcd
    • when Meta starts, it should participate in leader election, and initialize all managers with memory cache, start related RPC services and sub-tasks only after it get selected.
    • refactor current lease register logic and using officer election APIs in etcd library. The operator in k8s will keep watching the election result and do service binding and DNS refreshing in cloud side.
  • [ ] support refresh meta address in meta_client
    • standby Meta should reject all requests from other components and response with leader address, to avoid some potentially uncertain issues in k8s.
6
database serverless sql rust postgresql cloud-native stream-processing distributed-database

Looking forward to work on this task. However I am still working on a different kernel task and will be at KubeCon next week. I will start with this the earliest at the beginning of November.

0

added a new task "support connecting to multiple meta-node on compute-node"

2

First draft version ready: Failover works, but Hummock crashes: https://github.com/risingwavelabs/risingwave/issues/6534

0

support connecting to multiple meta-node on compute-node

@yezizp2012 I do not understand this part: My understanding is that the meta HA setup is a single-leader-system (see design docs). Therefore we only ever connect to one instance. Is my understanding correct @fuyufjh ?

0

support connecting to multiple meta-node on compute-node

@yezizp2012 I do not understand this part: My understanding is that the meta HA setup is a single-leader-system (see design docs). Therefore we only ever connect to one instance. Is my understanding correct @fuyufjh ?

IIUC, this should refers to the refactoring of the meta client: meta client should be able to connect and retrieve leader information from multi-meta nodes, which requires a get_leader_addr interface in meta.

0

IIUC, this should refers to the refactoring of the meta client: meta client should be able to connect and retrieve leader information from multi-meta nodes, which requires a get_leader_addr interface in meta.

Yes. After that we can modify risedev to provide multiple meta addresses to compute nodes and frontend nodes. Currently it only picks the first one (see risedev warning)

0

Thank you very much for the clarifying. I would suggest that the change in the client and the change in the meta enabling a follower service which supports get_leader_addr will be done in separate PRs.

0

First version of the failover handling is done. Please have a look: https://github.com/risingwavelabs/risingwave/pull/6466

There are a few things in the PR missing:

  • e2e tests. I do not know how best to approach that. Please let me know if you have suggestions
  • Currently we do busy-wait to do elections. ETCD also provides an update API afaik. I guess that this will require quite a reqwrite, which is why I would like to do this as future work.
  • Meta crashes on failover because of https://github.com/risingwavelabs/risingwave/issues/6534. Will be addressed in future PR
0

added a new task "support connecting to multiple meta-node on compute-node"

Is this still up to date? I do not see the subtask @skyzh

0

added a new task "support connecting to multiple meta-node on compute-node"

Is this still up to date? I do not see the subtask @skyzh

That should be a part of #6755 .

1

Tests are still running, but my guess is that https://github.com/risingwavelabs/risingwave/pull/6771 is ready for review. This is a minor tasks overall, but merging it would still simplify further steps. Let me know if you have any objections or suggestions to the PR.

0

I currently have 2 PRs that are ready for review.

  • https://github.com/risingwavelabs/risingwave/pull/6771: Introduces graceful shutdown of services. This is only indirectly related to the HA setup. It will simplify further development.

  • https://github.com/risingwavelabs/risingwave/pull/6792: Introduces the election mechanism, but does not introduce any failover (see features and limitations here). My expectation is that this does not affect our current setup, because we only ever deploy one meta node, which hopefully always becomes leader. PR reduces code coverage, because I delete a unittest which is no longer needed in a HA setup (also see #3398 )

CC @arkbriar

1

https://github.com/risingwavelabs/risingwave/pull/6937 is ready for review. Please view the issue for features and limitations.

1

The fencing PR is ready for review. The CI pipeline is green. Also see https://github.com/risingwavelabs/risingwave/issues/6786.

Maybe we can get this merged before New Years :)

0

Fencing is merged. Thank you very much for your guidance and approval @yezizp2012

0
© 2022 pullanswer.com - All rights reserved.
Privacy policy