# launch multinode gcp cluster
sky launch -y -d -c test --num-nodes 2 --cloud gcp tests/test_yamls/minimal.yaml
# Try sky status --refresh. This works fine.
sky status test --refresh
# Schedule autostop
sky autostop -y test -i 1
# Wait for autostop
sleep 120
# Try refreshing status - fails
sky status test --refresh
[email protected]:/sky_repo$ sky status test --refresh
Refreshing status for 1 cluster ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:--W 01-26 08:49:45 backend_utils.py:1346] Expected 1 worker IP(s); found 0: []
W 01-26 08:49:45 backend_utils.py:1346] This could happen if there is extra output from `ray get-worker-ips`, which should be inspected below.
W 01-26 08:49:45 backend_utils.py:1346] == Output ==
W 01-26 08:49:45 backend_utils.py:1346]
W 01-26 08:49:45 backend_utils.py:1346]
W 01-26 08:49:45 backend_utils.py:1346] == Output ends ==
Clusters
NAME LAUNCHED RESOURCES STATUS AUTOSTOP COMMAND
test 5 mins ago 2x GCP(n2-standard-8) STOPPED 1m sky launch -y -d -c test ...
1 cluster has auto{stop,down} scheduled. Refresh statuses with: sky status --refresh
sky status test --refresh
afterwards works normally and shows status as stopped.sky stop test
and then refresh status.test_smoke.py::test_autostop
Just to add, in test_autostop
the state is shown to be at INIT
instead of UP
or STOPPED
, which causes the test to fail. Logs here.