Jira Link: DB-5141
Logs: logs.zip
testpitrycqlconsistentrestoredml-k8s-rf3 is failing for 2.14. It started failing from 2.14.7.0-b11(it did pass in 2.14.7.0-b7 thus is a regression) and has been failing in every re-run. Note that AWS counterpart is working as expected.
Test description:
1. Create a Universe with consistent restore enabled
2. Create a keyspace
3. Create a snapshot schedule
4. Note down absolute time
5. Create a table, note down absolute time
6. Perform the following DML operations repeatedly:
i. Insert a new row
ii. Update an existing row
iii.Delete an existing row
To perform this step we create a thread dml_operation_thread.
7. Start dml_operation_thread
8. Do following operations for '10' times:
i. Wait for 10 seconds and collect intermediate absolute time.
ii. Wait for 5 seconds and restore to collected absolute time in #8
iii. Validate successful restoration
iv. Validate that expected number of inconsistencies in data is <=1.
9. Restore to empty table state using collected absolute time in #5
10. Validate successful restoration
11. Sleep for a minute.
12. Validate that expected number of inconsistencies in data is 0.
13. Restore to no table state using collected absolute time in #4
14. Wait for restore to be completed & validated
15. Wait for dml_operation_thread to complete.
16. Validate that the table doesn't exist.
17. Destroy Universe
Test is failing at 3rd iteration of step 8.iv as there are >3 jumps found in data after 3 PITRs
