A consistency guard which sleeps for configured period of time only on APPEAR. It is a no-op for DISAPPEAR.
This is specifically for S3A filesystem and here is the rational.
This guard is used when deleting data files corresponding to marker files that needs to be deleted.
There are two tricky cases that needs to be considered. Case 1 : A data file creation is eventually consistent and hence
when issuing deletes, it may not be found. Case 2: a data file was never created in the first place since the process crashed.
In S3A, GET and LIST are eventually consistent, and delete() implementation internally does a LIST/EXISTS.
Prior to this patch, hudi was leveraging
FailSafeConsistencyGuard which was doing the following to delete data files.
Step1: wait for all files to appear with linear backoff.
Step2: issue deletes
Step3: wait for all files to disappear with linear backoff.
Step1 and Step2 is handled by
FailSafeConsistencyGuard.
We are simplifying these steps with
OptimisticConsistencyGuard.
Step1: Check if all files adhere to visibility event. If yes, proceed to Step 3.
Step2: If not, Sleep for a configured threshold and then proceed to next step.
Step3: issue deletes.
With this, if any files that was created, should be available within configured threshold(eventual consistency).
Delete() will return false if FileNotFound. So, both cases are taken care of this
ConsistencyGuard.