Type Members

1.  abstract class BaseCachedTable extends AnyRef

   Base class representing a cached table with common properties.

2.  class CachedTable extends BaseCachedTable

   Represents a cached table entry with a single refresher function.

   Represents a cached table entry with a single refresher function. All the queries sent to the same table can share the same refresher to update file URLs.

   Example: If two queries share the same cached table entry, they will both rely on the same refresher function to update the file URLs. When the refresher is invoked, it will generate a new mapping of file IDs to pre-signed URLs, along with a new expiration timestamp and refresh token. This ensures that both queries benefit from the updated URLs without duplicating the refresh logic or state.

3.  class CachedTableManager extends Logging
4.  class PreSignedUrlCacheEndpoint extends RpcEndpoint

   An RpcEndpoint running in Spark driver to allow executors to fetch pre signed urls.

5.  class PreSignedUrlFetcher extends Logging

   A pre signed url fetcher that monitors the pre signed url expiration time and fetches a new one from the driver when it's expiring (under refreshThresholdMs).

6.  class QuerySpecificCachedTable extends BaseCachedTable

   Represents a cached table entry with a unique refresher for each query.

   Represents a cached table entry with a unique refresher for each query. This design ensures that even if two queries are identical in structure, their refreshers can maintain independent states. The refreshFunction is responsible for refreshing the pre-signed URLs for the table, while the RefresherWrapper allows additional customization or state management during the refresh process. This is necessary because the server may require specific states in the refresher function to refresh the pre-signed URLs. The refresh results can be shared across the same query for the same table.

   Example: Query 1: SELECT * FROM table WHERE col1 = 'value1' Query 2: SELECT * FROM table WHERE col1 = 'value1' Both queries can share the same cache entry for the table, but their refreshers maintain independent states to handle server-specific requirements for refreshing pre-signed URLs. When Query 2 ends, and Query 1 keeps running, we need to maintain Query 1's refresher state to ensure it continues to refresh the pre-signed URLs as needed.

7.  case class TableRefreshResult(idToUrl: Map[String, String], expirationTimestamp: Option[Long], refreshToken: Option[String]) extends Product with Serializable