Type Members

1.  trait LogKeyShims extends LogKey

   All structured logging keys used in MDC must be extends LogKey

   All structured logging keys used in MDC must be extends LogKey

   LogKeys serve as identifiers for mapped diagnostic contexts (MDC) within logs. Follow these guidelines when adding a new LogKey:

   - Define all structured logging keys in DeltaLogKeys.scala, and sort them alphabetically for ease of search.

   - Use UPPER_SNAKE_CASE for key names.

   - Key names should be both simple and broad, yet include specific identifiers like STAGE_ID, TASK_ID, and JOB_ID when needed for clarity. For instance, use MAX_ATTEMPTS as a general key instead of creating separate keys for each scenario such as EXECUTOR_STATE_SYNC_MAX_ATTEMPTS and MAX_TASK_FAILURES. This balances simplicity with the detail needed for effective logging.

   - Use abbreviations in names if they are widely understood, such as APP_ID for APPLICATION_ID, and K8S for KUBERNETES.

   - For time-related keys, use milliseconds as the unit of time.

2.  trait LoggingShims extends Logging

   Guidelines for the Structured Logging Framework - Scala Logging

   Guidelines for the Structured Logging Framework - Scala Logging

   Use the org.apache.spark.internal.Logging trait for logging in Scala code: Logging Messages with Variables: When logging a message with variables, wrap all the variables with MDCs and they will be automatically added to the Mapped Diagnostic Context (MDC). This allows for structured logging and better log analysis.

   logInfo(log"Trying to recover app: ${MDC(DeltaLogKeys.APP_ID, app.id)}")

   Constant String Messages: If you are logging a constant string message, use the log methods that accept a constant string.

   logInfo("StateStore stopped")

   Exceptions: To ensure logs are compatible with Spark SQL and log analysis tools, avoid Exception.printStackTrace(). Use logError, logWarning, and logInfo methods from the Logging trait to log exceptions, maintaining structured and parsable logs.

   If you want to output logs in scala code through the structured log framework, you can define custom LogKey and use it in scala code as follows:

   // To add a custom LogKey, implement LogKey case object CUSTOM_LOG_KEY extends LogKey import org.apache.spark.internal.MDC; logInfo(log"${MDC(CUSTOM_LOG_KEY, "key")}")