Type Members

1.  class LogEntry extends AnyRef

   Companion class for lazy evaluation of the MessageWithContext instance.

2.  trait LogKey extends AnyRef

   All structured logging keys used in MDC must be extends LogKey

   All structured logging keys used in MDC must be extends LogKey

   LogKeys serve as identifiers for mapped diagnostic contexts (MDC) within logs. Follow these guidelines when adding a new LogKey:

   - Define all structured logging keys in LogKey.scala, and sort them alphabetically for ease of search.

   - Use UPPER_SNAKE_CASE for key names.

   - Key names should be both simple and broad, yet include specific identifiers like STAGE_ID, TASK_ID, and JOB_ID when needed for clarity. For instance, use MAX_ATTEMPTS as a general key instead of creating separate keys for each scenario such as EXECUTOR_STATE_SYNC_MAX_ATTEMPTS and MAX_TASK_FAILURES. This balances simplicity with the detail needed for effective logging.

   - Use abbreviations in names if they are widely understood, such as APP_ID for APPLICATION_ID, and K8S for KUBERNETES.

   - For time-related keys, use milliseconds as the unit of time.

3.  trait Logging extends AnyRef

   Utility trait for classes that want to log data.

   Utility trait for classes that want to log data. Creates a SLF4J logger for the class and allows logging messages at different levels using methods that only evaluate parameters lazily if the log level is enabled.

4.  case class MDC(key: LogKey, value: Any) extends Product with Serializable

   Mapped Diagnostic Context (MDC) that will be used in log messages.

   Mapped Diagnostic Context (MDC) that will be used in log messages. The values of the MDC will be inline in the log message, while the key-value pairs will be part of the ThreadContext.

5.  case class MessageWithContext(message: String, context: Map[String, String]) extends Product with Serializable

   Wrapper class for log messages that include a logging context.

   Wrapper class for log messages that include a logging context. This is used as the return type of the string interpolator LogStringContext.

6.  class SparkLogger extends AnyRef

   Guidelines for the Structured Logging Framework - Java Logging

   Guidelines for the Structured Logging Framework - Java Logging

   Use the org.apache.spark.internal.SparkLoggerFactory to get the logger instance in Java code: Getting Logger Instance: Instead of using org.slf4j.LoggerFactory, use org.apache.spark.internal.SparkLoggerFactory to ensure structured logging.

   import org.apache.spark.internal.SparkLogger; import org.apache.spark.internal.SparkLoggerFactory; private static final SparkLogger logger = SparkLoggerFactory.getLogger(JavaUtils.class);

   Logging Messages with Variables: When logging messages with variables, wrap all the variables with MDCs and they will be automatically added to the Mapped Diagnostic Context (MDC).

   import org.apache.spark.internal.LogKeys; import org.apache.spark.internal.MDC; logger.error("Unable to delete file for partition {}", MDC.of(LogKeys.PARTITION_ID$.MODULE$, i));

   Constant String Messages: For logging constant string messages, use the standard logging methods.

   logger.error("Failed to abort the writer after failing to write map output.", e);

   If you want to output logs in java code through the structured log framework, you can define custom LogKey and use it in java code as follows:

   // To add a custom LogKey, implement LogKey public static class CUSTOM_LOG_KEY implements LogKey { } import org.apache.spark.internal.MDC; logger.error("Unable to delete key {} for cache", MDC.of(CUSTOM_LOG_KEY, "key"));

7.  class SparkLoggerFactory extends AnyRef