Package org.apache.beam.sdk.extensions.ml

Class DLPInspectText

  • All Implemented Interfaces:
    java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData
    public abstract class DLPInspectText
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<java.lang.String,​java.lang.String>>,​org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<java.lang.String,​com.google.privacy.dlp.v2.InspectContentResponse>>>
    A PTransform connecting to Cloud DLP (https://cloud.google.com/dlp/docs/libraries) and inspecting text for identifying data according to provided settings. The transform supports both delimited columnar input data (eg. CSV) and unstructured input.

    If the headerColumns property is set and a sideinput with table headers is added to the PTransform, delimiter also should be set, else the results will be incorrect. If headerColumns is neither set nor passed as sideinput, input is assumed to be unstructured.

    Batch size defines how big are batches sent to DLP at once in bytes.

    The transform consumes KV of Strings (assumed to be filename as key and contents as value) and outputs KV of String (eg. filename) and InspectContentResponse, which will contain a list of InspectResult for the user to consume.

    Either inspectTemplateName (String) or inspectConfig InspectConfig need to be set.

    Batch size defines how big are batches sent to DLP at once in bytes.

    See Also:
    Serialized Form

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      static class  DLPInspectText.Builder  

    • Field Summary

      Fields 
      Modifier and Type Field Description
      static java.lang.Integer DLP_PAYLOAD_LIMIT_BYTES  

      • Fields inherited from class org.apache.beam.sdk.transforms.PTransform

        annotations, displayData, name, resourceHints

    • Constructor Summary

      Constructors 
      Constructor Description
      DLPInspectText()  

    • Method Summary

      All Methods Static Methods Instance Methods Abstract Methods Concrete Methods 
      Modifier and Type Method Description
      org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<java.lang.String,​com.google.privacy.dlp.v2.InspectContentResponse>> expand​(org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<java.lang.String,​java.lang.String>> input)
      The transform converts the contents of input PCollection into Table.Rows and then calls Cloud DLP service to perform the data inspection according to provided settings.
      abstract int getBatchSizeBytes()
      Returns size of input elements batch to be sent to Cloud DLP service in one request.
      abstract @Nullable java.lang.String getColumnDelimiter()
      Returns delimiter to be used when splitting values from input strings into columns.
      abstract @Nullable org.apache.beam.sdk.values.PCollectionView<java.util.List<java.lang.String>> getHeaderColumns()
      Returns list of column names if the input KV value is a delimited row.
      abstract @Nullable com.google.privacy.dlp.v2.InspectConfig getInspectConfig()
      Returns configuration object for data inspection.
      abstract @Nullable java.lang.String getInspectTemplateName()
      Returns template name for data inspection.
      abstract java.lang.String getProjectId()
      Returns ID of Google Cloud project to be used when deidentifying data.
      static DLPInspectText.Builder newBuilder()  

      • Methods inherited from class org.apache.beam.sdk.transforms.PTransform

        addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

    • Field Detail

      • DLP_PAYLOAD_LIMIT_BYTES

        public static final java.lang.Integer DLP_PAYLOAD_LIMIT_BYTES

    • Constructor Detail

      • DLPInspectText

        public DLPInspectText()

    • Method Detail

      • getInspectTemplateName

        public abstract @Nullable java.lang.String getInspectTemplateName()
        Returns template name for data inspection.

      • getInspectConfig

        public abstract @Nullable com.google.privacy.dlp.v2.InspectConfig getInspectConfig()
        Returns configuration object for data inspection. If present, supersedes the template settings.

      • getBatchSizeBytes

        public abstract int getBatchSizeBytes()
        Returns size of input elements batch to be sent to Cloud DLP service in one request.

      • getProjectId

        public abstract java.lang.String getProjectId()
        Returns ID of Google Cloud project to be used when deidentifying data.

      • getColumnDelimiter

        public abstract @Nullable java.lang.String getColumnDelimiter()
        Returns delimiter to be used when splitting values from input strings into columns.

      • getHeaderColumns

        public abstract @Nullable org.apache.beam.sdk.values.PCollectionView<java.util.List<java.lang.String>> getHeaderColumns()
        Returns list of column names if the input KV value is a delimited row.

      • expand

        public org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<java.lang.String,​com.google.privacy.dlp.v2.InspectContentResponse>> expand​(org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<java.lang.String,​java.lang.String>> input)
        The transform converts the contents of input PCollection into Table.Rows and then calls Cloud DLP service to perform the data inspection according to provided settings.
        Specified by:
        expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<java.lang.String,​java.lang.String>>,​org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<java.lang.String,​com.google.privacy.dlp.v2.InspectContentResponse>>>
        Parameters:
        input - input PCollection
        Returns:
        PCollection after transformations