Package org.apache.druid.sql.calcite.external

Class SchemaAwareUserDefinedTableMacro

java.lang.Object
org.apache.calcite.sql.SqlOperator
org.apache.calcite.sql.SqlFunction
org.apache.calcite.sql.validate.SqlUserDefinedTableMacro
org.apache.druid.sql.calcite.external.BaseUserDefinedTableMacro
org.apache.druid.sql.calcite.external.SchemaAwareUserDefinedTableMacro
All Implemented Interfaces:
org.apache.calcite.sql.SqlTableFunction, AuthorizableOperator
Direct Known Subclasses:
DruidUserDefinedTableMacro
public abstract class SchemaAwareUserDefinedTableMacro extends BaseUserDefinedTableMacro implements AuthorizableOperator
Table macro designed for use with the Druid EXTEND operator. Example: 
 INSERT INTO dst
 SELECT *
 FROM TABLE(staged(
    source => 'inline',
    format => 'csv',
    data => 'a,b,1
 c,d,2
 '
   ))
   EXTEND (x VARCHAR, y VARCHAR, z BIGINT)
 PARTITIONED BY ALL TIME

Calcite supports the Apache Phoenix EXTEND operator of the form: 

 SELECT ..
 FROM myTable EXTEND (x VARCHAR, ...)

For Druid, we want the above form: extend a table function, not a literal table. Since we can't change the Calcite parser, we instead use tricks within the constraints of the parser.

  • The Calcite parser is revised to add the EXTEND rule for a table function.
  • Calcite expects the EXTEND operator to have two arguments: an identifier and the column list. Since our case has a function call as the first argument, we can't let Calcite see our AST. So, we use a rewrite trick to convert the EXTEND node into the usual TABLE(.) node, and we modify the associated macro to hold onto the schema, which is now out of sight of Calcite, and so will not cause problems with rules that don't understand our usage.
  • Calcite will helpfully rewrite calls, replacing our modified operator with the original. So, we override those to keep our modified operator.
  • When asked to produce a table (apply(.)), we call a Druid-specific version that passes along the schema saved previously.
  • The extended DruidTableMacro uses the schema to define the input source.
  • Care is taken that the same DruidTableMacro can be used without EXTEND. In this case, the schema will be empty and the input source must have a way of providing the schema. The batch ingest feature does not yet support this use case, but it seems a reasonable extension. Example: CSV that has a header row, or a "classic" lookup table that, by definition, has only two columns.

Note that unparsing is a bit of a nuisance. Our trick places the EXTEND list in the wrong place, and we'll unparse SQL as: 

 FROM TABLE(fn(arg1, arg2) EXTEND (x VARCHAR, ...))
Since we seldom use unparse, we can perhaps live with this limitation for now.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static interface 
    SchemaAwareUserDefinedTableMacro.ExtendedTableMacro
     
    protected static class 
    SchemaAwareUserDefinedTableMacro.ShimTableMacro
    Calcite table macro created dynamically to squirrel away the schema provided by the EXTEND clause to allow

  • Field Summary

    Fields inherited from class org.apache.druid.sql.calcite.external.BaseUserDefinedTableMacro

    macro

    Fields inherited from class org.apache.calcite.sql.SqlOperator

    kind, MDX_PRECEDENCE, NL

  • Constructor Summary

    Constructors
    Constructor
    Description
    SchemaAwareUserDefinedTableMacro(org.apache.calcite.sql.SqlIdentifier opName, org.apache.calcite.sql.type.SqlReturnTypeInference returnTypeInference, org.apache.calcite.sql.type.SqlOperandTypeInference operandTypeInference, org.apache.calcite.sql.type.SqlOperandMetadata operandMetadata, SchemaAwareUserDefinedTableMacro.ExtendedTableMacro tableMacro)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.calcite.sql.SqlBasicCall
    rewriteCall(org.apache.calcite.sql.SqlBasicCall oldCall, org.apache.calcite.sql.SqlNodeList schema)
    Rewrite the call to the original table macro function to a new "shim" version that holds both the original one and the schema from EXTEND.

    Methods inherited from class org.apache.druid.sql.calcite.external.BaseUserDefinedTableMacro

    convertArguments, getTable

    Methods inherited from class org.apache.calcite.sql.validate.SqlUserDefinedTableMacro

    getOperandTypeChecker, getParamNames, getRowTypeInference

    Methods inherited from class org.apache.calcite.sql.SqlFunction

    deriveType, getFunctionType, getNameAsId, getParamTypes, getSqlIdentifier, getSyntax, isQuantifierAllowed, unparse, validateCall, validateQuantifier

    Methods inherited from class org.apache.calcite.sql.SqlOperator

    acceptCall, acceptCall, adjustType, allowsFraming, argumentMustBeScalar, checkOperandCount, checkOperandTypes, constructArgNameList, constructArgTypeList, constructOperandList, createCall, createCall, createCall, createCall, createCall, createCall, createCall, deriveOperandType, equals, getAllowedSignatures, getAllowedSignatures, getKind, getLeftPrec, getMonotonicity, getMonotonicity, getName, getOperandCountRange, getOperandTypeInference, getReturnTypeInference, getRightPrec, getSignatureTemplate, getStrongPolicyInference, hashCode, inferReturnType, inferReturnType, isAggregator, isDeterministic, isDynamicFunction, isGroup, isGroupAuxiliary, isName, isSymmetrical, leftPrec, not, preValidateCall, requiresDecimalExpansion, requiresOrder, requiresOver, reverse, rewriteCall, rightPrec, toString, unparseListClause, unparseListClause, validateOperands, validRexOperands

    Methods inherited from class java.lang.Object

    clone, finalize, getClass, notify, notifyAll, wait, wait, wait

    Methods inherited from interface org.apache.druid.sql.calcite.expression.AuthorizableOperator

    computeResources

    Methods inherited from interface org.apache.calcite.sql.SqlTableFunction

    tableCharacteristic

  • Constructor Details

    • SchemaAwareUserDefinedTableMacro

      public SchemaAwareUserDefinedTableMacro(org.apache.calcite.sql.SqlIdentifier opName, org.apache.calcite.sql.type.SqlReturnTypeInference returnTypeInference, org.apache.calcite.sql.type.SqlOperandTypeInference operandTypeInference, org.apache.calcite.sql.type.SqlOperandMetadata operandMetadata, SchemaAwareUserDefinedTableMacro.ExtendedTableMacro tableMacro)

  • Method Details

    • rewriteCall

      public org.apache.calcite.sql.SqlBasicCall rewriteCall(org.apache.calcite.sql.SqlBasicCall oldCall, org.apache.calcite.sql.SqlNodeList schema)
      Rewrite the call to the original table macro function to a new "shim" version that holds both the original one and the schema from EXTEND.