org.bytedeco.tensorrt.nvinfer

Class IPluginV2DynamicExt

java.lang.Object

org.bytedeco.javacpp.Pointer

org.bytedeco.tensorrt.nvinfer.IPluginV2

org.bytedeco.tensorrt.nvinfer.IPluginV2Ext

org.bytedeco.tensorrt.nvinfer.IPluginV2DynamicExt

All Implemented Interfaces:

AutoCloseable

@Namespace(value="nvinfer1")
 @NoOffset
 @Properties(inherit=nvinfer.class)
public class IPluginV2DynamicExt
extends IPluginV2Ext

\class IPluginV2DynamicExt Similar to IPluginV2Ext, but with support for dynamic shapes. Clients should override the public methods, including the following inherited methods: virtual int32_t getNbOutputs() const noexcept = 0; virtual nvinfer1::DataType getOutputDataType(int32_t index, nvinfer1::DataType const* inputTypes, int32_t nbInputs) const noexcept = 0; virtual size_t getSerializationSize() const noexcept = 0; virtual void serialize(void* buffer) const noexcept = 0; virtual void destroy() noexcept = 0; virtual void setPluginNamespace(char const* pluginNamespace) noexcept = 0; virtual char const* getPluginNamespace() const noexcept = 0; For getOutputDataType, the inputTypes will always be DataType::kFLOAT or DataType::kINT32, and the returned type is canonicalized to DataType::kFLOAT if it is DataType::kHALF or DataType:kINT8. Details about the floating-point precision are elicited later by method supportsFormatCombination.

Nested Class Summary

Nested classes/interfaces inherited from class org.bytedeco.javacpp.Pointer

Pointer.CustomDeallocator, Pointer.Deallocator, Pointer.NativeDeallocator, Pointer.ReferenceCounter

Field Summary

Fields

Modifier and Type	Field and Description
static int	kFORMAT_COMBINATION_LIMIT

Fields inherited from class org.bytedeco.javacpp.Pointer

address, capacity, limit, position

Constructor Summary

Constructors

Constructor and Description
IPluginV2DynamicExt(Pointer p) Pointer cast constructor.

Method Summary

Modifier and Type	Method and Description
IPluginV2DynamicExt	clone() \brief Clone the plugin object.
void	configurePlugin(DynamicPluginTensorDesc in, int nbInputs, DynamicPluginTensorDesc out, int nbOutputs) \brief Configure the plugin.
int	enqueue(PluginTensorDesc inputDesc, PluginTensorDesc outputDesc, Pointer inputs, Pointer outputs, Pointer workspace, org.bytedeco.cuda.cudart.CUstream_st stream)
int	enqueue(PluginTensorDesc inputDesc, PluginTensorDesc outputDesc, PointerPointer inputs, PointerPointer outputs, Pointer workspace, org.bytedeco.cuda.cudart.CUstream_st stream) \brief Execute the layer.
DimsExprs	getOutputDimensions(int outputIndex, DimsExprs inputs, int nbInputs, IExprBuilder exprBuilder) \brief Get expressions for computing dimensions of an output tensor from dimensions of the input tensors.
long	getWorkspaceSize(PluginTensorDesc inputs, int nbInputs, PluginTensorDesc outputs, int nbOutputs) \brief Find the workspace size required by the layer.
static int	kFORMAT_COMBINATION_LIMIT() Limit on number of format combinations accepted.
boolean	supportsFormatCombination(int pos, PluginTensorDesc inOut, int nbInputs, int nbOutputs) \brief Return true if plugin supports the format and datatype for the input/output indexed by pos.

Methods inherited from class org.bytedeco.tensorrt.nvinfer.IPluginV2Ext

attachToContext, canBroadcastInputAcrossBatch, configurePlugin, configurePlugin, configurePlugin, configurePlugin, configurePlugin, configurePlugin, detachFromContext, getOutputDataType, getOutputDataType, getOutputDataType, isOutputBroadcastAcrossBatch, isOutputBroadcastAcrossBatch

Methods inherited from class org.bytedeco.tensorrt.nvinfer.IPluginV2

configureWithFormat, configureWithFormat, destroy, enqueue, enqueue, getNbOutputs, getOutputDimensions, getPluginNamespace, getPluginType, getPluginVersion, getSerializationSize, getTensorRTVersion, getWorkspaceSize, initialize, serialize, setPluginNamespace, setPluginNamespace, supportsFormat, supportsFormat, terminate

Methods inherited from class org.bytedeco.javacpp.Pointer

address, asBuffer, asByteBuffer, availablePhysicalBytes, calloc, capacity, capacity, close, deallocate, deallocate, deallocateReferences, deallocator, deallocator, equals, fill, formatBytes, free, getDirectBufferAddress, getPointer, getPointer, getPointer, getPointer, hashCode, interruptDeallocatorThread, isNull, isNull, limit, limit, malloc, maxBytes, maxPhysicalBytes, memchr, memcmp, memcpy, memmove, memset, offsetAddress, offsetof, offsetof, parseBytes, physicalBytes, physicalBytesInaccurate, position, position, put, realloc, referenceCount, releaseReference, retainReference, setNull, sizeof, sizeof, toString, totalBytes, totalCount, totalPhysicalBytes, withDeallocator, zero

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

kFORMAT_COMBINATION_LIMIT

public static final int kFORMAT_COMBINATION_LIMIT

Constructor Detail

IPluginV2DynamicExt

public IPluginV2DynamicExt(Pointer p)

Pointer cast constructor. Invokes Pointer(Pointer).

Method Detail

clone

@NoException(value=true)
public IPluginV2DynamicExt clone()

Description copied from class: IPluginV2Ext

\brief Clone the plugin object. This copies over internal plugin parameters as well and returns a new plugin object with these parameters. If the source plugin is pre-configured with configurePlugin(), the returned object should also be pre-configured. The returned object should allow attachToContext() with a new execution context. Cloned plugin objects can share the same per-engine immutable resource (e.g. weights) with the source object (e.g. via ref-counting) to avoid duplication. \usage - Allowed context for the API call - Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads when building networks on multiple devices sharing the same plugin.

Overrides:

clone in class IPluginV2Ext

getOutputDimensions

@ByVal
 @NoException(value=true)
public DimsExprs getOutputDimensions(int outputIndex,
                                                                      @Const
                                                                      DimsExprs inputs,
                                                                      int nbInputs,
                                                                      @ByRef
                                                                      IExprBuilder exprBuilder)

\brief Get expressions for computing dimensions of an output tensor from dimensions of the input tensors.

Parameters:

outputIndex - The index of the output tensor

inputs - Expressions for dimensions of the input tensors

nbInputs - The number of input tensors

exprBuilder - Object for generating new expressions This function is called by the implementations of IBuilder during analysis of the network. Example #1: A plugin has a single output that transposes the last two dimensions of the plugin's single input. The body of the override of getOutputDimensions can be: DimsExprs output(inputs[0]); std::swap(output.d[output.nbDims-1], output.d[output.nbDims-2]); return output; Example #2: A plugin concatenates its two inputs along the first dimension. The body of the override of getOutputDimensions can be: DimsExprs output(inputs[0]); output.d[0] = exprBuilder.operation(DimensionOperation::kSUM, *inputs[0].d[0], *inputs[1].d[0]); return output;

kFORMAT_COMBINATION_LIMIT

@MemberGetter
public static int kFORMAT_COMBINATION_LIMIT()

Limit on number of format combinations accepted.

supportsFormatCombination

@Cast(value="bool")
 @NoException(value=true)
public boolean supportsFormatCombination(int pos,
                                                                                       @Const
                                                                                       PluginTensorDesc inOut,
                                                                                       int nbInputs,
                                                                                       int nbOutputs)

\brief Return true if plugin supports the format and datatype for the input/output indexed by pos. For this method inputs are numbered 0..(nbInputs-1) and outputs are numbered nbInputs..(nbInputs+nbOutputs-1). Using this numbering, pos is an index into InOut, where 0 <= pos < nbInputs+nbOutputs. TensorRT invokes this method to ask if the input/output indexed by pos supports the format/datatype specified by inOut[pos].format and inOut[pos].type. The override should return true if that format/datatype at inOut[pos] are supported by the plugin. If support is conditional on other input/output formats/datatypes, the plugin can make its result conditional on the formats/datatypes in inOut[0..pos-1], which will be set to values that the plugin supports. The override should not inspect inOut[pos+1..nbInputs+nbOutputs-1], which will have invalid values. In other words, the decision for pos must be based on inOut[0..pos] only. Some examples: * A definition for a plugin that supports only FP16 NCHW: return inOut.format[pos] == TensorFormat::kLINEAR && inOut.type[pos] == DataType::kHALF; * A definition for a plugin that supports only FP16 NCHW for its two inputs, and FP32 NCHW for its single output: return inOut.format[pos] == TensorFormat::kLINEAR && (inOut.type[pos] == (pos < 2 ? DataType::kHALF : DataType::kFLOAT)); * A definition for a "polymorphic" plugin with two inputs and one output that supports any format or type, but the inputs and output must have the same format and type: return pos == 0 || (inOut.format[pos] == inOut.format[0] && inOut.type[pos] == inOut.type[0]); Warning: TensorRT will stop asking for formats once it finds kFORMAT_COMBINATION_LIMIT on combinations.

configurePlugin

@NoException(value=true)
public void configurePlugin(@Const
                                                     DynamicPluginTensorDesc in,
                                                     int nbInputs,
                                                     @Const
                                                     DynamicPluginTensorDesc out,
                                                     int nbOutputs)

\brief Configure the plugin. configurePlugin() can be called multiple times in both the build and execution phases. The build phase happens before initialize() is called and only occurs during creation of an engine by IBuilder. The execution phase happens after initialize() is called and occurs during both creation of an engine by IBuilder and execution of an engine by IExecutionContext. Build phase: IPluginV2DynamicExt->configurePlugin is called when a plugin is being prepared for profiling but not for any specific input size. This provides an opportunity for the plugin to make algorithmic choices on the basis of input and output formats, along with the bound of possible dimensions. The min and max value of the DynamicPluginTensorDesc correspond to the kMIN and kMAX value of the current profile that the plugin is being profiled for, with the desc.dims field corresponding to the dimensions of plugin specified at network creation. Wildcard dimensions will exist during this phase in the desc.dims field. Execution phase: IPluginV2DynamicExt->configurePlugin is called when a plugin is being prepared for executing the plugin for a specific dimensions. This provides an opportunity for the plugin to change algorithmic choices based on the explicit input dimensions stored in desc.dims field. * IBuilder will call this function once per profile, with desc.dims resolved to the values specified by the kOPT field of the current profile. Wildcard dimensions will not exist during this phase. * IExecutionContext will call this during the next subsequent instance enqueue[V2]() or execute[V2]() if: - The batch size is changed from previous call of execute()/enqueue() if hasImplicitBatchDimension() returns true. - The optimization profile is changed via setOptimizationProfile() or setOptimizationProfileAsync(). - An input shape binding is changed via setInputShapeBinding(). - An input execution binding is changed via setBindingDimensions(). \warning The execution phase is timing critical during IExecutionContext but is not part of the timing loop when called from IBuilder. Performance bottlenecks of configurePlugin won't show up during engine building but will be visible during execution after calling functions that trigger layer resource updates.

Parameters:

in - The input tensors attributes that are used for configuration.

nbInputs - Number of input tensors.

out - The output tensors attributes that are used for configuration.

nbOutputs - Number of output tensors.

getWorkspaceSize

@Cast(value="size_t")
 @NoException(value=true)
public long getWorkspaceSize(@Const
                                                                             PluginTensorDesc inputs,
                                                                             int nbInputs,
                                                                             @Const
                                                                             PluginTensorDesc outputs,
                                                                             int nbOutputs)

\brief Find the workspace size required by the layer. This function is called after the plugin is configured, and possibly during execution. The result should be a sufficient workspace size to deal with inputs and outputs of the given size or any smaller problem.

Returns:

The workspace size.

enqueue

@NoException(value=true)
public int enqueue(@Const
                                            PluginTensorDesc inputDesc,
                                            @Const
                                            PluginTensorDesc outputDesc,
                                            @Cast(value="const void*const*")
                                            PointerPointer inputs,
                                            @Cast(value="void*const*")
                                            PointerPointer outputs,
                                            Pointer workspace,
                                            org.bytedeco.cuda.cudart.CUstream_st stream)

\brief Execute the layer.

Parameters:

inputDesc - how to interpret the memory for the input tensors.

outputDesc - how to interpret the memory for the output tensors.

inputs - The memory for the input tensors.

outputs - The memory for the output tensors.

workspace - Workspace for execution.

stream - The stream in which to execute the kernels.

Returns:

0 for success, else non-zero (which will cause engine termination).

enqueue

@NoException(value=true)
public int enqueue(@Const
                                            PluginTensorDesc inputDesc,
                                            @Const
                                            PluginTensorDesc outputDesc,
                                            @Cast(value="const void*const*") @ByPtrPtr
                                            Pointer inputs,
                                            @Cast(value="void*const*") @ByPtrPtr
                                            Pointer outputs,
                                            Pointer workspace,
                                            org.bytedeco.cuda.cudart.CUstream_st stream)