Interface GkeInferenceQuickstartGrpc.AsyncService

All Known Implementing Classes:
GkeInferenceQuickstartGrpc.GkeInferenceQuickstartImplBase
Enclosing class:
GkeInferenceQuickstartGrpc

public static interface GkeInferenceQuickstartGrpc.AsyncService
 GKE Inference Quickstart (GIQ) service provides profiles with performance
 metrics for popular models and model servers across multiple accelerators.
 These profiles help generate optimized best practices for running inference
 on GKE.
 
  • Method Details

    • fetchModels

      default void fetchModels(FetchModelsRequest request, io.grpc.stub.StreamObserver<FetchModelsResponse> responseObserver)
       Fetches available models. Open-source models follow the Huggingface Hub
       `owner/model_name` format.
       
    • fetchModelServers

      default void fetchModelServers(FetchModelServersRequest request, io.grpc.stub.StreamObserver<FetchModelServersResponse> responseObserver)
       Fetches available model servers. Open-source model servers use simplified,
       lowercase names (e.g., `vllm`).
       
    • fetchModelServerVersions

      default void fetchModelServerVersions(FetchModelServerVersionsRequest request, io.grpc.stub.StreamObserver<FetchModelServerVersionsResponse> responseObserver)
       Fetches available model server versions. Open-source servers use their own
       versioning schemas (e.g., `vllm` uses semver like `v1.0.0`).
       Some model servers have different versioning schemas depending on the
       accelerator. For example, `vllm` uses semver on GPUs, but returns nightly
       build tags on TPUs. All available versions will be returned when different
       schemas are present.
       
    • fetchProfiles

      default void fetchProfiles(FetchProfilesRequest request, io.grpc.stub.StreamObserver<FetchProfilesResponse> responseObserver)
       Fetches available profiles. A profile contains performance metrics and
       cost information for a specific model server setup. Profiles can be
       filtered by parameters. If no filters are provided, all profiles are
       returned.
       Profiles display a single value per performance metric based on the
       provided performance requirements. If no requirements are given, the
       metrics represent the inflection point. See [Run best practice inference
       with GKE Inference Quickstart
       recipes](https://cloud.google.com/kubernetes-engine/docs/how-to/machine-learning/inference/inference-quickstart#how)
       for details.
       
    • generateOptimizedManifest

      default void generateOptimizedManifest(GenerateOptimizedManifestRequest request, io.grpc.stub.StreamObserver<GenerateOptimizedManifestResponse> responseObserver)
       Generates an optimized deployment manifest for a given model and model
       server, based on the specified accelerator, performance targets, and
       configurations. See [Run best practice inference with GKE Inference
       Quickstart
       recipes](https://cloud.google.com/kubernetes-engine/docs/how-to/machine-learning/inference/inference-quickstart)
       for deployment details.
       
    • fetchBenchmarkingData

      default void fetchBenchmarkingData(FetchBenchmarkingDataRequest request, io.grpc.stub.StreamObserver<FetchBenchmarkingDataResponse> responseObserver)
       Fetches all of the benchmarking data available for a profile. Benchmarking
       data returns all of the performance metrics available for a given model
       server setup on a given instance type.