Interface GkeInferenceQuickstartGrpc.AsyncService
- All Known Implementing Classes:
GkeInferenceQuickstartGrpc.GkeInferenceQuickstartImplBase
- Enclosing class:
- GkeInferenceQuickstartGrpc
public static interface GkeInferenceQuickstartGrpc.AsyncService
GKE Inference Quickstart (GIQ) service provides profiles with performance metrics for popular models and model servers across multiple accelerators. These profiles help generate optimized best practices for running inference on GKE.
-
Method Summary
Modifier and TypeMethodDescriptiondefault voidfetchBenchmarkingData(FetchBenchmarkingDataRequest request, io.grpc.stub.StreamObserver<FetchBenchmarkingDataResponse> responseObserver) Fetches all of the benchmarking data available for a profile.default voidfetchModels(FetchModelsRequest request, io.grpc.stub.StreamObserver<FetchModelsResponse> responseObserver) Fetches available models.default voidfetchModelServers(FetchModelServersRequest request, io.grpc.stub.StreamObserver<FetchModelServersResponse> responseObserver) Fetches available model servers.default voidfetchModelServerVersions(FetchModelServerVersionsRequest request, io.grpc.stub.StreamObserver<FetchModelServerVersionsResponse> responseObserver) Fetches available model server versions.default voidfetchProfiles(FetchProfilesRequest request, io.grpc.stub.StreamObserver<FetchProfilesResponse> responseObserver) Fetches available profiles.default voidgenerateOptimizedManifest(GenerateOptimizedManifestRequest request, io.grpc.stub.StreamObserver<GenerateOptimizedManifestResponse> responseObserver) Generates an optimized deployment manifest for a given model and model server, based on the specified accelerator, performance targets, and configurations.
-
Method Details
-
fetchModels
default void fetchModels(FetchModelsRequest request, io.grpc.stub.StreamObserver<FetchModelsResponse> responseObserver) Fetches available models. Open-source models follow the Huggingface Hub `owner/model_name` format.
-
fetchModelServers
default void fetchModelServers(FetchModelServersRequest request, io.grpc.stub.StreamObserver<FetchModelServersResponse> responseObserver) Fetches available model servers. Open-source model servers use simplified, lowercase names (e.g., `vllm`).
-
fetchModelServerVersions
default void fetchModelServerVersions(FetchModelServerVersionsRequest request, io.grpc.stub.StreamObserver<FetchModelServerVersionsResponse> responseObserver) Fetches available model server versions. Open-source servers use their own versioning schemas (e.g., `vllm` uses semver like `v1.0.0`). Some model servers have different versioning schemas depending on the accelerator. For example, `vllm` uses semver on GPUs, but returns nightly build tags on TPUs. All available versions will be returned when different schemas are present.
-
fetchProfiles
default void fetchProfiles(FetchProfilesRequest request, io.grpc.stub.StreamObserver<FetchProfilesResponse> responseObserver) Fetches available profiles. A profile contains performance metrics and cost information for a specific model server setup. Profiles can be filtered by parameters. If no filters are provided, all profiles are returned. Profiles display a single value per performance metric based on the provided performance requirements. If no requirements are given, the metrics represent the inflection point. See [Run best practice inference with GKE Inference Quickstart recipes](https://cloud.google.com/kubernetes-engine/docs/how-to/machine-learning/inference/inference-quickstart#how) for details.
-
generateOptimizedManifest
default void generateOptimizedManifest(GenerateOptimizedManifestRequest request, io.grpc.stub.StreamObserver<GenerateOptimizedManifestResponse> responseObserver) Generates an optimized deployment manifest for a given model and model server, based on the specified accelerator, performance targets, and configurations. See [Run best practice inference with GKE Inference Quickstart recipes](https://cloud.google.com/kubernetes-engine/docs/how-to/machine-learning/inference/inference-quickstart) for deployment details.
-
fetchBenchmarkingData
default void fetchBenchmarkingData(FetchBenchmarkingDataRequest request, io.grpc.stub.StreamObserver<FetchBenchmarkingDataResponse> responseObserver) Fetches all of the benchmarking data available for a profile. Benchmarking data returns all of the performance metrics available for a given model server setup on a given instance type.
-