Hybrid AI in Flutter: On-Device and Cloud Routing

A Flutter app does not have to choose permanently between an on-device model and a cloud model. A better architecture evaluates each request and sends it to the environment that can handle it safely and well. Private or offline work stays on the device; requests that need deeper reasoning, a larger context window, or current knowledge go to a protected cloud service.

This is hybrid AI: not two unrelated integrations, but one routing policy with two execution paths. In this guide, we will design that policy, implement a testable Dart router, connect it to local and cloud adapters, and handle the failure cases that matter in production.

Quick answer: start with deterministic rules. Route sensitive data locally, keep an offline-only path when the network is unusable, send knowledge-heavy or complex work to the cloud, and use on-device inference for simple, frequent requests. Never silently switch providers after a streamed response has started, and never let a fallback violate the original privacy decision.

What hybrid AI means in a Flutter app

Hybrid AI is a per-request decision. The user can ask four questions in the same screen and receive answers from different models without learning which SDK produced each response. The application owns the policy; models are replaceable execution engines behind it.

Request	Preferred route	Reason
Summarize a private note	On-device only	The content should not leave the phone.
Rewrite a short message in airplane mode	On-device	The cloud is unavailable and the task is small.
Plan a detailed seven-day trip	Cloud	The task benefits from stronger reasoning and broader knowledge.
Classify a support message	On-device	It is short, repetitive, latency-sensitive, and inexpensive locally.

The useful mental model is a policy engine with hard constraints and soft preferences. Privacy and availability are hard constraints. Quality, latency, battery use, and cost are preferences that can be tuned after measuring real traffic.

Why not use only the cloud?

Cloud models offer a high capability ceiling, but every call depends on connectivity and introduces network latency. It can also create usage costs and a data-governance obligation. Shipping an API key inside a mobile binary is not a safe shortcut: mobile applications can be inspected, so cloud credentials and authorization decisions belong behind a service you control or a managed proxy designed for client applications.

An on-device model removes the round trip, continues working offline, and keeps input on the user's hardware. It is a strong fit for summarization, classification, rewriting, embeddings, and private retrieval over local data. The tradeoff is a real hardware ceiling. Model files consume storage, inference consumes memory and battery, and performance varies across the device fleet.

Hybrid routing accepts both sets of constraints. The goal is not to make a small local model imitate a datacenter model. It is to reserve the cloud for the requests that justify it and make the local path a first-class product experience rather than an emergency placeholder.

Choose a routing approach

Build adapters yourself

A small application can wrap its local runtime and cloud endpoint behind one Dart interface. This gives complete control and keeps the product policy obvious. It also means your team owns message conversion, streaming, telemetry, retries, and model lifecycle management.

Use Firebase AI Logic

Firebase AI Logic provides Flutter access to cloud-hosted Gemini models and protects the upstream API through Firebase services such as App Check. Firebase also documents hybrid inference for selected native and web SDKs. Check the supported platform and SDK combination before treating its on-device path as a single cross-platform Flutter solution.

Use Genkit with Flutter Gemma

Genkit for Dart provides model abstractions, flows, streaming, middleware, tracing, and evaluation tooling. genkit_flutter_gemma registers a Flutter Gemma model with the same Genkit generation interface used by other providers. This is useful when the app needs explicit, testable routing and the freedom to replace either side.

The package ecosystem changes quickly. The versions used while preparing this guide in September 2026 are genkit 0.17.0, genkit_flutter_gemma 0.5.0, genkit_google_genai 0.3.1, and flutter_gemma 1.8.4. Pin compatible versions in a real project and review their changelogs before upgrading.

1. Define the application-facing contract

Keep routing inputs explicit. The router should not guess whether text is private by running a few regular expressions over the prompt. The screen or feature that collected the data usually knows more: a message written in a medical-record view is sensitive by context, even when it contains no obvious identifier.

enum AIExecutionTarget { local, cloud }

class HybridRequest {
  const HybridRequest({
    required this.prompt,
    this.containsPrivateData = false,
    this.needsFreshKnowledge = false,
    this.isComplex = false,
  });

  final String prompt;
  final bool containsPrivateData;
  final bool needsFreshKnowledge;
  final bool isComplex;
}

abstract interface class AIService {
  Stream<String> generate(String prompt);
}

The feature passes facts, not a provider name. That separation prevents UI code from accumulating rules such as “use cloud on this button but local on that screen.” It also makes the policy easy to unit test without loading a model or making a network request.

2. Make the routing order unambiguous

Evaluate hard constraints first. A sensible starting order is privacy, usable connectivity, capability, and then cost or latency. “Usable connectivity” should be based on whether your service can be reached within a reasonable timeout, not only whether the operating system reports Wi-Fi or mobile data.

class HybridRoutingPolicy {
  const HybridRoutingPolicy();

  AIExecutionTarget select({
    required HybridRequest request,
    required bool cloudIsReachable,
    required bool localModelIsReady,
  }) {
    if (request.containsPrivateData) {
      if (!localModelIsReady) {
        throw StateError('Private request requires the on-device model.');
      }
      return AIExecutionTarget.local;
    }

    if (!cloudIsReachable) {
      if (!localModelIsReady) {
        throw StateError('No AI provider is currently available.');
      }
      return AIExecutionTarget.local;
    }

    if (request.needsFreshKnowledge || request.isComplex) {
      return AIExecutionTarget.cloud;
    }

    return localModelIsReady
        ? AIExecutionTarget.local
        : AIExecutionTarget.cloud;
  }
}

This policy is deliberately boring. That is an advantage: a security reviewer can see why a request stays local, and product analytics can explain which rule selected the cloud. Model-based classification can be added later for ambiguous requests, but it should not replace deterministic privacy rules.

3. Connect the on-device model

Flutter Gemma separates its core package from opt-in inference backends. Add the backend required by the model format you plan to ship, initialize it during application startup, and install the model before registering it with Genkit. The host application remains responsible for model download, consent, progress, activation, and deletion.

flutter pub add flutter_gemma flutter_gemma_litertlm
flutter pub add genkit genkit_flutter_gemma

After the model is installed, register its configuration with the Flutter Gemma Genkit plugin. Use the exact model name and type from your installation flow:

final localAI = Genkit(
  plugins: [
    GenkitFlutterGemmaPlugin(
      models: [
        FlutterGemmaModelConfig(
          name: 'gemma-3-1b-it',
          modelType: ModelType.gemmaIt,
        ),
      ],
    ),
  ],
);

class LocalAIService implements AIService {
  LocalAIService(this.ai);

  final Genkit ai;

  @override
  Stream<String> generate(String prompt) {
    return ai
        .generateStream(
          model: flutterGemma.model('gemma-3-1b-it'),
          prompt: prompt,
        )
        .map((chunk) => chunk.text);
  }
}

A production app should expose model readiness as state. Download large files only after explaining their storage cost, persist the installed version, and pre-warm inference when the device is idle instead of making the user's first message pay the entire cold-start cost.

4. Put the cloud model behind a protected endpoint

The Flutter application should call an authenticated endpoint that you control. That endpoint can run a Genkit flow and use a cloud plugin such as genkit_google_genai. A flow adds typed inputs and outputs, observability, and a stable HTTP boundary while keeping provider credentials off the device.

final ai = Genkit(plugins: [googleAI(), RetryPlugin()]);

final answerFlow = ai.defineFlow(
  name: 'answer',
  inputSchema: .string(),
  outputSchema: .string(),
  fn: (prompt, _) async {
    final response = await ai.generate(
      model: googleAI.gemini('gemini-flash-latest'),
      prompt: prompt,
      use: [retry(maxRetries: 2)],
    );
    return response.text;
  },
);

Authenticate the caller, enforce quotas, validate request size, and avoid logging raw prompts by default. Firebase AI Logic is another appropriate cloud path when a direct client SDK and App Check fit the product. In either design, the routing decision remains in application code while secrets and billing controls remain server-side.

5. Add safe fallback without corrupting a stream

Fallback is useful only when it preserves the request's constraints. A private request must never fall back to the cloud. An offline request cannot fall back to an unreachable endpoint. For an ordinary request, fallback is safe only before the primary provider emits its first chunk.

class HybridAIService {
  HybridAIService({
    required this.local,
    required this.cloud,
    required this.policy,
  });

  final AIService local;
  final AIService cloud;
  final HybridRoutingPolicy policy;

  Stream<String> generate(
    HybridRequest request, {
    required bool cloudIsReachable,
    required bool localModelIsReady,
  }) async* {
    final target = policy.select(
      request: request,
      cloudIsReachable: cloudIsReachable,
      localModelIsReady: localModelIsReady,
    );

    final primary = target == AIExecutionTarget.local ? local : cloud;
    final canFallBack = !request.containsPrivateData &&
        cloudIsReachable &&
        localModelIsReady;
    final fallback = target == AIExecutionTarget.local ? cloud : local;
    var emittedFirstChunk = false;

    try {
      await for (final chunk in primary.generate(request.prompt)) {
        emittedFirstChunk = true;
        yield chunk;
      }
    } catch (_) {
      if (emittedFirstChunk || !canFallBack) rethrow;
      yield* fallback.generate(request.prompt);
    }
  }
}

Once text is visible, switching models can repeat content, change tone, or complete a sentence with a contradictory answer. Stop the stream and offer a retry instead. The user should know that the response failed; the app should not splice two providers together invisibly.

6. Test the policy before testing the models

Router tests should be deterministic and fast. Use fake services that record calls, then cover each decision branch:

A private request always selects local inference.
A private request fails closed when the local model is unavailable.
An offline request never attempts the cloud.
A complex request selects the cloud when it is reachable.
A simple request prefers a ready local model.
A provider may fall back only before the first streamed chunk.
No fallback can override a privacy constraint.

Model quality requires a separate evaluation set made from representative tasks. Compare local and cloud answers using the same prompts, measure latency on low- and mid-range devices, and record how often the policy escalates. Complexity routing should be introduced only when those evaluations show a reliable boundary. Otherwise a simple rule will be easier to understand and often more accurate.

Production details that are easy to miss

Model delivery

Large model files should normally be downloaded on demand rather than bundled with the application. Show file size, progress, retry state, and a way to remove the model. Plan for resumable downloads and enough free storage for both the download and installation process.

Memory and battery

Test on the weakest device you support, not only a development flagship. Release model sessions when the feature is idle, avoid loading multiple large models simultaneously, and account for thermal throttling during long generations.

Platform capability

“Flutter supports the platform” does not guarantee that one model artifact or inference backend works identically everywhere. Validate model formats, hardware acceleration, minimum operating systems, and browser requirements independently. Make capability detection part of startup and expose a clear fallback state to the UI.

Observability without surveillance

Record the chosen route, timing, model version, token counts, error category, and fallback outcome. Do not record private prompts merely because tracing is available. Redaction should happen before telemetry leaves the device, and sensitive local-only operations may need metadata-only traces.

A practical rollout sequence

Ship one local model and one protected cloud endpoint behind the same interface.
Implement privacy and offline routing with deterministic rules.
Measure local latency, failure rates, and answer quality across real devices.
Add cloud retry for transient failures and safe pre-stream fallback.
Introduce cost or complexity routing only after evaluations justify it.

Most applications do not need an autonomous model to choose another model. A small policy object, explicit request metadata, and two well-tested adapters are enough to deliver meaningful privacy, offline support, and cost control.

Final takeaway

Hybrid AI works best when routing is treated as product policy rather than SDK glue. Keep sensitive information on-device, protect cloud credentials behind a backend or managed proxy, fail closed when a hard constraint cannot be satisfied, and measure quality before adding sophisticated classifiers. The local and cloud models will change; a clear contract and testable policy let the application evolve without rewriting every AI feature.

This guide was independently written and updated using current package documentation. It was inspired by Sasha Denisov's article “Hybrid AI in Flutter: Routing Between On-Device and Cloud Models”.

Hybrid AI in Flutter: Routing Between On-Device and Cloud Models