Runtime control gives a backend team a way to change an approved behavior after deployment. For an AI-assisted application, that might mean selecting a prompt version, limiting an expensive path, or exposing a feature to a small cohort. The useful engineering question is concrete: can the team explain the decision, bound its effects, and restore a known behavior when something fails?
This checklist is for a team preparing its first controlled rollout. It extends the architectural discussion in Aortem’s runtime-control overview into release decisions. It does not assume that a feature flag, an AI model, or a control panel can reverse every consequence of a request.
Define the runtime control boundary
Write down what the control is allowed to change. A prompt selector can choose among reviewed prompts; a rollout flag can choose which users see a compatible implementation. Neither should grant access to another tenant, skip authentication, or bypass a spending limit. Keep authorization and resource admission in server-side checks that run independently of rollout eligibility.
For each control, record an owner, a default, an expiry or review date, and the version of the behavior it selects. Give the control a name that describes a product decision. A name such as “new flow” becomes difficult to interpret when there are several new flows and a production incident.
- What behavior changes when the control is on?
- Which requests remain unchanged?
- What permissions and cost limits still apply?
- Who can approve an expansion or disable the behavior?
Make targeting reproducible and proportionate
A rollout should be explainable for a particular request. OpenFeature’s evaluation-context specification describes the attributes available to flag evaluation and how context can be merged. Some providers use a targeting key for consistent fractional evaluation. Confirm the behavior of the provider you actually use instead of assuming that all providers bucket requests in the same way.
Prefer a stable, appropriate identifier when a user should stay in one cohort. Passing only a fresh request identifier can make a person switch experiences between requests. At the same time, avoid adding personal details just because the context accepts arbitrary attributes. Include only the attributes the rule needs, and review what the provider receives and retains.
Test an eligible request, an ineligible request, a missing attribute, and an unexpected attribute value. Capture the effective rule version and the selected behavior in a bounded diagnostic record. A successful evaluation says which path was chosen; it does not prove that the path completed correctly.
Choose a safe fallback before rollout
Decide what happens when configuration is unavailable or invalid. For a new optional feature, the previous reviewed behavior may be a reasonable fallback. For a sensitive action, refusing the action may be the correct choice. Make that decision explicitly rather than letting a library default define product policy.
Exercise the fallback under realistic conditions: a provider timeout, malformed configuration, an unavailable model, and exhausted capacity. Use synthetic data and a bounded environment. Check that errors do not expose secrets or customer content. Record the observed result, not just whether the test process exited successfully.
Rehearse rollback and irreversible effects
Switching off a flag can stop new requests from taking a path. It cannot automatically retract a sent notification, refund a payment, undo an external API call, or restore data written in an incompatible format. List those effects before the first rollout and decide how they are prevented, reconciled, or recovered.
Run a rehearsal with a small synthetic cohort. Turn the behavior on, complete an operation, turn it off, and inspect both new and in-flight work. Confirm that older code can still read any records the new path created. If a schema transition is required, review it separately and define the compatible transition period.
Expand only on observed evidence
Choose a modest initial cohort and an observation window before exposure. Define the signals that matter: successful completion, error rates, latency, resource use, and relevant user feedback. A model response arriving is not the same as a useful operation completing. Compare like-for-like cohorts and account for sparse samples.
Keep a short release record containing the tested version, cohort, fallback, rollback rehearsal, thresholds, and decision owner. Expand when the evidence supports expansion; investigate or stop when a guardrail is crossed. Retire obsolete controls so that yesterday’s experiment does not become an undocumented dependency.
Preparing a controlled Dart or Flutter backend rollout? Explore Aortem’s developer products and use this checklist to identify the controls and evidence your own application needs.