MLOps & Model Lifecycle
Open-source tools that put a model's lifecycle under control: run tracking and a registry, data versioning, Kubernetes pipelines and drift reports. Interviewers ask which tool closes which gap.
on this pageshowhide
explore
- MLflow23 questions
- Experiment Tracking6 questions
- Registered Models & Aliases6 questions
- MLmodel Flavors & Signatures5 questions
- Scoring Server & Deployments6 questions
- DVC17 questions
- Data Versioning & Cache6 questions
- Remotes & Shared Storage5 questions
- Pipelines & Experiments6 questions
- Kubeflow12 questions
- KFP SDK & Runs6 questions
- Trainer & Katib6 questions
- Feature Store Platforms12 questions
- Feast6 questions
- Hosted Vendor Offerings6 questions
- Model Monitoring Tooling12 questions
- Data Validation Libraries6 questions
- Drift & Performance Platforms6 questions
- SHAP12 questions
- Explainers & Background Data6 questions
- Base Values & Plots6 questions
- LIME6 questions
questions
94 · 7 sectionsIn MLflow, when do you record a value with log_param rather than log_metric, and what happens if a param is logged twice?
basics
~20 slog_param records a run's configuration once per key; logging that key again with a different value raises "Changing param values is not allowed". log_metric records numeric results with a step and keeps every value, so anything that changes is a metric.
In MLflow, how does passing `registered_model_name=` to `log_model` differ from calling `mlflow.register_model` later, and how are versions numbered?
basics
~20 sregistered_model_name in log_model logs and registers in one call; mlflow.register_model registers an already-logged model later. Both create the registered model if it is missing and add the next version number under that name, starting at 1.
What request body does MLflow's scoring server accept at POST /invocations, and what does it send back?
basics
~10 sMLflow's scoring server takes a JSON object with exactly one of dataframe_split, dataframe_records, instances or inputs (plus optional params), or a text/csv or Parquet body, and replies with JSON shaped {"predictions": ...}.
In MLflow, what does a logged model's MLmodel file declare, and how does it let one scikit-learn model load through both mlflow.sklearn and mlflow.pyfunc?
basics
~20 sMLmodel is the YAML manifest at the root of a logged MLflow model directory, listing its flavors, signature, input example and environment files. The sklearn flavor loads the native Pipeline; python_function names mlflow.sklearn as its loader_module, so pyfunc gets one generic predict.
In MLflow, how do you log a nightly sweep's gradient-boosting trials as nested runs and then find the best trial from code?
basics
~10 sOpen a parent with mlflow.start_run, open each trial inside it with mlflow.start_run(nested=True), then call mlflow.search_runs filtered on tags.mlflow.parentRunId and ordered by the validation metric. Search ranks each run by that metric's latest logged value.
In DVC, after `dvc add data/camera-traps`, what do you commit to Git, and where do the 1.4 million photos actually live?
basics
~20 sYou commit data/camera-traps.dvc and data/.gitignore. The pointer records the directory's MD5 hash, size, file count and path; the photos move into DVC's content-addressed cache under .dvc/cache and are linked or copied back into the workspace.
In a DVC pipeline, what do `dvc.yaml` and `dvc.lock` each record, and why do both get committed to Git?
basics
~20 sdvc.yaml declares each stage's command, dependencies, outputs and tracked parameters; dvc.lock records what the last run actually used, with content hashes of dependencies and outputs and the parameter values. Committing both ties a commit to exact pipeline results.
In DVC, why does a fresh `git clone` of a project lack its tracked data, and how do `dvc push` and `dvc pull` move it?
basics
~20 sGit carries only the small .dvc and dvc.lock pointer files; the bytes stay in each machine's DVC cache. dvc push uploads the cached objects a remote lacks; dvc pull downloads them and places them in the workspace.
In DVC, after `git checkout` of last spring's tag, why are the workspace photos still this month's, and what restores the spring version?
basics
~20 sgit checkout only swaps the small .dvc pointer files; Git never tracked the photos. Running dvc checkout reads the restored pointers and rebuilds the workspace from the local cache, and dvc install adds a Git hook that runs it after every git checkout.
In DVC, an S3 secret key set with `dvc remote modify` ended up in a Git commit — why, and where should remote credentials live?
basics
~20 sWithout a level flag, dvc remote modify writes .dvc/config, the project file that Git tracks. Secrets belong in the Git-ignored .dvc/config.local (written with --local) or outside DVC config entirely: environment variables, a credentials file or profile, SSH config.
In the KFP v2 SDK, how do @dsl.component, @dsl.pipeline and the compiler turn Python functions into a run on a Kubeflow Pipelines backend?
basics
~20 s@dsl.component makes a typed function a container step, @dsl.pipeline wires steps into a graph through their outputs, Compiler().compile writes that graph as IR YAML, and kfp.Client submits it as a run in which each step gets its own container.
In Kubeflow Trainer v2, what does a TrainJob's runtimeRef point at, and which part of the training job does that runtime own?
basics
~20 sA TrainJob's runtimeRef names a ClusterTrainingRuntime (the default kind) or a same-namespace TrainingRuntime. That platform-owned runtime holds the JobSet template and mlPolicy; the TrainJob adds only run values such as image, command, node count and resources.
In a KFP v2 pipeline, when should a step pass data as an Input[Dataset] or Output[Model] artifact instead of a str parameter, and why?
basics
~20 sParameters (str, int, float, bool, list, dict) are small JSON values passed with a task's inputs. Datasets, models and metrics belong in artifacts: written to .path, then copied by the backend to object storage with lineage.
In a KFP v2 pipeline, how do you run the deploy step only when the new model beats the current one, and what replaced dsl.Condition?
basics
~20 sHave the evaluate step return both scores as parameter outputs and call deploy inside a with dsl.If(...) block that compares them; the backend evaluates the condition at run time. dsl.If, with dsl.Elif and dsl.Else, replaced the deprecated dsl.Condition.
Why does a KFP v2 lightweight @dsl.component that relies on a module-level import and a same-file helper fail on the cluster with NameError?
basics
~20 sA lightweight component ships only its own function source, run in base_image after KFP's own imports, so module-level imports and helpers never arrive. Import inside the body, list packages in packages_to_install, pass helpers through additional_funcs, or build a containerized component.
With Feast, how do `get_historical_features` and `get_online_features` differ in what you pass, which store each reads, and which value each row receives?
basics
~20 sget_historical_features takes entity rows with timestamps, reads the offline store and returns each row's latest value at or before its timestamp within ttl; get_online_features takes keys only and returns the latest value loaded into the online store.
Compared with self-hosting Feast, what do Databricks Feature Store, SageMaker Feature Store and Agent Platform Feature Store each run for you?
basics
~20 sFeast layers over storage you run: your online database, your materialization schedule, your feature server. Each managed service provisions its online store, keeps it in sync with its offline store, and serves reads through its own API.
With Feast, where does `feast materialize-incremental` take its start time from, and why can it skip warehouse rows that landed late?
basics
~20 sfeast materialize-incremental END_TS starts each feature view at the latest end recorded in the registry, or at now minus ttl on a first run. Rows stamped before that start are never loaded, however late they land.
Why does a Feast training set from `get_historical_features` show nulls for a nightly `spend_7d` feature, and how does the FeatureView's `ttl` cause it?
basics
~20 sA FeatureView's ttl caps how far back from each entity row's timestamp the historical join looks; a nightly value older than ttl at that instant is dropped, leaving a null. Set ttl above the refresh interval plus load delay.
In Databricks Feature Store, why can a model logged with FeatureEngineeringClient.log_model be scored with only primary keys, and what breaks that?
basics
~20 sfe.log_model stores the training set's FeatureLookup specifications with the model, so score_batch and Model Serving fetch the features by primary key themselves. Anything you did to load_df's DataFrame before training is not replayed at inference.
With TensorFlow Data Validation (TFDV), what do `generate_statistics_from_tfrecord`, `infer_schema` and `validate_statistics` each produce when you check a training batch?
basics
~10 sgenerate_statistics_from_tfrecord returns a DatasetFeatureStatisticsList summarising each feature; infer_schema turns those statistics into a draft Schema you review; validate_statistics checks a later batch's statistics against it and returns an Anomalies proto, empty when clean.
In Evidently 0.7, how do you build a drift report with pass/fail tests for a weekly churn batch, and what does running it return?
basics
~10 sWrap DataDriftPreset() in Report([...], include_tests=True) and call report.run(current, reference). The call returns a Snapshot holding per-column drift results plus pass/fail tests, which you save as HTML or JSON or upload to an Evidently workspace.
Should a weekly batch churn model use Evidently or NannyML in its scoring job, Arize AX or Fiddler, or a cloud monitor, and what decides it?
basics
~20 sThe choice turns on data residency, who operates storage and alerting, and label latency: libraries run in the job, managed platforms hold history and joins. SageMaker Model Monitor is closed to new customers; Google Model Monitoring v2 is in Preview.
In Deequ, how do a `VerificationSuite`'s Error-level and Warning-level checks decide the run's status, and does `run()` fail the Spark job?
basics
~20 sA failing Check reports Error or Warning according to its CheckLevel, and the VerificationResult status is the worst status across checks. run() only returns that result, so the job must test the status and fail itself.
In TFDV, when is a feature's `skew_comparator` evaluated versus its `drift_comparator`, and why can an `infinity_norm` threshold miss a numeric feature?
basics
~20 sThe skew_comparator runs only when validate_statistics receives serving_statistics (training against serving); the drift_comparator only when it receives previous_statistics (current against an earlier batch). An infinity_norm threshold reads categorical value counts, so numeric features need jensen_shannon_divergence.
In the shap library, which explainer fits a LightGBM regressor, a PyTorch network and an opaque predict function, and why?
basics
~20 sUse TreeExplainer for the LightGBM model, DeepExplainer or GradientExplainer for the PyTorch network, and PermutationExplainer or KernelExplainer for the opaque function: the more of the model an explainer can read, the faster and more exact it is.
How does `shap.plots.waterfall` build one patient's readmission explanation from the Explanation's `.base_values`, `.values` and `.data`?
basics
~20 sA shap waterfall takes one row of an Explanation: it starts at base_values, the average model output E[f(X)], adds each feature's SHAP value from .values as a bar labelled with its .data value, and ends at that patient's output f(x).
A shap waterfall for a binary XGBoost readmission model shows f(x) = -0.85 while `predict_proba` gives 0.30 — why, and how do you reconcile them?
basics
~20 sBy default TreeExplainer explains a binary XGBoost model on its raw log-odds margin, so base value plus SHAP values equals -0.85 in log-odds. The logistic function maps it to 0.30, the predicted probability; individual bars have no fixed probability equivalent.
When shap.Explainer runs with algorithm="auto", how does it decide which explainer to build for a given model and masker?
basics
~20 sshap.Explainer tries the linear, tree and additive explainers in turn; failing those, a callable model gets exact or permutation by masker type and feature count, or partition for text and image maskers. Kernel, Deep and Gradient are never chosen.
When shap.Explainer is handed a 20,000-row DataFrame as its masker, how many rows does it use, and how should you size that background?
basics
~10 sIt uses 100 rows: the DataFrame is wrapped in shap.maskers.Independent, whose max_samples=100 sub-samples it with shap.utils.sample. Size the background on purpose, with max_samples or shap.sample, because model-agnostic cost grows linearly with it.
In lime, what does LimeTabularExplainer.explain_instance return for one row of a classifier, and how do you read its as_list output?
basics
~20 sIt returns an Explanation object. as_list(label=1) gives (feature condition, weight) pairs sorted by absolute weight: coefficients of a local linear surrogate fitted to the classifier's probability for that label, so positive weights push that class up.
When choosing between the lime and shap packages to explain individual predictions of a production model, which conditions decide the choice?
basics
~20 sThe conditions decide, not the brand: a tree ensemble shap can read gets exact, additive attributions without sampling; a black box means sampling in either package; lime gives readable bin conditions without additivity, and its last release was in 2020.
In lime's LimeTabularExplainer, what do categorical_features and discretize_continuous change about an explanation, and how must categorical columns be encoded?
basics
~20 scategorical_features lists integer-coded columns that lime samples from their training frequencies and explains as column=value. discretize_continuous=True, the default, bins continuous columns at training quartiles, so a weight describes being in the row's bin, not a per-unit slope.
In lime 0.2.0.1's tabular explainer, which settings go on the LimeTabularExplainer constructor and which on explain_instance, and why does the split matter?
basics
~20 sThe constructor fixes everything derived from training data: kernel_width, discretisation, categorical columns, feature_selection and random_state. explain_instance takes per-row choices: num_features, num_samples, labels or top_labels, distance_metric and model_regressor. Changing a constructor setting means building a new explainer.
A lime 0.2.0.1 tabular explanation shows exp.score near 0.3 and exp.local_pred far from the model's probability; what do they measure and what do you change?
basics
~20 sexp.score is the weighted R squared of lime's ridge surrogate on its perturbed samples, and exp.local_pred is that surrogate's prediction for the row. A low score or a distant local_pred means the weights describe the model poorly around this row.