From 910110f48c8c37cb6ea1094b665c9fdfd029ba2a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Javier=20de=20la=20R=C3=BAa=20Mart=C3=ADnez?= Date: Thu, 18 Jun 2026 15:49:12 +0200 Subject: [PATCH] [FSTORE-1412] Add feature/model monitoring docs --- .../fs/feature_group/feature_monitoring.md | 1 + .../fs/feature_view/feature_monitoring.md | 9 +- docs/concepts/mlops/model_monitoring.md | 37 +++ docs/concepts/mlops/serving.md | 1 + .../fs/feature_group/feature_monitoring.md | 119 +++++++-- .../distribution_comparison.md | 63 +++++ .../feature_monitoring_advanced.md | 125 ---------- .../fs/feature_monitoring/index.md | 24 +- .../scheduled_statistics.md | 4 +- .../statistics_comparison.md | 9 +- .../fs/feature_view/feature_monitoring.md | 233 +++++++++++------- .../mlops/model_monitoring/index.md | 31 +++ .../model_monitoring/model_monitoring.md | 193 +++++++++++++++ .../mlops/serving/inference-logger.md | 4 + mkdocs.yml | 13 +- 15 files changed, 599 insertions(+), 267 deletions(-) create mode 100644 docs/concepts/mlops/model_monitoring.md create mode 100644 docs/user_guides/fs/feature_monitoring/distribution_comparison.md delete mode 100644 docs/user_guides/fs/feature_monitoring/feature_monitoring_advanced.md create mode 100644 docs/user_guides/mlops/model_monitoring/index.md create mode 100644 docs/user_guides/mlops/model_monitoring/model_monitoring.md diff --git a/docs/concepts/fs/feature_group/feature_monitoring.md b/docs/concepts/fs/feature_group/feature_monitoring.md index fb7ff8a0ba..b34d85ac30 100644 --- a/docs/concepts/fs/feature_group/feature_monitoring.md +++ b/docs/concepts/fs/feature_group/feature_monitoring.md @@ -14,6 +14,7 @@ Statistics are computed on the whole or a subset of feature data (i.e., detectio ## Statistics Comparison In addition to scheduled statistics, you can enable the comparison of statistics against a reference subset of feature data (i.e., reference window) and define the criteria for this comparison including the statistics metric to compare and a threshold to identify anomalous values. +The comparison can be done on a single scalar metric (e.g., the mean) or on the whole feature distribution using distance metrics such as PSI or KL divergence. !!! info "Feature Monitoring Guide" More information can be found in the [Feature monitoring guide](../../../user_guides/fs/feature_monitoring/index.md). diff --git a/docs/concepts/fs/feature_view/feature_monitoring.md b/docs/concepts/fs/feature_view/feature_monitoring.md index b1c74dcc9e..9d1608dc25 100644 --- a/docs/concepts/fs/feature_view/feature_monitoring.md +++ b/docs/concepts/fs/feature_view/feature_monitoring.md @@ -14,6 +14,13 @@ Statistics are computed on the whole or a subset of feature data (i.e., detectio ## Statistics Comparison In addition to scheduled statistics, you can enable the comparison of statistics against a reference subset of feature data (i.e., reference window), typically a training dataset, and define the criteria for this comparison including the statistics metric to compare and a threshold to identify anomalous values. +The comparison can be done on a single scalar metric (e.g., the mean) or on the whole feature distribution using distance metrics such as PSI or KL divergence. + +## Model Monitoring + +A Feature View backs the features served to a model in production. +By comparing the model's logged inference data against the training dataset it was trained on, you can detect drift between training and serving and decide when to retrain. +This is known as model monitoring and reuses the same statistics and distribution comparison machinery as feature monitoring. !!! info "Feature Monitoring Guide" - More information can be found in the [Feature monitoring guide](../../../user_guides/fs/feature_monitoring/index.md). + More information can be found in the [Feature monitoring guide](../../../user_guides/fs/feature_monitoring/index.md) and the [Model Monitoring guide](../../../user_guides/mlops/model_monitoring/index.md). diff --git a/docs/concepts/mlops/model_monitoring.md b/docs/concepts/mlops/model_monitoring.md new file mode 100644 index 0000000000..af6319ce0b --- /dev/null +++ b/docs/concepts/mlops/model_monitoring.md @@ -0,0 +1,37 @@ +Model monitoring lets you track how a deployed model behaves in production by comparing the data it serves against the data it was trained on. + +When a model runs in production, the statistical properties of its inputs and predictions can drift away from those of the training data. +This degrades model quality silently, without any error being raised. +Model monitoring detects this drift early so you can decide whether to retrain the model. + +## How it works + +Model monitoring builds on two existing Hopsworks capabilities: + +- **Feature logging**: a model deployment logs the features it serves and its predictions to the feature view's logging feature group through the Feature View logging APIs. + See the [Feature Logging guide](../../user_guides/fs/feature_view/feature_logging.md). +- **Feature monitoring**: Hopsworks computes statistics over windows of feature data and compares them against a reference, optionally raising alerts on significant shifts. + See the [Feature Monitoring concept](../fs/feature_view/feature_monitoring.md). + +!!! info "Feature logging vs. the inference logger" + Hopsworks provides two separate inference logging mechanisms. + The [inference logger](../../user_guides/mlops/serving/inference-logger.md) stores the model inputs and predictions from inference requests and responses into Kafka, for later consumption and analysis. + [Feature logging](../../user_guides/fs/feature_view/feature_logging.md) supports more fine-grained logging of inference logs and features, enabling feature monitoring and model monitoring. + Model monitoring relies on feature logging, not on the inference logger. + +A model monitoring configuration is a feature monitoring configuration over the logging feature group, filtered to a single model and version. +The detection window covers the recently served inference data, and the reference defaults to the training dataset version that was used to train that model. +By comparing the two — on a scalar metric or on the whole feature distribution — Hopsworks detects training/serving skew and drift over time. + +## Where to configure it + +Because monitoring is anchored on the feature view that backs the model, you can configure model monitoring from whichever entity is most convenient: + +- a **model deployment**, when operating a model in production. +- a **model** in the model registry. +- a **feature view**, when working directly with the feature data. + +All three resolve to the same underlying configuration. + +!!! info "Model Monitoring Guide" + More information can be found in the [Model Monitoring guide](../../user_guides/mlops/model_monitoring/index.md). diff --git a/docs/concepts/mlops/serving.md b/docs/concepts/mlops/serving.md index be8168c033..c0a4196150 100644 --- a/docs/concepts/mlops/serving.md +++ b/docs/concepts/mlops/serving.md @@ -14,6 +14,7 @@ A KServe model deployment can include the following components: **`Inference Logger`** : Hopsworks logs inputs and outputs of transformers and predictors to a ^^Kafka topic^^ that is part of the same project as the model. + This is for storing inference requests and responses for later consumption and analysis, and is separate from the feature logging that powers [Model Monitoring](model_monitoring.md). Not available for vLLM deployments. **`Inference Batcher`** diff --git a/docs/user_guides/fs/feature_group/feature_monitoring.md b/docs/user_guides/fs/feature_group/feature_monitoring.md index b6a06e1c37..4c6766a501 100644 --- a/docs/user_guides/fs/feature_group/feature_monitoring.md +++ b/docs/user_guides/fs/feature_group/feature_monitoring.md @@ -7,7 +7,7 @@ Before continuing with this guide, see the [Feature monitoring guide](../feature !!! warning "Limited UI support" Currently, feature monitoring can only be configured using the [Hopsworks Python library](https://pypi.org/project/hopsworks). - However, you can enable/disable a feature monitoring configuration or trigger the statistics comparison manually from the UI, as shown in the [Advanced guide](../feature_monitoring/feature_monitoring_advanced.md). + However, you can enable/disable a feature monitoring configuration or trigger the statistics comparison manually from the UI. ## Code @@ -48,6 +48,8 @@ Connect the client running your notebooks to Hopsworks. fs = project.get_feature_store() ``` +See the API reference for [`hopsworks.login`][hopsworks.login] and [`Project.get_feature_store`][hopsworks_common.project.Project.get_feature_store]. + You will be prompted to paste your API key to connect the notebook to your project. The `fs` Feature Store entity is now ready to be used to insert or read data from Hopsworks. @@ -76,6 +78,8 @@ The following is a code example for getting or creating a Feature Group with nam trans_fg.insert(transactions_df) ``` +See the API reference for [`FeatureStore.get_feature_group`][hsfs.feature_store.FeatureStore.get_feature_group] and [`FeatureStore.get_or_create_feature_group`][hsfs.feature_store.FeatureStore.get_or_create_feature_group]. + ### Step 2: Initialize configuration #### Scheduled statistics @@ -86,56 +90,57 @@ You can setup statistics monitoring on a ==single feature or multiple features== ```python # compute statistics for all the features - fg_monitoring_config = trans_fg.create_statistics_monitoring( + fg_monitoring_config = trans_fg.create_scheduled_statistics( name="trans_fg_all_features_monitoring", description="Compute statistics on all data of all features of the Feature Group on a daily basis", ) - # or for a single feature - fg_monitoring_config = trans_fg.create_statistics_monitoring( + # or for one or more specific features + fg_monitoring_config = trans_fg.create_scheduled_statistics( name="trans_fg_amount_monitoring", - description="Compute statistics on all data of a single feature of the Feature Group on a daily basis", - feature_name="amount", + description="Compute statistics on all data of selected features of the Feature Group on a daily basis", + feature_names=["amount"], ) ``` +See the API reference for [`FeatureGroup.create_scheduled_statistics`][hsfs.feature_group.FeatureGroup.create_scheduled_statistics]. + #### Statistics comparison -When enabling the comparison of statistics in a feature monitoring configuration, you need to specify a ==single feature== of your Feature Group. -You can create multiple feature monitoring configurations for the same Feature Group, but each of them should point to a single feature in the Feature Group. +When enabling the comparison of statistics in a feature monitoring configuration, the feature to compare is selected later in the `compare_on` (or `compare_on_distribution`) method, not in `create_feature_monitoring`. +You can create multiple feature monitoring configurations for the same Feature Group. === "Python" ```python fg_monitoring_config = trans_fg.create_feature_monitoring( name="trans_fg_amount_monitoring", - feature_name="amount", - description="Compute descriptive statistics on the amount Feature of the Feature Group on a daily basis", + description="Compute and compare descriptive statistics on the Feature Group on a daily basis", ) ``` -#### Custom schedule or percentage of window data +See the API reference for [`FeatureGroup.create_feature_monitoring`][hsfs.feature_group.FeatureGroup.create_feature_monitoring]. + +#### Custom schedule By default, the computation of statistics is scheduled to run endlessly, every day at 12PM. You can modify the default schedule by adjusting the `cron_expression`, `start_date_time` and `end_date_time` parameters. +To compute statistics on only a subset of the feature data, use the `row_percentage` parameter of `with_detection_window` (see Step 3). === "Python" ```python - fg_monitoring_config = trans_fg.create_statistics_monitoring( + fg_monitoring_config = trans_fg.create_scheduled_statistics( name="trans_fg_all_features_monitoring", description="Compute statistics on all data of all features of the Feature Group on a weekly basis", cron_expression="0 0 12 ? * MON *", # weekly - row_percentage=0.8, # use 80% of the data ) # or fg_monitoring_config = trans_fg.create_feature_monitoring( name="trans_fg_amount_monitoring", - feature_name="amount", - description="Compute descriptive statistics on the amount Feature of the Feature Group on a weekly basis", + description="Compute and compare descriptive statistics on the Feature Group on a weekly basis", cron_expression="0 0 12 ? * MON *", # weekly - row_percentage=0.8, # use 80% of the data ) ``` @@ -155,9 +160,12 @@ Additionally, you can specify the percentage of feature data on which statistics ) ``` +See the API reference for [`FeatureMonitoringConfig.with_detection_window`][hsfs.core.feature_monitoring_config.FeatureMonitoringConfig.with_detection_window]. + ### Step 4: (Optional) Define a reference window -When setting up feature monitoring for a Feature Group, reference windows can be either a regular window or a specific value (i.e., window of size 1). +When setting up feature monitoring for a Feature Group, you can compare the detection statistics against a reference window of feature data. +A reference window is defined with the `with_reference_window` method. === "Python" @@ -168,33 +176,68 @@ When setting up feature monitoring for a Feature Group, reference windows can be time_offset="2w", # starting from two weeks ago row_percentage=0.8, # use 80% of the data ) - - # or a specific value - fm_monitoring_config.with_reference_value( - value=100, - ) ``` -### Step 5: (Optional) Define the statistics comparison criteria +See the API reference for [`FeatureMonitoringConfig.with_reference_window`][hsfs.core.feature_monitoring_config.FeatureMonitoringConfig.with_reference_window]. + +!!! info "Comparing against a specific value" + Instead of a reference window, you can compare the detection statistics against a fixed reference value (i.e., a window of size 1). + In that case, skip this step and pass the `specific_value` parameter to `compare_on` in Step 5. + +### Step 5.A: (Optional) Compare on a scalar metric In order to compare detection and reference statistics, you need to provide the criteria for such comparison. -First, you select the metric to consider in the comparison using the `metric` parameter. +First, you select the feature and the metric to consider in the comparison using the `feature_name` and `metric` parameters. Then, you can define a relative or absolute threshold using the `threshold` and `relative` parameters. === "Python" ```python + # compare against a reference window fm_monitoring_config.compare_on( + feature_name="amount", # the feature to compare metric="mean", threshold=0.2, # a relative change over 20% is considered anomalous relative=True, # relative or absolute change strict=False, # strict or relaxed comparison ) + + # or compare against a specific value instead of a reference window + fm_monitoring_config.compare_on( + feature_name="amount", + metric="mean", + specific_value=100, + threshold=0.2, + relative=True, + ) ``` +See the API reference for [`FeatureMonitoringConfig.compare_on`][hsfs.core.feature_monitoring_config.FeatureMonitoringConfig.compare_on]. + !!! info "Difference values and thresholds" For more information about the computation of difference values and the comparison against threshold bounds see the [Comparison criteria section](../feature_monitoring/statistics_comparison.md#comparison-criteria) in the Statistics comparison guide. +### Step 5.B: (Optional) Compare on the whole distribution + +Alternatively, instead of a single scalar metric, you can detect drift in the shape of a feature's distribution using `compare_on_distribution`. +Select a distribution distance metric (e.g., `PSI`) and a threshold. +A reference window (Step 4) is required for distribution comparison. + +=== "Python" + + ```python + fm_monitoring_config.compare_on_distribution( + feature_name="amount", # the feature to compare + metric="PSI", + threshold=0.2, # a distance above 0.2 is considered a significant shift + ) + ``` + +See the API reference for [`FeatureMonitoringConfig.compare_on_distribution`][hsfs.core.feature_monitoring_config.FeatureMonitoringConfig.compare_on_distribution]. + +!!! tip "More distribution options" + See the [Distribution comparison guide](../feature_monitoring/distribution_comparison.md) for the full list of metrics and binning strategies. + ### Step 6: Save configuration Finally, you can save your feature monitoring configuration by calling the `save` method. @@ -206,5 +249,29 @@ Once the configuration is saved, the schedule for the statistics computation and fm_monitoring_config.save() ``` -!!! info "Next steps" - See the [Advanced guide](../feature_monitoring/feature_monitoring_advanced.md) to learn how to delete, disable or trigger feature monitoring manually. +See the API reference for [`FeatureMonitoringConfig.save`][hsfs.core.feature_monitoring_config.FeatureMonitoringConfig.save]. + +### Retrieve configurations and history + +Once saved, you can retrieve your feature monitoring configurations and the results of past executions directly from the Feature Group. + +=== "Python" + + ```python + # fetch all configurations attached to the feature group + configs = trans_fg.get_feature_monitoring_configs() + + # or a single configuration by name + config = trans_fg.get_feature_monitoring_configs(name="trans_fg_amount_monitoring") + + # fetch the history of monitoring results (with computed statistics) + history = trans_fg.get_feature_monitoring_history( + config_name="trans_fg_amount_monitoring", + with_statistics=True, + ) + ``` + +See the API reference for [`FeatureGroup.get_feature_monitoring_configs`][hsfs.feature_group.FeatureGroup.get_feature_monitoring_configs] and [`FeatureGroup.get_feature_monitoring_history`][hsfs.feature_group.FeatureGroup.get_feature_monitoring_history]. + +!!! info "Explore the API" + The [`FeatureMonitoringConfig`][hsfs.core.feature_monitoring_config.FeatureMonitoringConfig] reference documents the full set of available methods, such as enabling or disabling a configuration, triggering it manually, or deleting it. diff --git a/docs/user_guides/fs/feature_monitoring/distribution_comparison.md b/docs/user_guides/fs/feature_monitoring/distribution_comparison.md new file mode 100644 index 0000000000..0f206f2547 --- /dev/null +++ b/docs/user_guides/fs/feature_monitoring/distribution_comparison.md @@ -0,0 +1,63 @@ +# Distribution Comparison + +Distribution comparison lets you detect drift in the **shape of a feature's distribution** between a detection window and a reference window, rather than comparing a single scalar metric such as the mean. +It is configured with the `compare_on_distribution` method, as an alternative to [`compare_on`](statistics_comparison.md#comparison-criteria). + +A single scalar metric can miss meaningful changes. +For example, the mean of a feature can stay constant while its variance grows or while a unimodal distribution becomes bimodal. +Distribution comparison captures these changes by computing a distance between the detection and reference distributions and comparing it against a threshold. + +!!! info "Reference window required" + Distribution comparison always compares two windows, so a reference window is mandatory. + Define it with `with_reference_window` or `with_reference_training_dataset` before calling `compare_on_distribution`. + +!!! tip "Reference training dataset" + On a Feature View, you can use a training dataset as the reference distribution with `with_reference_training_dataset(training_dataset_version=...)`. + This is the basis for [Model Monitoring](../../mlops/model_monitoring/index.md), where a model's production inference data is compared against the distribution of its training dataset. + +## Use cases + +Distribution comparison is most valuable when changes in your data are not captured by a single scalar metric, but by a change in the overall shape of the feature distribution. +It can be enabled on both Feature Groups and Feature Views, but for different purposes. + +For **Feature Groups**, distribution comparison helps you detect when newly ingested feature data drifts in shape from a baseline window of historical data, surfacing issues such as a new category becoming dominant or a numeric feature shifting from unimodal to multimodal, even when its mean stays stable. +See the [Feature Monitoring for Feature Groups](../feature_group/feature_monitoring.md) guide to configure it. + +For **Feature Views**, distribution comparison helps you quantify how much the distribution of newly inserted Feature Group data has drifted from your training dataset, and decide whether to retrain your ML models on a new training dataset version before the drift degrades model performance. +See the [Feature Monitoring for Feature Views](../feature_view/feature_monitoring.md) guide to configure it. + +## Distance metrics + +You select the distance metric with the `metric` parameter. +The following metrics are available: + +- **PSI** (Population Stability Index) — the default metric, widely used to monitor drift in production. + It is the only metric with a built-in default threshold of `0.2`. +- **KL_DIVERGENCE** — Kullback–Leibler divergence; asymmetric, sensitive to regions where the reference has low probability. +- **JS_DIVERGENCE** — Jensen–Shannon divergence; a symmetric, bounded smoothing of KL divergence. +- **HELLINGER** — Hellinger distance; symmetric and bounded in `[0, 1]`. +- **WASSERSTEIN** — Wasserstein (earth mover's) distance; numeric features only. +- **KOLMOGOROV_SMIRNOV** — Kolmogorov–Smirnov statistic; numeric features only. + +For every metric other than PSI, you must provide a `threshold` explicitly. + +!!! warning "Numeric-only metrics" + `WASSERSTEIN` and `KOLMOGOROV_SMIRNOV` require a numeric feature. + Applying them to a categorical feature raises an error. + +## Binning + +To compute a distance, Hopsworks first discretizes the feature values into a probability distribution over bins. +You control the binning with the following parameters: + +- `binning_strategy` — how to build the bins. + One of `EQUI_WIDTH` (equal-width bins), `EQUI_FREQUENCY` (equal-frequency / quantile bins), `CUSTOM_EDGES` (user-provided bin edges) or `CATEGORICAL` (one bin per category). + Defaults to `EQUI_FREQUENCY` for numeric features and `CATEGORICAL` otherwise. +- `bin_count` — the number of bins for numeric strategies. + Defaults to `10`. +- `custom_bin_edges` — the list of bin edges, required when `binning_strategy` is `CUSTOM_EDGES`. +- `smoothing_epsilon` — a small additive constant applied to bins to avoid `log(0)` in log-based metrics such as PSI and KL divergence. + Defaults to `1e-6`. + +!!! info "Next steps" + Distribution comparison results integrate with the same [alerting](index.md#alerting) and [interactive graph](interactive_graph.md) tooling as scalar statistics comparison. diff --git a/docs/user_guides/fs/feature_monitoring/feature_monitoring_advanced.md b/docs/user_guides/fs/feature_monitoring/feature_monitoring_advanced.md deleted file mode 100644 index c29d2be7a1..0000000000 --- a/docs/user_guides/fs/feature_monitoring/feature_monitoring_advanced.md +++ /dev/null @@ -1,125 +0,0 @@ -# Advanced guide - -An introduction to Feature Monitoring can be found in the guides for [Feature Groups](../feature_group/feature_monitoring.md) and [Feature Views](../feature_view/feature_monitoring.md). -In addition, you can get started quickly by running our [tutorial for feature monitoring](https://github.com/logicalclocks/hopsworks-tutorials/blob/master/api_examples/feature_monitoring.ipynb). - -## Retrieve feature monitoring configurations - -### Retrieve feature monitoring configurations from UI - -An overview of all feature monitoring configurations is listed in the ^^Feature Monitoring^^ section in the Feature Group and Feature View overview page. - -### Retrieve feature monitoring configurations from Python - -You can retrieve one or more feature monitoring configurations from the Feature Group and Feature View Python objects and filter them by name, configuration id or feature name. - -=== "Python" - - ```python - # retrieve all configurations - fm_configs = trans_fg.get_feature_monitoring_configs() # from a feature group - fm_configs = trans_fv.get_feature_monitoring_configs() # or a feature view - - # retrieve a configuration by name - fm_config = trans_fg.get_feature_monitoring_configs( - name="trans_fg_all_features_monitoring", - ) - - # or by config id - fm_config = trans_fg.get_feature_monitoring_configs( - config_id=1, - ) - - # or for a specific feature - fm_configs = trans_fg.get_feature_monitoring_configs( - feature_name="amount", - ) - ``` - -## Disable feature monitoring - -You can enable or disable feature monitoring while keeping the historical statistics and comparison results. - -### Disable feature monitoring from UI - -In the overview page for feature monitoring, you can enable or disable a specific configuration by clicking on the ^^Disable^^ button. - -![Disable button in a feature monitoring configuration](../../../assets/images/guides/fs/feature_monitoring/fm-config-disable-arrow.png) - -### Disable feature monitoring from Python - -You can easily enable or disable a specific feature monitoring configuration using the Python object. - -=== "Python" - - ```python - # disable a specific feature monitoring configuration - fm_config.disable() - - # disable a specific feature monitoring configuration - fm_config.enable() - ``` - -## Run the statistics comparison manually - -You can trigger the feature monitoring job on demand, to compute and compare statistics on the detection and reference windows according to the feature monitoring configuration. - -### Run the statistics comparison manually from UI - -In the overview page for feature monitoring, you can trigger the computation and comparison of statistics for a specific configuration by clicking on the ^^Run once^^ button. - -!!! note - Triggering the feature monitoring job manually does **not** affect the underlying schedule. - -![Run once button in a feature monitoring configuration](../../../assets/images/guides/fs/feature_monitoring/fm-config-run-once-arrow.png) - -### Run the statistics comparison manually from Python - -To trigger the feature monitoring job once from the Python API, use the feature monitoring Python object as shown in the example below. - -=== "Python" - - ```python - # run the feature monitoring job once - fm_config.run_job() - ``` - -## Get feature monitoring results - -### Get feature monitoring results from UI - -The easiest way to explore the statistics and comparison results is using the Hopsworks ==interactive graph== for Feature Monitoring. -See more information on the [Interactive graph guide](interactive_graph.md). - -![Visualize statistics on a time series](../../../assets/images/guides/fs/feature_monitoring/fm-reference-plot.png) - -### Get feature monitoring results from Python - -Alternatively, you can retrieve all the statistics and comparison results using the feature monitoring configuration Python object as shown in the example below. - -=== "Python" - - ```python - # retrieve all feature monitoring results from a specific config - fm_results = fm_config.get_history() - - # or filter results by date - fm_results = fm_config.get_history( - start_time="2023-01-01", - end_time="2023-01-31", - ) - ``` - -## Delete a feature monitoring configuration - -Deleting a feature monitoring configuration also deletes the historical statistics and comparison results attached to this configuration. - -### Delete a feature monitoring configuration from Python - -You can delete feature monitoring configurations using the Python API only, as shown in the example below. - -=== "Python" - - ```python - fm_config.delete() - ``` diff --git a/docs/user_guides/fs/feature_monitoring/index.md b/docs/user_guides/fs/feature_monitoring/index.md index 49757e5184..55268cb6fa 100644 --- a/docs/user_guides/fs/feature_monitoring/index.md +++ b/docs/user_guides/fs/feature_monitoring/index.md @@ -3,29 +3,25 @@ ## Introduction Feature Monitoring complements the Hopsworks data validation capabilities by allowing you to monitor your data once they have been ingested into the Feature Store. -Hopsworks feature monitoring user interface is centered around two functionalities: +Hopsworks feature monitoring user interface is centered around three functionalities: - **Scheduled Statistics**: The user defines a _detection window_ over its data for which Hopsworks will compute the statistics on a regular basis. The results are stored in Hopsworks and enable the user to visualise the temporal evolution of statistical metrics on its data. This can be enabled for a whole Feature Group or Feature View, or for a particular Feature. For more details, see the [Scheduled statistics guide](scheduled_statistics.md). -- **Statistics Comparison**: Enabled only for individual features, this variant allows the user to schedule the statistics computation on both a _detection_ and a _reference window_. +- **Statistics Comparison**: This variant allows the user to schedule the statistics computation on both a _detection_ and a _reference window_, and compare them on a selected feature using a single scalar metric (e.g., the mean). By providing information about how to compare those statistics, you can setup alerts to quickly detect critical change in the data. For more details, see the [Statistics comparison guide](statistics_comparison.md). -!!! important - To enable feature monitoring in Hopsworks, you need to set the `enable_feature_monitoring` [configuration option](../../../setup_installation/admin/variables.md) to `true`. - This can also be achieved in the cluster definition by setting the following attribute: +- **Data Distribution Comparison**: Instead of a single scalar metric, this variant compares the whole distribution of a feature between the _detection_ and _reference windows_ using distance metrics such as PSI or KL divergence. + This helps detect changes in the shape of the data that a single metric might miss. + For more details, see the [Distribution comparison guide](distribution_comparison.md). - ```yaml - hopsworks: - enable_feature_monitoring: "true" - ``` +## Define windows over feature data -## Statistics computation on windows of feature data - -Statistics are computed on feature data defined by windows. +Windows define the boundaries of the feature data on which Hopsworks operates. +Both statistics and data distributions are computed over the feature data delimited by a window. There are different types of windows depending on how they evolve over time. A window can have either a _fixed_ length (e.g., static window) or _variable_ length (e.g., expanding window). Moreover, windows can stick to a _specific point in time_ (e.g., static window) or _move_ over time (e.g., sliding or rolling window). @@ -40,9 +36,9 @@ Different types of windows allows for different use cases depending on whether y See more details about _detection_ and _reference_ windows in the [Detection windows](./scheduled_statistics.md#detection-windows) and [Reference windows](./statistics_comparison.md#reference-windows) guides. -## Visualize statistics on a time series +## Visualize metrics on a time series -Hopsworks provides an interactive graph to make the exploration of statistics and statistics comparison results more efficient and help you find unexpected trends or anomalous values faster. +Hopsworks provides an interactive graph to make the exploration of statistics and metrics (e.g., distribution-based distances) more efficient and help you find unexpected trends or anomalous values faster. See the [Interactive graph guide](interactive_graph.md) for more information. ![Feature monitoring graph](../../../assets/images/guides/fs/feature_monitoring/fm-show-shifted-points-arrow.png) diff --git a/docs/user_guides/fs/feature_monitoring/scheduled_statistics.md b/docs/user_guides/fs/feature_monitoring/scheduled_statistics.md index b739be5d5a..afbadc3e0e 100644 --- a/docs/user_guides/fs/feature_monitoring/scheduled_statistics.md +++ b/docs/user_guides/fs/feature_monitoring/scheduled_statistics.md @@ -15,8 +15,10 @@ Scheduled statistics monitoring is a powerful tool that allows you to monitor yo It can be enabled in both Feature Groups and Feature Views, but for different purposes. For **Feature Groups**, scheduled statistics enables you to analyze how your Feature Group data evolve over time, and leverage your intuition to identify trends or detect noisy values in the inserted feature data. +See the [Feature Monitoring for Feature Groups](../feature_group/feature_monitoring.md) guide to configure it. For **Feature Views**, scheduled statistics enables you to analyze the statistical properties of potentially new training dataset versions without having to actually create new training datasets and, thus, helping you decide when your training data show sufficient significant changes to create a new version. +See the [Feature Monitoring for Feature Views](../feature_view/feature_monitoring.md) guide to configure it. ## Detection windows @@ -25,7 +27,7 @@ Detection windows can be defined on the whole feature data or a subset of featur ![Types of detection windows](../../../assets/images/guides/fs/feature_monitoring/fm-detection-windows.png) -In [a previous section](index.md#statistics-computation-on-windows-of-feature-data) we described different types of windows available. +In [a previous section](index.md#define-windows-over-feature-data) we described different types of windows available. Taking a Feature Group as an example, the figure above describes how these windows are applied to Feature Group data, resulting in three different applications: - A _expanding window_ covering the whole Feature Group data from its creation until the time when statistics are computing. diff --git a/docs/user_guides/fs/feature_monitoring/statistics_comparison.md b/docs/user_guides/fs/feature_monitoring/statistics_comparison.md index dcdc56a8e3..1ab0232db6 100644 --- a/docs/user_guides/fs/feature_monitoring/statistics_comparison.md +++ b/docs/user_guides/fs/feature_monitoring/statistics_comparison.md @@ -16,8 +16,10 @@ Feature monitoring is a powerful tool that allows you to monitor your data over It can be enabled in both Feature Groups and Feature Views, but for different purposes. For **Feature Groups**, feature monitoring helps you rapidly identify unexpected trends or anomalous values in your Feature Group data, facilitating the debugging of possible root causes such as newly introduced changes in your feature pipelines. +See the [Feature Monitoring for Feature Groups](../feature_group/feature_monitoring.md) guide to configure it. For **Feature Views**, feature monitoring helps you quickly detect when newly inserted Feature Group data differs statistically from your existing training datasets, and decide whether to retrain your ML models using a new training dataset version or analyze possible issues in your feature pipelines or inference pipelines. +See the [Feature Monitoring for Feature Views](../feature_view/feature_monitoring.md) guide to configure it. ## Reference windows @@ -26,7 +28,7 @@ Reference windows can be defined in different ways depending on whether you are ![Types of reference windows](../../../assets/images/guides/fs/feature_monitoring/fm-reference-windows.png) -In [a previous section](index.md#statistics-computation-on-windows-of-feature-data) we described different types of windows available. +In [a previous section](index.md#define-windows-over-feature-data) we described different types of windows available. Taking a Feature View as an example, the figure above describes how these windows are applied to Feature Group data read by a Feature View query and Training data, resulting in the following applications: - A _expanding window_ covering the whole Feature Group data from its creation until the time when statistics are computing. @@ -44,6 +46,11 @@ See more details on how to define a reference window for your Feature Groups and ## Comparison criteria After defining the detection and reference windows, you can specify the criteria under which computed statistics will be compared. +The criteria described below apply to the comparison of a single scalar metric using the `compare_on` method. + +!!! tip "Distribution comparison" + Alternatively, you can compare the whole distribution of a feature between the detection and reference windows using metrics such as PSI or KL divergence. + See the [Distribution comparison guide](distribution_comparison.md) for details. ??? no-icon "Statistics metric" diff --git a/docs/user_guides/fs/feature_view/feature_monitoring.md b/docs/user_guides/fs/feature_view/feature_monitoring.md index 1568633466..3e710e38bd 100644 --- a/docs/user_guides/fs/feature_view/feature_monitoring.md +++ b/docs/user_guides/fs/feature_view/feature_monitoring.md @@ -7,36 +7,25 @@ Before continuing with this guide, see the [Feature monitoring guide](../feature !!! warning "Limited UI support" Currently, feature monitoring can only be configured using the [Hopsworks Python library](https://pypi.org/project/hopsworks). - However, you can enable/disable a feature monitoring configuration or trigger the statistics comparison manually from the UI, as shown in the [Advanced guide](../feature_monitoring/feature_monitoring_advanced.md). + However, you can enable/disable a feature monitoring configuration or trigger the statistics comparison manually from the UI. ## Code -In this section, we show you how to setup feature monitoring in a Feature View using the ==Hopsworks Python library==. +In this section, we show you how to set up feature monitoring on a Feature View using the ==Hopsworks Python library==. Alternatively, you can get started quickly by running our [tutorial for feature monitoring](https://github.com/logicalclocks/hopsworks-tutorials/blob/master/api_examples/feature_monitoring.ipynb). -First, checkout the pre-requisite and Hopsworks setup to follow the guide below. -Create a project, install the [Hopsworks Python library](https://pypi.org/project/hopsworks) in your environment and connect via the generated API key. -The second step is to start a new configuration for feature monitoring. +!!! info "Prerequisites" + - A Hopsworks project. + If you don't have one yet, go to [app.hopsworks.ai](https://app.hopsworks.ai), sign up with your email and create your first project. + - An API key, which you can get from "Account Settings" on [app.hopsworks.ai](https://app.hopsworks.ai). + - The [Hopsworks Python library](https://pypi.org/project/hopsworks) installed in your client. + See the [installation guide](../../client_installation/index.md). + - A Feature View and a Training Dataset. -After that, you can optionally define a detection window of data to compute statistics on, or use the default detection window (i.e., whole feature data). -If you want to setup scheduled statistics alone, you can jump to the last step to save your configuration. -Otherwise, the third and fourth steps are also optional and show you how to setup the comparison of statistics on a schedule by defining a reference window and specifying the statistics metric to be compared. +### Step 1: Connect to Hopsworks -### Step 1: Pre-requisite - -In order to setup feature monitoring for a Feature View, you will need: - -- A Hopsworks project. - If you don't have a project yet you can go to [app.hopsworks.ai](https://app.hopsworks.ai), signup with your email and create your first project. -- An API key, you can get one by going to "Account Settings" on [app.hopsworks.ai](https://app.hopsworks.ai). -- The [Hopsworks Python library](https://pypi.org/project/hopsworks) installed in your client. - See the [installation guide](../../client_installation/index.md). -- A Feature View -- A Training Dataset - -#### Connect your notebook to Hopsworks - -Connect the client running your notebooks to Hopsworks. +Connect the client running your notebook to Hopsworks. +You will be prompted to paste your API key to connect the notebook to your project. === "Python" @@ -49,16 +38,12 @@ Connect the client running your notebooks to Hopsworks. fs = project.get_feature_store() ``` -You will be prompted to paste your API key to connect the notebook to your project. -The `fs` Feature Store entity is now ready to be used to insert or read data from Hopsworks. +See the API reference for [`hopsworks.login`][hopsworks.login] and [`Project.get_feature_store`][hopsworks_common.project.Project.get_feature_store]. -#### Get or create a Feature View +### Step 2: Get or create a Feature View Feature monitoring can be enabled on already created Feature Views. -We suggest you read the [Feature View](../../../concepts/fs/feature_view/fv_overview.md) concept page to understand what a feature view is. -We also suggest you familiarize with the APIs to [create a feature view](overview.md) and how to create them using the [query abstraction](query.md). - -The following is a code example for getting or creating a Feature View with name `trans_fv` for transaction data. +We suggest you read the [Feature View](../../../concepts/fs/feature_view/fv_overview.md) concept page and familiarize yourself with the APIs to [create a feature view](overview.md) using the [query abstraction](query.md). === "Python" @@ -76,9 +61,11 @@ The following is a code example for getting or creating a Feature View with name ) ``` -#### Get or create a Training Dataset +See the API reference for [`FeatureStore.get_feature_view`][hsfs.feature_store.FeatureStore.get_feature_view] and [`FeatureStore.create_feature_view`][hsfs.feature_store.FeatureStore.create_feature_view]. -The following is a code example for creating a training dataset with two splits using a previously created feature view. +### Step 3: Get or create a Training Dataset + +A Training Dataset can be used later as a reference window to compare against (see Step 6). === "Python" @@ -92,73 +79,46 @@ The following is a code example for creating a training dataset with two splits ) ``` -### Step 2: Initialize configuration +See the API reference for [`FeatureView.create_train_validation_test_split`][hsfs.feature_view.FeatureView.create_train_validation_test_split]. -#### Scheduled statistics +### Step 4: Create a monitoring configuration -You can setup statistics monitoring on a ==single feature or multiple features== of your Feature Group data, included in your Feature View query. +Start a new configuration on the Feature View. +Use `create_scheduled_statistics` to only compute statistics on a schedule, or `create_feature_monitoring` to also compare them against a reference. -=== "Python" +=== "Scheduled statistics" ```python - # compute statistics for all the features - fg_monitoring_config = trans_fv.create_statistics_monitoring( + # compute statistics on one or more features on a schedule + fm_monitoring_config = trans_fv.create_scheduled_statistics( name="trans_fv_all_features_monitoring", - description="Compute statistics on all data of all features of the Feature Group data on a daily basis", - ) - - # or for a single feature - fg_monitoring_config = trans_fv.create_statistics_monitoring( - name="trans_fv_amount_monitoring", - description="Compute statistics on all data of a single feature of the Feature Group data on a daily basis", - feature_name="amount", + description="Compute statistics on the Feature View data on a daily basis", + feature_names=["amount"], # omit to monitor all features ) ``` -#### Statistics comparison - -When enabling the comparison of statistics in a feature monitoring configuration, you need to specify a ==single feature== of your Feature Group data, included in your Feature View query. -You can create multiple feature monitoring configurations on the same Feature View, but each of them should point to a single feature in the Feature View query. - -=== "Python" +=== "Statistics comparison" ```python - fg_monitoring_config = trans_fv.create_feature_monitoring( + # the feature to compare is selected later in + # compare_on / compare_on_distribution (Step 7.A / 7.B), not here + fm_monitoring_config = trans_fv.create_feature_monitoring( name="trans_fv_amount_monitoring", - feature_name="amount", - description="Compute descriptive statistics on the amount Feature of the Feature Group data on a daily basis", + description="Compute and compare descriptive statistics on the Feature View data on a daily basis", ) ``` -#### Custom schedule or percentage of window data - -By default, the computation of statistics is scheduled to run endlessly, every day at 12PM. -You can modify the default schedule by adjusting the `cron_expression`, `start_date_time` and `end_date_time` parameters. +See the API reference for [`FeatureView.create_scheduled_statistics`][hsfs.feature_view.FeatureView.create_scheduled_statistics] and [`FeatureView.create_feature_monitoring`][hsfs.feature_view.FeatureView.create_feature_monitoring]. -=== "Python" - - ```python - fg_monitoring_config = trans_fv.create_statistics_monitoring( - name="trans_fv_all_features_monitoring", - description="Compute statistics on all data of all features of the Feature Group data on a weekly basis", - cron_expression="0 0 12 ? *MON*", # weekly - row_percentage=0.8, # use 80% of the data - ) - - # or - fg_monitoring_config = trans_fv.create_feature_monitoring( - name="trans_fv_amount_monitoring", - feature_name="amount", - description="Compute descriptive statistics on the amount Feature of the Feature Group data on a weekly basis", - cron_expression="0 0 12 ? * MON *", # weekly - row_percentage=0.8, # use 80% of the data - ) - ``` +!!! info "Custom schedule" + By default, the computation of statistics is scheduled to run endlessly, every day at 12PM. + You can modify the default schedule by adjusting the `cron_expression`, `start_date_time` and `end_date_time` parameters (e.g., `cron_expression="0 0 12 ? * MON *"` for a weekly run). + To compute statistics on only a subset of the feature data, use the `row_percentage` parameter of `with_detection_window` (see Step 5). -### Step 3: (Optional) Define a detection window +### Step 5: (Optional) Define a detection window -By default, the detection window is an *expanding window* covering the whole Feature Group data. -You can define a different detection window using the `window_length` and `time_offset` parameters provided in the `with_detection_window` method. +By default, the detection window is an _expanding window_ covering the whole Feature Group data. +You can define a different detection window using the `window_length` and `time_offset` parameters of the `with_detection_window` method. Additionally, you can specify the percentage of feature data on which statistics will be computed using the `row_percentage` parameter. === "Python" @@ -171,9 +131,14 @@ Additionally, you can specify the percentage of feature data on which statistics ) ``` -### Step 4: (Optional) Define a reference window +See the API reference for [`FeatureMonitoringConfig.with_detection_window`][hsfs.core.feature_monitoring_config.FeatureMonitoringConfig.with_detection_window]. + +### Step 6: (Optional) Define a reference window + +When setting up feature monitoring for a Feature View, the reference can be either a reference window of feature data or a training dataset. -When setting up feature monitoring for a Feature View, reference windows can be either a regular window, a specific value (i.e., window of size 1) or a training dataset. +!!! tip "Basis for Model Monitoring" + Using a training dataset as the reference is the basis for [Model Monitoring](../../mlops/model_monitoring/index.md), where a model's production inference data is compared against the distribution of its training dataset. === "Python" @@ -185,27 +150,29 @@ When setting up feature monitoring for a Feature View, reference windows can be row_percentage=0.8, # use 80% of the data ) - # or a specific value - fm_monitoring_config.with_reference_value( - value=100, - ) - # or a training dataset fm_monitoring_config.with_reference_training_dataset( training_dataset_version=1, # use the training dataset used to train your production model ) ``` -### Step 5: (Optional) Define the statistics comparison criteria +See the API reference for [`FeatureMonitoringConfig.with_reference_window`][hsfs.core.feature_monitoring_config.FeatureMonitoringConfig.with_reference_window] and [`FeatureMonitoringConfig.with_reference_training_dataset`][hsfs.core.feature_monitoring_config.FeatureMonitoringConfig.with_reference_training_dataset]. + +!!! info "Comparing against a specific value" + Instead of a reference window or training dataset, you can compare the detection statistics against a fixed reference value (i.e., a window of size 1). + In that case, skip this step and pass the `specific_value` parameter to `compare_on` in Step 7. + +### Step 7.A: (Optional) Compare on a scalar metric In order to compare detection and reference statistics, you need to provide the criteria for such comparison. -First, you select the metric to consider in the comparison using the `metric` parameter. +First, you select the feature and the metric to consider in the comparison using the `feature_name` and `metric` parameters. Then, you can define a relative or absolute threshold using the `threshold` and `relative` parameters. === "Python" ```python fm_monitoring_config.compare_on( + feature_name="amount", # the feature to compare metric="mean", threshold=0.2, # a relative change over 20% is considered anomalous relative=True, # relative or absolute change @@ -213,10 +180,33 @@ Then, you can define a relative or absolute threshold using the `threshold` and ) ``` +See the API reference for [`FeatureMonitoringConfig.compare_on`][hsfs.core.feature_monitoring_config.FeatureMonitoringConfig.compare_on]. + !!! info "Difference values and thresholds" For more information about the computation of difference values and the comparison against threshold bounds see the [Comparison criteria section](../feature_monitoring/statistics_comparison.md#comparison-criteria) in the Statistics comparison guide. -### Step 6: Save configuration +### Step 7.B: (Optional) Compare on the whole distribution + +Alternatively, instead of a single scalar metric, you can detect drift in the shape of a feature's distribution using `compare_on_distribution`. +Select a distribution distance metric (e.g., `PSI`) and a threshold. +A reference window or training dataset (Step 6) is required for distribution comparison. + +=== "Python" + + ```python + fm_monitoring_config.compare_on_distribution( + feature_name="amount", # the feature to compare + metric="PSI", + threshold=0.2, # a distance above 0.2 is considered a significant shift + ) + ``` + +See the API reference for [`FeatureMonitoringConfig.compare_on_distribution`][hsfs.core.feature_monitoring_config.FeatureMonitoringConfig.compare_on_distribution]. + +!!! tip "More distribution options" + See the [Distribution comparison guide](../feature_monitoring/distribution_comparison.md) for the full list of metrics and binning strategies. + +### Step 8: Save the configuration Finally, you can save your feature monitoring configuration by calling the `save` method. Once the configuration is saved, the schedule for the statistics computation and comparison will be activated automatically. @@ -227,5 +217,62 @@ Once the configuration is saved, the schedule for the statistics computation and fm_monitoring_config.save() ``` -!!! info "Next steps" - See the [Advanced guide](../feature_monitoring/feature_monitoring_advanced.md) to learn how to delete, disable or trigger feature monitoring manually. +See the API reference for [`FeatureMonitoringConfig.save`][hsfs.core.feature_monitoring_config.FeatureMonitoringConfig.save]. + +### Step 9: Retrieve configurations and history + +Once saved, you can retrieve your feature monitoring configurations and the results of past executions directly from the Feature View. + +=== "Python" + + ```python + # fetch all configurations attached to the feature view + configs = trans_fv.get_feature_monitoring_configs() + + # or a single configuration by name + config = trans_fv.get_feature_monitoring_configs(name="trans_fv_amount_monitoring") + + # fetch the history of monitoring results (with computed statistics) + history = trans_fv.get_feature_monitoring_history( + config_name="trans_fv_amount_monitoring", + with_statistics=True, + ) + ``` + +See the API reference for [`FeatureView.get_feature_monitoring_configs`][hsfs.feature_view.FeatureView.get_feature_monitoring_configs] and [`FeatureView.get_feature_monitoring_history`][hsfs.feature_view.FeatureView.get_feature_monitoring_history]. + +### API Reference + +[`FeatureView`][hsfs.feature_view.FeatureView] + +## Monitor a model in production + +A Feature View can also monitor the inference data of a model served in production, comparing it against the training dataset the model was trained on. +This is the feature-view entry point to [Model Monitoring](../../mlops/model_monitoring/index.md). + +It targets the feature view's logging feature group, so feature logging must be enabled with `feature_view.enable_logging()`, and filters the detection window by the given model name and version. +The reference defaults to the training dataset version used to train the model. + +=== "Python" + + ```python + fm_monitoring_config = trans_fv.create_model_monitoring( + name="trans_fv_model_monitoring", + model_name="my_model", + model_version=1, + ).with_detection_window( + time_offset="1d", + window_length="1d", + ).with_reference_training_dataset( + # omitted -> defaults to the model's training dataset version + ).compare_on_distribution( + feature_name="amount", + metric="PSI", + threshold=0.2, + ).save() + ``` + +See the API reference for [`FeatureView.create_model_monitoring`][hsfs.feature_view.FeatureView.create_model_monitoring]. + +!!! info "Explore the API" + The [`FeatureMonitoringConfig`][hsfs.core.feature_monitoring_config.FeatureMonitoringConfig] reference documents the full set of available methods, such as enabling or disabling a configuration, triggering it manually, or deleting it. diff --git a/docs/user_guides/mlops/model_monitoring/index.md b/docs/user_guides/mlops/model_monitoring/index.md new file mode 100644 index 0000000000..eb228d386b --- /dev/null +++ b/docs/user_guides/mlops/model_monitoring/index.md @@ -0,0 +1,31 @@ +# Model Monitoring + +Model monitoring lets you detect drift between the data a model was trained on and the data it serves in production. +It is built on top of [feature monitoring](../../fs/feature_monitoring/index.md): a model monitoring configuration computes statistics over the model's logged inference data (the _detection window_) and compares them against the training dataset the model was trained on (the _reference window_). + +You can configure model monitoring from a [model deployment](model_monitoring.md), a [model](model_monitoring.md), or a [feature view](model_monitoring.md) — all three resolve to the same underlying configuration on the feature view's logging feature group, filtered by the model name and version. + +## Prerequisites + +To enable model monitoring, you need: + +- A **feature view** with **feature logging enabled**. + Logging is what captures the inference data that monitoring analyzes. + If logging is not enabled, enable it with `feature_view.enable_logging()`. +- A **registered model** trained from that feature view, with a **recorded training dataset version**. + The training dataset version is recorded automatically when the model is created from a feature view, and is used as the default reference distribution. +- A **model deployment** that logs its inference data to the feature view via the feature view `log` methods (typically from its predictor script). + Feature logging is what produces the data that monitoring analyzes, so the deployment must write its inference data to the feature view's logging feature group. + This is currently only supported for model deployments with a predictor script. + See the [Feature Logging guide](../../fs/feature_view/feature_logging.md). + +## How it relates to feature monitoring + +Model monitoring is feature monitoring applied to a model's inference logs: + +- The **detection window** covers the data recently served by the model, read from the logging feature group and filtered by model name and version. +- The **reference window** defaults to the training dataset version used to train the model, so you compare production data against training data out of the box. +- The **comparison criteria** uses the same [scalar metric](../../fs/feature_monitoring/statistics_comparison.md) and [data distribution](../../fs/feature_monitoring/distribution_comparison.md) criteria as feature monitoring. + +!!! info "Next steps" + See the [Model Monitoring Creation guide](model_monitoring.md) for code examples from a model deployment, a model, and a feature view. diff --git a/docs/user_guides/mlops/model_monitoring/model_monitoring.md b/docs/user_guides/mlops/model_monitoring/model_monitoring.md new file mode 100644 index 0000000000..caa4318e60 --- /dev/null +++ b/docs/user_guides/mlops/model_monitoring/model_monitoring.md @@ -0,0 +1,193 @@ +# Model Monitoring Creation + +This guide shows how to configure monitoring for a model in production using the ==Hopsworks Python library==. +Make sure you have read the [Model Monitoring overview](index.md) first. + +!!! info "Prerequisites" + Make sure you meet the [prerequisites](index.md#prerequisites): a feature view with feature logging enabled, a registered model with a recorded training dataset version, and a model deployment with a predictor script that logs its inference data. + +!!! warning "Limited UI support" + Like feature monitoring, model monitoring can currently only be configured using the [Hopsworks Python library](https://pypi.org/project/hopsworks). + +## Code + +In this section, we show you how to set up model monitoring from a model or a model deployment using the ==Hopsworks Python library==. + +### Step 1: Connect to Hopsworks + +Connect the client running your notebook to Hopsworks and get the Model Registry and Model Serving handles. + +=== "Python" + + ```python + import hopsworks + + + project = hopsworks.login() + + mr = project.get_model_registry() + ms = project.get_model_serving() + ``` + +See the API reference for [`hopsworks.login`][hopsworks.login], [`Project.get_model_registry`][hopsworks_common.project.Project.get_model_registry] and [`Project.get_model_serving`][hopsworks_common.project.Project.get_model_serving]. + +### Step 2: Get a model or deployment + +Retrieve the model or the deployment whose inference data you want to monitor. + +=== "From a deployment" + + ```python + my_deployment = ms.get_deployment("my_deployment") + ``` + +=== "From a model" + + ```python + my_model = mr.get_model("my_model", version=1) + ``` + +See the API reference for [`ModelServing.get_deployment`][hsml.model_serving.ModelServing.get_deployment] and [`ModelRegistry.get_model`][hsml.model_registry.ModelRegistry.get_model]. + +### Step 3: Create a model monitoring configuration + +Start a new configuration from the deployment or the model. +Hopsworks resolves the model's parent feature view from its provenance and fills in the model name and version for you. + +=== "From a deployment" + + ```python + fm_monitoring_config = my_deployment.create_model_monitoring( + name="model_psi_monitoring", + ) + ``` + +=== "From a model" + + ```python + fm_monitoring_config = my_model.create_model_monitoring( + name="model_psi_monitoring", + ) + ``` + +See the API reference for [`Deployment.create_model_monitoring`][hsml.deployment.Deployment.create_model_monitoring] and [`Model.create_model_monitoring`][hsml.model.Model.create_model_monitoring]. + +!!! tip "Configuring from a feature view" + You can also configure model monitoring directly from the feature view backing the model, using `feature_view.create_model_monitoring`. + See the [Feature Monitoring guide for Feature Views](../../fs/feature_view/feature_monitoring.md#monitor-a-model-in-production). + +!!! info "Custom schedule" + By default, monitoring runs every day at 12PM. + You can modify the schedule by adjusting the `cron_expression`, `start_date_time` and `end_date_time` parameters of `create_model_monitoring` (UTC, Quartz specification). + +!!! warning "Sub-hourly schedules" + The inference-log feature group materializes its offline data at most once per hour. + A cron expression that fires more than once per hour produces redundant or incomplete detection windows and triggers a warning, so use a schedule that fires at most once per hour. + +### Step 4: Define a detection window + +The detection window covers the inference data recently served by the model. +Define it using the `window_length` and `time_offset` parameters of the `with_detection_window` method. + +=== "Python" + + ```python + fm_monitoring_config.with_detection_window( + time_offset="1d", # data served by this model in the last day + window_length="1d", + ) + ``` + +See the API reference for [`FeatureMonitoringConfig.with_detection_window`][hsfs.core.feature_monitoring_config.FeatureMonitoringConfig.with_detection_window]. + +### Step 5: Define the reference training dataset + +By default, the reference is the training dataset version used to train the model, recorded at model registration time. +You can call `with_reference_training_dataset()` without arguments to make this explicit. + +=== "Python" + + ```python + fm_monitoring_config.with_reference_training_dataset( + # omitted -> defaults to the model's training dataset version + ) + ``` + +See the API reference for [`FeatureMonitoringConfig.with_reference_training_dataset`][hsfs.core.feature_monitoring_config.FeatureMonitoringConfig.with_reference_training_dataset]. + +!!! info "Training dataset version validation" + If you pass a specific version, it must match the model's recorded training dataset version, otherwise the call raises. + This guards against accidentally comparing production data against a training dataset the model was never trained on. + +### Step 6.A: Compare on a scalar metric + +Select the feature and the metric to compare, and define a relative or absolute threshold. + +=== "Python" + + ```python + fm_monitoring_config.compare_on( + feature_name="amount", # the feature to compare + metric="mean", + threshold=0.2, # a relative change over 20% is considered anomalous + relative=True, # relative or absolute change + strict=False, # strict or relaxed comparison + ) + ``` + +See the API reference for [`FeatureMonitoringConfig.compare_on`][hsfs.core.feature_monitoring_config.FeatureMonitoringConfig.compare_on]. + +### Step 6.B: Compare on the whole distribution + +Alternatively, instead of a single scalar metric, you can detect drift in the shape of the feature's distribution using `compare_on_distribution`. +Select a distribution distance metric (e.g., `PSI`) and a threshold. + +=== "Python" + + ```python + fm_monitoring_config.compare_on_distribution( + feature_name="amount", # the feature to compare + metric="PSI", + threshold=0.2, # a distance above 0.2 is considered a significant shift + ) + ``` + +See the API reference for [`FeatureMonitoringConfig.compare_on_distribution`][hsfs.core.feature_monitoring_config.FeatureMonitoringConfig.compare_on_distribution]. + +!!! tip "More distribution options" + See the [Distribution comparison guide](../../fs/feature_monitoring/distribution_comparison.md) for the full list of metrics and binning strategies. + +### Step 7: Save the configuration + +Finally, save the configuration by calling the `save` method. +Once saved, the schedule for the statistics computation and comparison is activated automatically. + +=== "Python" + + ```python + fm_monitoring_config.save() + ``` + +See the API reference for [`FeatureMonitoringConfig.save`][hsfs.core.feature_monitoring_config.FeatureMonitoringConfig.save]. + +### Step 8: Retrieve configurations + +You can list the monitoring configurations attached to a model or deployment. + +=== "From a deployment" + + ```python + fm_configs = my_deployment.get_monitoring_configs() + ``` + +=== "From a model" + + ```python + fm_configs = my_model.get_monitoring_configs() + ``` + +See the API reference for [`Deployment.get_monitoring_configs`][hsml.deployment.Deployment.get_monitoring_configs] and [`Model.get_monitoring_configs`][hsml.model.Model.get_monitoring_configs]. + +!!! info "Next steps" + Model monitoring results integrate with the same [alerting](../../fs/feature_monitoring/index.md#alerting) and [interactive graph](../../fs/feature_monitoring/interactive_graph.md) tooling as feature monitoring. + See the [`FeatureMonitoringConfig`][hsfs.core.feature_monitoring_config.FeatureMonitoringConfig] reference to learn how to disable, manually trigger, or delete a configuration. diff --git a/docs/user_guides/mlops/serving/inference-logger.md b/docs/user_guides/mlops/serving/inference-logger.md index 09227b9c14..2f435120bb 100644 --- a/docs/user_guides/mlops/serving/inference-logger.md +++ b/docs/user_guides/mlops/serving/inference-logger.md @@ -8,6 +8,10 @@ Hopsworks supports logging both inference requests and predictions as events to !!! warning "Inference logging is not supported for vLLM deployments." +!!! info "Inference logger vs. feature logging" + The inference logger described here stores model inputs and predictions from inference requests and responses into a Kafka topic, for later consumption and analysis. + It is separate from [feature logging](../../fs/feature_view/feature_logging.md), which supports more fine-grained logging of inference logs and features and powers [feature monitoring](../../fs/feature_monitoring/index.md) and [model monitoring](../model_monitoring/index.md). + !!! info "Logging modes" Three logging modes are available: diff --git a/mkdocs.yml b/mkdocs.yml index cc3af76099..e954d5ca3d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -48,6 +48,7 @@ nav: - Model Training: concepts/mlops/training.md - Model Registry: concepts/mlops/registry.md - Model Serving: concepts/mlops/serving.md + - Model Monitoring: concepts/mlops/model_monitoring.md - Vector Database: concepts/mlops/opensearch.md - BI Tools: concepts/mlops/bi_tools.md - Data Transformations: concepts/mlops/data_transformations.md @@ -89,9 +90,7 @@ nav: - Getting started: user_guides/fs/feature_group/data_validation.md - Advanced guide: user_guides/fs/feature_group/data_validation_advanced.md - Best practices: user_guides/fs/feature_group/data_validation_best_practices.md - - Feature Monitoring: - - Getting started: user_guides/fs/feature_group/feature_monitoring.md - - Advanced guide: user_guides/fs/feature_monitoring/feature_monitoring_advanced.md + - Feature Monitoring: user_guides/fs/feature_group/feature_monitoring.md - Notification: user_guides/fs/feature_group/notification.md - On-Demand Transformations: user_guides/fs/feature_group/on_demand_transformations.md - Online Ingestion Observability: user_guides/fs/feature_group/online_ingestion_observability.md @@ -107,9 +106,7 @@ nav: - Helper Columns: user_guides/fs/feature_view/helper-columns.md - Model-Dependent Transformation Functions: user_guides/fs/feature_view/model-dependent-transformations.md - Spines: user_guides/fs/feature_view/spine-query.md - - Feature Monitoring: - - Getting started: user_guides/fs/feature_view/feature_monitoring.md - - Advanced guide: user_guides/fs/feature_monitoring/feature_monitoring_advanced.md + - Feature Monitoring: user_guides/fs/feature_view/feature_monitoring.md - Feature Logging: user_guides/fs/feature_view/feature_logging.md - Vector Similarity Search: user_guides/fs/vector_similarity_search.md - Transformation Functions: user_guides/fs/transformation_functions.md @@ -140,6 +137,7 @@ nav: - user_guides/fs/feature_monitoring/index.md - Scheduled Statistics: user_guides/fs/feature_monitoring/scheduled_statistics.md - Statistics Comparison: user_guides/fs/feature_monitoring/statistics_comparison.md + - Distribution Comparison: user_guides/fs/feature_monitoring/distribution_comparison.md - Interactive Graph: user_guides/fs/feature_monitoring/interactive_graph.md - Projects: - user_guides/projects/index.md @@ -239,6 +237,9 @@ nav: - REST API: user_guides/mlops/serving/rest-api.md - Troubleshooting: user_guides/mlops/serving/troubleshooting.md - External Access: user_guides/mlops/serving/external-access.md + - Model Monitoring: + - user_guides/mlops/model_monitoring/index.md + - Model Monitoring Creation: user_guides/mlops/model_monitoring/model_monitoring.md - Vector Database: user_guides/fs/vector_similarity_search.md - Provenance: user_guides/mlops/provenance/provenance.md - Agents: