skmultiflow.evaluation.
EvaluatePrequentialDelayed
The prequential evaluation delayed method.
The prequential evaluation delayed is designed specifically for stream settings, in the sense that each sample serves two purposes, and that samples are analysed sequentially, in order of arrival, and are used to update the model only when their label are available, given their timestamps (arrival and available times).
This method consists of using each sample to test the model, which means to make a predictions, and then the same sample is used to train the model (partial fit) after its label is available after a certain delay. This way the model is always tested on samples that it hasn’t seen yet and updated on samples that have their labels available.
The number of samples to process between each test. Also defines when to update the plot if show_plot=True. Note that setting n_wait too small can significantly slow the evaluation process.
show_plot=True
n_wait
The maximum number of samples to process during the evaluation.
The number of samples to pass at a time to the model(s).
The number of samples to use to train the model before starting the evaluation. Used to enforce a ‘warm’ start.
The maximum duration of the simulation (in seconds).
File name to save the summary of the evaluation.
If True, a plot will show the progress of the evaluation. Warning: Plotting can slow down the evaluation process.
If True, the stream is restarted once the evaluation is complete.
If True, the visualization used is a cloud of data points (only works for classification) and default performance metrics are ignored. If specific metrics are required, then they must be explicitly set using the metrics attribute.
metrics
Notes
This evaluator can process a single learner to track its performance; or multiple learners at a time, to compare different models on the same stream.
The metric ‘true_vs_predicted’ is intended to be informative only. It corresponds to evaluations at a specific moment which might not represent the actual learner performance across all instances.
The metrics running_time and model_size ` are not plotted when the `show_plot option is set. Only their current value is displayed at the bottom of the figure. However, their values over the evaluation are written into the resulting csv file if the output_file option is set.
Examples
>>> # The first example demonstrates how to evaluate one model >>> import numpy as np >>> import pandas as pd >>> from skmultiflow.data import TemporalDataStream >>> from skmultiflow.trees import HoeffdingTreeClassifier >>> from skmultiflow.evaluation import EvaluatePrequentialDelayed >>> >>> # Columns used to get the data, label and time from iris_timestamp dataset >>> DATA_COLUMNS = ["sepal_length", "sepal_width", "petal_length", "petal_width"] >>> LABEL_COLUMN = "label" >>> TIME_COLUMN = "timestamp" >>> >>> # Read a csv with stream data >>> data = pd.read_csv("https://raw.githubusercontent.com/scikit-multiflow/streaming-datasets/" >>> "master/iris_timestamp.csv") >>> # Convert time column to datetime >>> data[TIME_COLUMN] = pd.to_datetime(data[TIME_COLUMN]) >>> # Sort data by time >>> data = data.sort_values(by=TIME_COLUMN) >>> # Get X, y and time >>> X = data[DATA_COLUMNS].values >>> y = data[LABEL_COLUMN].values >>> time = data[TIME_COLUMN].values >>> >>> >>> # Set a delay of 1 day >>> delay_time = np.timedelta64(1, "D") >>> # Set the stream >>> stream = TemporalDataStream(X, y, time, sample_delay=delay_time, ordered=False) >>> >>> # Set the model >>> ht = HoeffdingTreeClassifier() >>> >>> # Set the evaluator >>> >>> evaluator = EvaluatePrequentialDelayed(batch_size=1, >>> pretrain_size=X.shape[0]//2, >>> max_samples=X.shape[0], >>> output_file='results_delay.csv', >>> metrics=['accuracy', 'recall', 'precision', 'f1', 'kappa']) >>> >>> # Run evaluation >>> evaluator.evaluate(stream=stream, model=ht, model_names=['HT'])
>>> # The second example demonstrates how to compare two models >>> import numpy as np >>> import pandas as pd >>> from skmultiflow.data import TemporalDataStream >>> from skmultiflow.trees import HoeffdingTreeClassifier >>> from skmultiflow.bayes import NaiveBayes >>> from skmultiflow.evaluation import EvaluatePrequentialDelayed >>> >>> # Columns used to get the data, label and time from iris_timestamp dataset >>> DATA_COLUMNS = ["sepal_length", "sepal_width", "petal_length", "petal_width"] >>> LABEL_COLUMN = "label" >>> TIME_COLUMN = "timestamp" >>> >>> # Read a csv with stream data >>> data = pd.read_csv("../data/datasets/iris_timestamp.csv") >>> # Convert time column to datetime >>> data[TIME_COLUMN] = pd.to_datetime(data[TIME_COLUMN]) >>> # Sort data by time >>> data = data.sort_values(by=TIME_COLUMN) >>> # Get X, y and time >>> X = data[DATA_COLUMNS].values >>> y = data[LABEL_COLUMN].values >>> time = data[TIME_COLUMN].values >>> >>> >>> # Set a delay of 30 minutes >>> delay_time = np.timedelta64(30, "m") >>> # Set the stream >>> stream = TemporalDataStream(X, y, time, sample_delay=delay_time, ordered=False) >>> >>> # Set the models >>> ht = HoeffdingTreeClassifier() >>> nb = NaiveBayes() >>> >>> evaluator = EvaluatePrequentialDelayed(batch_size=1, >>> pretrain_size=X.shape[0]//2, >>> max_samples=X.shape[0], >>> output_file='results_delay.csv', >>> metrics=['accuracy', 'recall', 'precision', 'f1', 'kappa']) >>> >>> # Run evaluation >>> evaluator.evaluate(stream=stream, model=[ht, nb], model_names=['HT', 'NB'])
Methods
evaluate(self, stream, model[, model_names])
evaluate
Evaluates a model or set of models on samples from a stream.
evaluation_summary(self)
evaluation_summary
get_current_measurements(self[, model_idx])
get_current_measurements
Get current measurements from the evaluation (measured on last n_wait samples).
get_info(self)
get_info
Collects and returns the information about the configuration of the estimator
get_mean_measurements(self[, model_idx])
get_mean_measurements
Get mean measurements from the evaluation.
get_measurements(self[, model_idx])
get_measurements
Get measurements from the evaluation.
get_params(self[, deep])
get_params
Get parameters for this estimator.
partial_fit(self, X, y[, classes, sample_weight])
partial_fit
Partially fit all the models on the given data.
predict(self, X)
predict
Predicts with the estimator(s) being evaluated.
reset(self)
reset
Resets the estimator to its initial state.
set_params(self, **params)
set_params
Set the parameters of this estimator.
update_progress_bar(curr, total, steps, time)
update_progress_bar
The stream from which to draw the samples.
The model or list of models to evaluate.
A list with the names of the models.
The trained model(s).
Indicates the index of the model as defined in evaluate(model). If None, returns a list with the measurements for each model.
for each model.
Configuration of the estimator.
is a a list with the measurements for each model.
If True, will return the parameters for this estimator and contained subobjects that are estimators.
Parameter names mapped to their values.
The data upon which the estimators will be trained.
The classification labels / target values for all samples in X.
Stores all the classes that may be encountered during the classification task. Not used for regressors.
Samples weight. If not provided, uniform weights are assumed.
self
All the samples we want to predict the label for.
Model(s) predictions
The method works on simple estimators as well as on nested objects (such as pipelines). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.
<component>__<parameter>