org.apache.spark.ml.classification

Class Classifier<FeaturesType,E extends Classifier<FeaturesType,E,M>,M extends ClassificationModel<FeaturesType,M>>

java.lang.Object

org.apache.spark.ml.PipelineStage

org.apache.spark.ml.Estimator<M>

org.apache.spark.ml.Predictor<FeaturesType,E,M>

org.apache.spark.ml.classification.Classifier<FeaturesType,E,M>

All Implemented Interfaces:

java.io.Serializable, Params, Identifiable

Direct Known Subclasses:

ProbabilisticClassifier

public abstract class Classifier<FeaturesType,E extends Classifier<FeaturesType,E,M>,M extends ClassificationModel<FeaturesType,M>>
extends Predictor<FeaturesType,E,M>

:: DeveloperApi ::

Single-label binary or multiclass classification. Classes are indexed {0, 1, ..., numClasses - 1}.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
Classifier()

Method Summary

Methods

Modifier and Type	Method and Description
protected RDD<LabeledPoint>	extractLabeledPoints(Dataset<?> dataset, int numClasses) Extract labelCol and featuresCol from the given dataset, and put it in an RDD with strong types.
Param<java.lang.String>	featuresCol() Param for features column name.
java.lang.String	getFeaturesCol()
java.lang.String	getLabelCol()
protected int	getNumClasses(Dataset<?> dataset, int maxNumClasses) Get the number of classes.
java.lang.String	getPredictionCol()
java.lang.String	getRawPredictionCol()
Param<java.lang.String>	labelCol() Param for label column name.
Param<java.lang.String>	predictionCol() Param for prediction column name.
Param<java.lang.String>	rawPredictionCol() Param for raw prediction (a.k.a.
E	setRawPredictionCol(java.lang.String value)
StructType	validateAndTransformSchema(StructType schema, boolean fitting, DataType featuresDataType)
StructType	validateAndTransformSchema(StructType schema, boolean fitting, DataType featuresDataType) Validates and transforms the input schema with the provided param map.

Methods inherited from class org.apache.spark.ml.Predictor

copy, extractLabeledPoints, fit, setFeaturesCol, setLabelCol, setPredictionCol, train, transformSchema

Methods inherited from class org.apache.spark.ml.Estimator

fit, fit, fit, fit

Methods inherited from class org.apache.spark.ml.PipelineStage

transformSchema

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.ml.param.Params

clear, copy, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn, validateParams

Methods inherited from interface org.apache.spark.ml.util.Identifiable

toString, uid

Constructor Detail

Classifier

public Classifier()

Method Detail

setRawPredictionCol

public E setRawPredictionCol(java.lang.String value)

extractLabeledPoints

protected RDD<LabeledPoint> extractLabeledPoints(Dataset<?> dataset,
                                     int numClasses)

Extract labelCol and featuresCol from the given dataset, and put it in an RDD with strong types.

Parameters:

dataset - DataFrame with columns for labels (NumericType) and features (Vector). Labels are cast to DoubleType.

numClasses - Number of classes label can take. Labels must be integers in the range [0, numClasses).

Returns:

(undocumented)

Throws:

SparkException - if any label is not an integer >= 0

getNumClasses

protected int getNumClasses(Dataset<?> dataset,
                int maxNumClasses)

Get the number of classes. This looks in column metadata first, and if that is missing, then this assumes classes are indexed 0,1,...,numClasses-1 and computes numClasses by finding the maximum label value.

Label validation (ensuring all labels are integers >= 0) needs to be handled elsewhere, such as in extractLabeledPoints().

Parameters:

dataset - Dataset which contains a column labelCol

maxNumClasses - Maximum number of classes allowed when inferred from data. If numClasses is specified in the metadata, then maxNumClasses is ignored.

Returns:

number of classes

Throws:

java.lang.IllegalArgumentException - if metadata does not specify numClasses, and the actual numClasses exceeds maxNumClasses

validateAndTransformSchema

public StructType validateAndTransformSchema(StructType schema,
                                    boolean fitting,
                                    DataType featuresDataType)

rawPredictionCol

public Param<java.lang.String> rawPredictionCol()

Param for raw prediction (a.k.a. confidence) column name.

Returns:

(undocumented)

getRawPredictionCol

public java.lang.String getRawPredictionCol()

validateAndTransformSchema

public StructType validateAndTransformSchema(StructType schema,
                                    boolean fitting,
                                    DataType featuresDataType)

Validates and transforms the input schema with the provided param map.

Parameters:

schema - input schema

fitting - whether this is in fitting

featuresDataType - SQL DataType for FeaturesType. E.g., VectorUDT for vector features.

Returns:

output schema

labelCol

public Param<java.lang.String> labelCol()

Param for label column name.

Returns:

(undocumented)

getLabelCol

public java.lang.String getLabelCol()

featuresCol

public Param<java.lang.String> featuresCol()

Param for features column name.

Returns:

(undocumented)

getFeaturesCol

public java.lang.String getFeaturesCol()

predictionCol

public Param<java.lang.String> predictionCol()

Param for prediction column name.

Returns:

(undocumented)

getPredictionCol

public java.lang.String getPredictionCol()