org.apache.spark.scheduler

Class InputFormatInfo

java.lang.Object

org.apache.spark.scheduler.InputFormatInfo

public class InputFormatInfo
extends java.lang.Object

:: DeveloperApi :: Parses and holds information about inputFormat (and files) specified as a parameter.

Constructor Summary

Constructors

Constructor and Description
InputFormatInfo(org.apache.hadoop.conf.Configuration configuration, java.lang.Class<?> inputFormatClazz, java.lang.String path)

Method Summary

Methods

Modifier and Type	Method and Description
static scala.collection.immutable.Map<java.lang.String,scala.collection.immutable.Set<SplitInfo>>	computePreferredLocations(scala.collection.Seq<InputFormatInfo> formats) Computes the preferred locations based on input(s) and returned a location to block map.
org.apache.hadoop.conf.Configuration	configuration()
boolean	equals(java.lang.Object other)
int	hashCode()
protected static void	initializeLogIfNecessary(boolean isInterpreter)
java.lang.Class<?>	inputFormatClazz()
protected static boolean	isTraceEnabled()
protected static org.slf4j.Logger	log()
protected static void	logDebug(scala.Function0<java.lang.String> msg)
protected static void	logDebug(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static void	logError(scala.Function0<java.lang.String> msg)
protected static void	logError(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static void	logInfo(scala.Function0<java.lang.String> msg)
protected static void	logInfo(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static java.lang.String	logName()
protected static void	logTrace(scala.Function0<java.lang.String> msg)
protected static void	logTrace(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static void	logWarning(scala.Function0<java.lang.String> msg)
protected static void	logWarning(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
boolean	mapredInputFormat()
boolean	mapreduceInputFormat()
java.lang.String	path()
java.lang.String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

InputFormatInfo

public InputFormatInfo(org.apache.hadoop.conf.Configuration configuration,
               java.lang.Class<?> inputFormatClazz,
               java.lang.String path)

Method Detail

computePreferredLocations

public static scala.collection.immutable.Map<java.lang.String,scala.collection.immutable.Set<SplitInfo>> computePreferredLocations(scala.collection.Seq<InputFormatInfo> formats)

Computes the preferred locations based on input(s) and returned a location to block map. Typical use of this method for allocation would follow some algo like this:

a) For each host, count number of splits hosted on that host. b) Decrement the currently allocated containers on that host. c) Compute rack info for each host and update rack -> count map based on (b). d) Allocate nodes based on (c) e) On the allocation result, ensure that we don't allocate "too many" jobs on a single node (even if data locality on that is very high) : this is to prevent fragility of job if a single (or small set of) hosts go down.

go to (a) until required nodes are allocated.

If a node 'dies', follow same procedure.

PS: I know the wording here is weird, hopefully it makes some sense !

Parameters:

formats - (undocumented)

Returns:

(undocumented)

logName

protected static java.lang.String logName()

log

protected static org.slf4j.Logger log()

logInfo

protected static void logInfo(scala.Function0<java.lang.String> msg)

logDebug

protected static void logDebug(scala.Function0<java.lang.String> msg)

logTrace

protected static void logTrace(scala.Function0<java.lang.String> msg)

logWarning

protected static void logWarning(scala.Function0<java.lang.String> msg)

logError

protected static void logError(scala.Function0<java.lang.String> msg)

logInfo

protected static void logInfo(scala.Function0<java.lang.String> msg,
           java.lang.Throwable throwable)

logDebug

protected static void logDebug(scala.Function0<java.lang.String> msg,
            java.lang.Throwable throwable)

logTrace

protected static void logTrace(scala.Function0<java.lang.String> msg,
            java.lang.Throwable throwable)

logWarning

protected static void logWarning(scala.Function0<java.lang.String> msg,
              java.lang.Throwable throwable)

logError

protected static void logError(scala.Function0<java.lang.String> msg,
            java.lang.Throwable throwable)

isTraceEnabled

protected static boolean isTraceEnabled()

initializeLogIfNecessary

protected static void initializeLogIfNecessary(boolean isInterpreter)

configuration

public org.apache.hadoop.conf.Configuration configuration()

inputFormatClazz

public java.lang.Class<?> inputFormatClazz()

path

public java.lang.String path()

mapreduceInputFormat

public boolean mapreduceInputFormat()

mapredInputFormat

public boolean mapredInputFormat()

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object

hashCode

public int hashCode()

Overrides:

hashCode in class java.lang.Object

equals

public boolean equals(java.lang.Object other)

Overrides:

equals in class java.lang.Object