public class JobManager extends Object implements IJobManager
| Modifier and Type | Class and Description |
|---|---|
protected static class |
JobManager.JobqueueRecord
Class for tracking existing jobqueue row data
|
protected static class |
JobManager.MutableInteger
Mutable integer class.
|
protected static class |
JobManager.QueueHashItem
This class contains information per job on how many queue items have so far been accumulated.
|
protected static class |
JobManager.ThrottleJobItem
This class represents the information stored PER JOB in the throttling structure.
|
protected static class |
JobManager.ThrottleLimit
This class provides the throttling limits for the job queueing query.
|
protected static class |
JobManager.ThrottleLimitSpec
This is a class which describes an individual throttle limit, in fetches.
|
ACTION_REMOVE, ACTION_RESCAN, DOCSTATE_NEVERPROCESSED, DOCSTATE_OUTOFSCOPE, DOCSTATE_PREVIOUSLYPROCESSED, DOCSTATUS_DELETING, DOCSTATUS_EXPIRING, DOCSTATUS_HOPCOUNTEXCEEDED, DOCSTATUS_INACTIVE, DOCSTATUS_PROCESSING, DOCSTATUS_READYFOREXPIRATION, DOCSTATUS_READYFORPROCESSING, DOCSTATUS_WAITINGFOREVER, DOCSTATUS_WAITINGFOREXPIRATION, DOCSTATUS_WAITINGFORPROCESSING, STOP_ERRORABORT, STOP_MANUALABORT, STOP_MANUALPAUSE, STOP_RESTART, STOP_SCHEDULEPAUSE| Constructor and Description |
|---|
JobManager(IThreadContext threadContext,
IDBInterface database)
Constructor.
|
| Modifier and Type | Method and Description |
|---|---|
protected void |
addBucketExtract(StringBuilder sb,
ArrayList list,
String columnPrefix,
String columnName,
BucketDescription bucketDesc)
Turn a bucket description into a return column.
|
protected boolean |
addCriteria(StringBuilder sb,
ArrayList list,
String fieldPrefix,
String connectionName,
StatusFilterCriteria criteria,
boolean whereEmitted)
Add criteria clauses to query.
|
void |
addDocument(String processID,
Long jobID,
String[] legalLinkTypes,
String docIDHash,
String docID,
String parentIdentifierHash,
String relationshipType,
int hopcountMethod,
String[] dataNames,
Object[][] dataValues,
IPriorityCalculator priority,
String[] prereqEventNames)
Add a document to the queue.
|
void |
addDocuments(String processID,
Long jobID,
String[] legalLinkTypes,
String[] docIDHashes,
String[] docIDs,
String parentIdentifierHash,
String relationshipType,
int hopcountMethod,
String[][] dataNames,
Object[][][] dataValues,
IPriorityCalculator[] documentPriorities,
String[][] prereqEventNames)
Add documents to the queue in bulk.
|
void |
addDocumentsInitial(String processID,
Long jobID,
String[] legalLinkTypes,
String[] docIDHashes,
String[] docIDs,
boolean overrideSchedule,
int hopcountMethod,
IPriorityCalculator[] documentPriorities,
String[][] prereqEventNames)
Add an initial set of documents to the queue.
|
protected void |
addLimits(StringBuilder sb,
int startRow,
int maxRowCount)
Add limit and offset.
|
protected void |
addOrdering(StringBuilder sb,
String[] completeFieldList,
SortOrder sort)
Add ordering.
|
void |
addRemainingDocumentsInitial(String processID,
Long jobID,
String[] legalLinkTypes,
String[] docIDHashes,
int hopcountMethod)
Add an initial set of remaining documents to the queue.
|
protected void |
addWhereClause(StringBuilder sb,
ArrayList list,
String whereClause,
ArrayList whereParams,
boolean wherePresent) |
void |
assessMarkedJobs()
Assess jobs marked to be in need of assessment for connector status changes.
|
boolean |
beginEventSequence(String processID,
String eventName)
Begin an event sequence.
|
protected void |
buildCountsUsingGroupBy(String whereClause,
ArrayList whereParams,
Map<Long,Long> set2Hash,
Map<Long,Long> set3Hash,
Map<Long,Long> set4Hash,
Map<Long,Boolean> set2Exact,
Map<Long,Boolean> set3Exact,
Map<Long,Boolean> set4Exact) |
protected void |
buildCountsUsingIndividualQueries(String whereClause,
ArrayList whereParams,
int maxCount,
Map<Long,Long> set2Hash,
Map<Long,Long> set3Hash,
Map<Long,Long> set4Hash,
Map<Long,Boolean> set2Exact,
Map<Long,Boolean> set3Exact,
Map<Long,Boolean> set4Exact) |
protected static ClauseDescription |
buildOutstandingClause() |
protected static ClauseDescription |
buildProcessedClause() |
protected static HashMap |
buildReorderMap(String[] originalIDHashes,
String[] reorderedIDHashes)
Build a reorder map, describing how to convert an original index into a reordered index.
|
protected DocumentDescription[] |
calculateAffectedDeleteCarrydownChildren(Long jobID,
String[] docIDHashes)
Helper method: Find the document descriptions that will be affected due to carrydown row deletions.
|
protected DocumentDescription[] |
calculateAffectedRestoreCarrydownChildren(Long jobID,
String[] parentIDHashes)
Helper method: Calculate the unique set of affected carrydown children resulting from a "restoreRecords" operation.
|
void |
carrydownChangeDocument(DocumentDescription documentDescription,
IPriorityCalculator docPriority)
Requeue a document because of carrydown changes.
|
void |
carrydownChangeDocumentMultiple(DocumentDescription[] documentDescriptions,
IPriorityCalculator[] docPriorities)
Requeue a document set because of carrydown changes.
|
boolean |
checkIfNotificationReference(String connectionName)
See if there's a reference to a notification connection name.
|
boolean |
checkIfOutputReference(String connectionName)
See if there's a reference to an output connection name.
|
boolean |
checkIfReference(String connectionName)
See if there's a reference to a connection name.
|
boolean |
checkIfTransformationReference(String connectionName)
See if there's a reference to a transformation connection name.
|
boolean |
checkJobActive(Long jobID)
Verify that a specific job is indeed still active.
|
boolean |
checkJobBusy(Long jobID)
Verify if a job is still processing documents, or no longer has any outstanding active documents
|
protected static Long |
checkTimeMatch(long startTime,
long currentTimestamp,
EnumeratedValues daysOfWeek,
EnumeratedValues daysOfMonth,
EnumeratedValues months,
EnumeratedValues years,
EnumeratedValues hours,
EnumeratedValues minutes,
String timezone,
Long duration)
Check if the specified job parameters have a 'hit' within the specified interval.
|
void |
cleanupProcessData()
Reset the job queue for all process IDs.
|
void |
cleanupProcessData(String processID)
Reset the job queue for an individual process ID.
|
void |
clearAllDocumentPriorities()
Clear all document priorities, in preparation for reprioritization of all previously-prioritized documents.
|
void |
clearJobSeedingState(Long jobID)
Clear job seeding state.
|
void |
completeEventSequence(String eventName)
Complete an event sequence.
|
IJobDescription |
createJob()
Create a new job.
|
void |
deinstall()
Uninstall.
|
void |
deleteIngestedDocumentIdentifiers(DocumentDescription[] identifiers)
Delete ingested document identifiers (as part of deleting the owning job).
|
void |
deleteJob(Long id)
Delete a job.
|
void |
deleteJobsReadyForDelete()
Delete jobs in need of being deleted (which are marked "ready for delete").
|
protected DocumentDescription[] |
doDeleteMultiple(Long jobID,
String[] legalLinkTypes,
DocumentDescription[] documentDescriptions,
int hopcountMethod)
Delete documents with no repercussions.
|
void |
doneDocumentsInitial(Long jobID,
String[] legalLinkTypes,
boolean isPartial,
int hopcountMethod)
Signal that a seeding pass has been done.
|
protected static String[] |
eliminateDuplicates(String[] docIDHashes)
Eliminate duplicates, and sort
|
protected boolean |
emitClauseStart(StringBuilder sb,
boolean whereEmitted)
Emit a WHERE or an AND, depending...
|
boolean |
errorAbort(Long jobID,
String errorText)
Abort a running job due to a fatal error condition.
|
void |
exportConfiguration(OutputStream os)
Export configuration
|
protected void |
fetchAndProcessDocuments(List<DocumentDescription> answers,
Long currentTimeValue,
Long currentPriorityValue,
JobManager.ThrottleLimit vList,
IRepositoryConnection[] connections,
String processID)
Fetch and process documents matching the passed-in criteria
|
boolean[] |
findHopCounts(Long jobID,
String[] legalLinkTypes,
String[] docIDHashes,
String linkType,
int limit,
int hopcountMethod)
Get the specified hop counts, with the limit as described.
|
IJobDescription[] |
findJobsForConnection(String connectionName)
Get the job IDs associated with a given connection name.
|
DocumentDescription[] |
finishDocuments(Long jobID,
String[] legalLinkTypes,
String[] parentIdentifierHashes,
int hopcountMethod)
Complete adding child documents to the queue, for a set of documents.
|
void |
finishJobResumes(long timestamp,
List<IJobDescription> modifiedJobs)
Complete the sequence that resumes jobs, either from a pause or from a scheduling window
wait.
|
void |
finishJobs()
Put all eligible jobs in the "shutting down" state.
|
void |
finishJobStops(long timestamp,
List<IJobDescription> modifiedJobs,
List<Integer> stopNotificationTypes)
Complete the sequence that stops jobs, either for abort, pause, or because of a scheduling
window.
|
IResultSet |
genDocumentStatus(String connectionName,
StatusFilterCriteria filterCriteria,
SortOrder sortOrder,
int startRow,
int rowCount)
Run a 'document status' report.
|
IResultSet |
genQueueStatus(String connectionName,
StatusFilterCriteria filterCriteria,
SortOrder sortOrder,
BucketDescription idBucketDescription,
int startRow,
int rowCount)
Run a 'queue status' report.
|
IJobDescription[] |
getAllJobs()
Load a sorted list of job descriptions.
|
String[] |
getAllSeeds(Long jobID)
Get all the current seeds.
|
JobStatus[] |
getAllStatus()
Get a list of all jobs, and their status information.
|
JobStatus[] |
getAllStatus(boolean includeCounts)
Get a list of all jobs, and their status information.
|
JobStatus[] |
getAllStatus(boolean includeCounts,
int maxCount)
Get a list of all jobs, and their status information.
|
DocumentSetAndFlags |
getExpiredDocuments(String processID,
int n,
long currentTime)
Get up to the next n documents to be expired.
|
JobStatus[] |
getFinishedJobs()
Get a list of completed jobs, and their statistics.
|
JobStatus[] |
getFinishedJobs(boolean includeCounts)
Get a list of completed jobs, and their statistics.
|
JobStatus[] |
getFinishedJobs(boolean includeCounts,
int maxCount)
Get a list of completed jobs, and their statistics.
|
protected String |
getHopLockName(Long jobID)
Get the hoplock for a given job ID
|
JobNotifyRecord[] |
getJobsReadyForDelete(String processID)
Find the list of jobs that need to have their connectors notified of job deletion.
|
JobDeleteRecord[] |
getJobsReadyForDeleteCleanup(String processID)
Get the list of jobs that are ready for delete cleanup.
|
JobNotifyRecord[] |
getJobsReadyForInactivity(String processID)
Find the list of jobs that need to have their connectors notified of job completion.
|
JobSeedingRecord[] |
getJobsReadyForSeeding(String processID,
long currentTime)
Get the list of jobs that are ready for seeding.
|
JobStartRecord[] |
getJobsReadyForStartup(String processID)
Get the list of jobs that are ready for startup.
|
DocumentSetAndFlags |
getNextCleanableDocuments(String processID,
int maxCount,
long currentTime)
Get list of cleanable document descriptions.
|
DocumentDescription[] |
getNextDeletableDocuments(String processID,
int maxCount,
long currentTime)
Get list of deletable document descriptions.
|
DocumentDescription[] |
getNextDocuments(String processID,
int n,
long currentTime,
long interval,
BlockingDocuments blockingDocuments,
PerformanceStatistics statistics,
DepthStatistics scanRecord)
/** Get up to the next n document(s) to be fetched and processed.
|
DocumentDescription[] |
getNextNotYetProcessedReprioritizationDocuments(String processID,
int n)
Get a list of not-yet-processed documents to reprioritize.
|
protected long |
getRandomAmount()
Sleep a random amount of time after a transaction abort.
|
JobStatus[] |
getRunningJobs()
Get a list of running jobs.
|
JobStatus[] |
getRunningJobs(boolean includeCounts)
Get a list of running jobs.
|
JobStatus[] |
getRunningJobs(boolean includeCounts,
int maxCount)
Get a list of running jobs.
|
JobStatus |
getStatus(Long jobID)
Get the status of a job.
|
JobStatus |
getStatus(Long jobID,
boolean includeCounts)
Get the status of a job.
|
JobStatus |
getStatus(Long jobID,
boolean includeCounts,
int maxCount)
Get the status of a job.
|
protected String[] |
getUnindexableDocumentIdentifiers(DocumentDescription[] documentIdentifiers,
String connectionName)
Get a list of document identifiers that should actually be deleted from the index, from a list that
might contain identifiers that are shared with other jobs, which are targeted to the same output connection.
|
void |
importConfiguration(InputStream is)
Import configuration
|
void |
inactivateJob(Long jobID)
Inactivate a job, from the notification state.
|
void |
install()
Install.
|
IJobDescription |
load(Long id)
Load a job for editing.
|
IJobDescription |
load(Long id,
boolean readOnly)
Load a job.
|
protected static String |
makeCompositeID(String docIDHash,
String connectionName)
Create a composite document hash key.
|
protected JobStatus[] |
makeJobStatus(String whereClause,
ArrayList whereParams,
boolean includeCounts,
int maxCount)
Make a job status array from a query result.
|
void |
manualAbort(Long jobID)
Manually abort a running job.
|
void |
manualAbortRestart(Long jobID)
Manually restart a running job.
|
void |
manualAbortRestart(Long jobID,
boolean requestMinimum)
Manually restart a running job.
|
void |
manualStart(Long jobID)
Manually start a job.
|
void |
manualStart(Long jobID,
boolean requestMinimum)
Manually start a job.
|
protected static Integer |
mapToNotificationType(int jobStatus,
boolean noErrorText) |
DocumentDescription[] |
markDocumentCleanedUp(Long jobID,
String[] legalLinkTypes,
DocumentDescription documentDescription,
int hopcountMethod)
Delete from queue as a result of cleaning up an unreachable document.
|
DocumentDescription[] |
markDocumentCleanedUpMultiple(Long jobID,
String[] legalLinkTypes,
DocumentDescription[] documentDescriptions,
int hopcountMethod)
Delete from queue as a result of cleaning up an unreachable document.
|
void |
markDocumentCompleted(DocumentDescription documentDescription)
Note completion of document processing by a job thread of a document.
|
void |
markDocumentCompletedMultiple(DocumentDescription[] documentDescriptions)
Note completion of document processing by a job thread of a document.
|
DocumentDescription[] |
markDocumentDeleted(Long jobID,
String[] legalLinkTypes,
DocumentDescription documentDescription,
int hopcountMethod)
Delete from queue as a result of processing of an active document.
|
DocumentDescription[] |
markDocumentDeletedMultiple(Long jobID,
String[] legalLinkTypes,
DocumentDescription[] documentDescriptions,
int hopcountMethod)
Delete from queue as a result of processing of an active document.
|
DocumentDescription[] |
markDocumentExpired(Long jobID,
String[] legalLinkTypes,
DocumentDescription documentDescription,
int hopcountMethod)
Delete from queue as a result of expiration of an active document.
|
DocumentDescription[] |
markDocumentExpiredMultiple(Long jobID,
String[] legalLinkTypes,
DocumentDescription[] documentDescriptions,
int hopcountMethod)
Delete from queue as a result of expiration of an active document.
|
DocumentDescription[] |
markDocumentHopcountRemoval(Long jobID,
String[] legalLinkTypes,
DocumentDescription documentDescription,
int hopcountMethod)
Mark hopcount removal from queue as a result of processing of an active document.
|
DocumentDescription[] |
markDocumentHopcountRemovalMultiple(Long jobID,
String[] legalLinkTypes,
DocumentDescription[] documentDescriptions,
int hopcountMethod)
Mark hopcount removal from queue as a result of processing of an active document.
|
protected int |
maxClauseProcessDeleteHashSet()
Get maximum count.
|
protected void |
noDocPriorities(Long jobID) |
void |
noteConnectionChange(String connectionName)
Note a change in connection configuration.
|
protected void |
noteConnectionDeregistration(List<String> list)
Note deregistration for a batch of connection names.
|
protected void |
noteConnectionRegistration(List<String> list)
Note registration for a batch of connection names.
|
void |
noteConnectorDeregistration(String[] connectionNames)
Note the deregistration of a connector used by the specified connections.
|
void |
noteConnectorRegistration(String[] connectionNames)
Note the registration of a connector used by the specified connections.
|
void |
noteJobDeleteStarted(Long jobID,
long startTime)
Note job delete started.
|
void |
noteJobSeeded(Long jobID,
String seedingVersion)
Note job seeded.
|
void |
noteJobStarted(Long jobID,
long startTime,
String seedingVersion)
Note job started.
|
void |
noteNotificationConnectionChange(String connectionName)
Note a change in notification connection configuration.
|
protected void |
noteNotificationConnectionDeregistration(List<String> list)
Note deregistration for a batch of notification connection names.
|
protected void |
noteNotificationConnectionRegistration(List<String> list)
Note registration for a batch of connection names.
|
void |
noteNotificationConnectorDeregistration(String[] connectionNames)
Note the deregistration of a notification connector used by the specified connections.
|
void |
noteNotificationConnectorRegistration(String[] connectionNames)
Note the registration of a notification connector used by the specified connections.
|
void |
noteOutputConnectionChange(String connectionName)
Note a change in output connection configuration.
|
protected void |
noteOutputConnectionDeregistration(List<String> list)
Note deregistration for a batch of output connection names.
|
protected void |
noteOutputConnectionRegistration(List<String> list)
Note registration for a batch of output connection names.
|
void |
noteOutputConnectorDeregistration(String[] connectionNames)
Note the deregistration of an output connector used by the specified connections.
|
void |
noteOutputConnectorRegistration(String[] connectionNames)
Note the registration of an output connector used by the specified connections.
|
void |
noteTransformationConnectionChange(String connectionName)
Note a change in transformation connection configuration.
|
protected void |
noteTransformationConnectionDeregistration(List<String> list)
Note deregistration for a batch of transformation connection names.
|
protected void |
noteTransformationConnectionRegistration(List<String> list)
Note registration for a batch of transformation connection names.
|
void |
noteTransformationConnectorDeregistration(String[] connectionNames)
Note the deregistration of a transformation connector used by the specified connections.
|
void |
noteTransformationConnectorRegistration(String[] connectionNames)
Note the registration of a transformation connector used by the specified connections.
|
void |
pauseJob(Long jobID)
Pause a job.
|
void |
prepareDeleteScan(Long jobID)
Prepare for a delete scan.
|
void |
prepareForClusterStart()
Prepare to start the entire cluster.
|
protected void |
prepareFullScan(Long jobID,
String[] legalLinkTypes,
int hopcountMethod)
Prepare for a full scan.
|
void |
prepareJobScan(Long jobID,
String[] legalLinkTypes,
int hopcountMethod,
int connectorModel,
boolean continuousJob,
boolean fromBeginningOfTime,
boolean requestMinimum)
Prepare a job to be run.
|
protected void |
processDeleteHashSet(Long jobID,
HashMap resultHash,
ArrayList list)
Helper method: look up rows affected by a deleteRecords operation.
|
protected void |
processParentHashSet(Long jobID,
HashMap resultHash,
ArrayList list)
Helper method: look up rows affected by a restoreRecords operation.
|
protected void |
queueAllExisting(Long jobID,
String[] legalLinkTypes)
Queue all existing.
|
protected EnumeratedValues |
readEnumeratedValues(InputStream is) |
void |
removeJob(Long jobID)
Remove a job, from the notification state.
|
void |
requeueDocument(DocumentDescription documentDescription,
Long executeTime,
int action)
Requeue a document for further processing in the future.
|
void |
requeueDocumentMultiple(DocumentDescription[] documentDescriptions,
Long[] executeTimes,
int[] actions)
Requeue a document for further processing in the future.
|
void |
resetCleaningDocument(DocumentDescription documentDescription,
long checkTime)
Reset a cleaning document back to its former state.
|
void |
resetCleaningDocumentMultiple(DocumentDescription[] documentDescriptions,
long checkTime)
Reset a set of cleaning documents for further processing in the future.
|
void |
resetDeleteNotifyJob(Long jobID)
Reset a job that is delete notifying back to "ready for delete notify"
state.
|
void |
resetDeleteStartupWorkerStatus(String processID)
Reset as part of restoring delete startup threads.
|
void |
resetDeletingDocument(DocumentDescription documentDescription,
long checkTime)
Reset a deleting document back to its former state.
|
void |
resetDeletingDocumentMultiple(DocumentDescription[] documentDescriptions,
long checkTime)
Reset a set of deleting documents for further processing in the future.
|
void |
resetDocCleanupWorkerStatus(String processID)
Reset as part of restoring doc cleanup threads.
|
void |
resetDocDeleteWorkerStatus(String processID)
Reset as part of restoring doc delete threads.
|
void |
resetDocument(DocumentDescription documentDescription,
long executeTime,
int action,
long failTime,
int failCount)
Reset an active document back to its former state.
|
void |
resetDocumentMultiple(DocumentDescription[] documentDescriptions,
long executeTime,
int action,
long failTime,
int failCount)
Reset a set of documents for further processing in the future.
|
void |
resetDocumentWorkerStatus(String processID)
Reset as part of restoring document worker threads.
|
void |
resetJobs(long currentTime,
List<IJobDescription> resetJobs)
Reset eligible jobs either back to the "inactive" state, or make them active again.
|
void |
resetJobSchedule(Long jobID)
Reset job schedule.
|
void |
resetNotificationWorkerStatus(String processID)
Reset as part of restoring notification threads.
|
void |
resetNotifyJob(Long jobID)
Reset a job that is notifying back to "ready for notify"
state.
|
void |
resetSeedingWorkerStatus(String processID)
Reset as part of restoring seeding threads.
|
void |
resetSeedJob(Long jobID)
Reset a seeding job back to "active" state.
|
void |
resetStartDeleteJob(Long jobID)
Reset a job starting for delete back to "ready for delete"
state.
|
void |
resetStartupJob(Long jobID)
Reset a starting job back to "ready for startup" state.
|
void |
resetStartupWorkerStatus(String processID)
Reset as part of restoring startup threads.
|
void |
restartJob(Long jobID)
Restart a paused job.
|
String[] |
retrieveParentData(Long jobID,
String docIDHash,
String dataName)
Retrieve specific parent data for a given document.
|
CharacterInput[] |
retrieveParentDataAsFiles(Long jobID,
String docIDHash,
String dataName)
Retrieve specific parent data for a given document.
|
void |
retryDeleteNotification(JobNotifyRecord jnr,
long failTime,
int failCount)
Retry delete notification.
|
void |
retryNotification(JobNotifyRecord jnr,
long failTime,
int failCount)
Retry notification.
|
void |
retrySeeding(JobSeedingRecord jsr,
long failTime,
int failCount)
Retry seeding.
|
void |
retryStartup(JobStartRecord jsr,
long failTime,
int failCount)
Retry startup.
|
void |
revertDocuments(Long jobID,
String[] legalLinkTypes,
String[] parentIdentifierHashes)
Undo the addition of child documents to the queue, for a set of documents.
|
void |
save(IJobDescription jobDescription)
Save a job.
|
protected void |
sleepFor(long amt) |
void |
startJobs(long currentTime,
List<Long> unwaitList)
Start all jobs in need of starting.
|
void |
waitJobs(long currentTime,
List<Long> waitList)
Put active or paused jobs in wait state, if they've exceeded their window.
|
void |
writeDocumentPriorities(DocumentDescription[] documentDescriptions,
IPriorityCalculator[] priorities)
Save a set of document priorities.
|
protected static void |
writeEnumeratedValues(OutputStream os,
EnumeratedValues ev) |
public static final String _rcsid
protected static final String stufferLock
protected static final String reprioritizationLock
protected static final String deleteStufferLock
protected static final String expireStufferLock
protected static final String cleanStufferLock
protected static final String jobStopLock
protected static final String jobResumeLock
protected static final String hopLock
protected final IDBInterface database
protected final IOutputConnectionManager outputMgr
protected final IRepositoryConnectionManager connectionMgr
protected final INotificationConnectionManager notificationMgr
protected final ITransformationConnectionManager transformationMgr
protected final IOutputConnectorManager outputConnectorMgr
protected final IConnectorManager connectorMgr
protected final ITransformationConnectorManager transformationConnectorMgr
protected final IRepositoryConnectorPool repositoryConnectorPool
protected final ILockManager lockManager
protected final IThreadContext threadContext
protected final JobQueue jobQueue
protected final Jobs jobs
protected final HopCount hopCount
protected final Carrydown carryDown
protected final EventManager eventManager
protected static Random random
public JobManager(IThreadContext threadContext, IDBInterface database) throws ManifoldCFException
threadContext - is the thread context.database - is the database.ManifoldCFExceptionpublic void install()
throws ManifoldCFException
install in interface IJobManagerManifoldCFExceptionpublic void deinstall()
throws ManifoldCFException
deinstall in interface IJobManagerManifoldCFExceptionpublic void exportConfiguration(OutputStream os) throws IOException, ManifoldCFException
exportConfiguration in interface IJobManagerIOExceptionManifoldCFExceptionprotected static void writeEnumeratedValues(OutputStream os, EnumeratedValues ev) throws IOException
IOExceptionpublic void importConfiguration(InputStream is) throws IOException, ManifoldCFException
importConfiguration in interface IJobManagerIOExceptionManifoldCFExceptionprotected EnumeratedValues readEnumeratedValues(InputStream is) throws IOException
IOExceptionpublic void noteConnectorDeregistration(String[] connectionNames) throws ManifoldCFException
noteConnectorDeregistration in interface IJobManagerconnectionNames - is the set of connection names.ManifoldCFExceptionprotected void noteConnectionDeregistration(List<String> list) throws ManifoldCFException
ManifoldCFExceptionpublic void noteConnectorRegistration(String[] connectionNames) throws ManifoldCFException
noteConnectorRegistration in interface IJobManagerconnectionNames - is the set of connection names.ManifoldCFExceptionprotected void noteConnectionRegistration(List<String> list) throws ManifoldCFException
ManifoldCFExceptionpublic void noteNotificationConnectorDeregistration(String[] connectionNames) throws ManifoldCFException
noteNotificationConnectorDeregistration in interface IJobManagerconnectionNames - is the set of connection names.ManifoldCFExceptionprotected void noteNotificationConnectionDeregistration(List<String> list) throws ManifoldCFException
ManifoldCFExceptionpublic void noteNotificationConnectorRegistration(String[] connectionNames) throws ManifoldCFException
noteNotificationConnectorRegistration in interface IJobManagerconnectionNames - is the set of connection names.ManifoldCFExceptionprotected void noteNotificationConnectionRegistration(List<String> list) throws ManifoldCFException
ManifoldCFExceptionpublic void noteOutputConnectorDeregistration(String[] connectionNames) throws ManifoldCFException
noteOutputConnectorDeregistration in interface IJobManagerconnectionNames - is the set of connection names.ManifoldCFExceptionprotected void noteOutputConnectionDeregistration(List<String> list) throws ManifoldCFException
ManifoldCFExceptionpublic void noteOutputConnectorRegistration(String[] connectionNames) throws ManifoldCFException
noteOutputConnectorRegistration in interface IJobManagerconnectionNames - is the set of connection names.ManifoldCFExceptionprotected void noteOutputConnectionRegistration(List<String> list) throws ManifoldCFException
ManifoldCFExceptionpublic void noteTransformationConnectorDeregistration(String[] connectionNames) throws ManifoldCFException
noteTransformationConnectorDeregistration in interface IJobManagerconnectionNames - is the set of connection names.ManifoldCFExceptionprotected void noteTransformationConnectionDeregistration(List<String> list) throws ManifoldCFException
ManifoldCFExceptionpublic void noteTransformationConnectorRegistration(String[] connectionNames) throws ManifoldCFException
noteTransformationConnectorRegistration in interface IJobManagerconnectionNames - is the set of connection names.ManifoldCFExceptionprotected void noteTransformationConnectionRegistration(List<String> list) throws ManifoldCFException
ManifoldCFExceptionpublic void noteConnectionChange(String connectionName) throws ManifoldCFException
noteConnectionChange in interface IJobManagerManifoldCFExceptionpublic void noteNotificationConnectionChange(String connectionName) throws ManifoldCFException
noteNotificationConnectionChange in interface IJobManagerManifoldCFExceptionpublic void noteOutputConnectionChange(String connectionName) throws ManifoldCFException
noteOutputConnectionChange in interface IJobManagerManifoldCFExceptionpublic void noteTransformationConnectionChange(String connectionName) throws ManifoldCFException
noteTransformationConnectionChange in interface IJobManagerManifoldCFExceptionpublic void assessMarkedJobs()
throws ManifoldCFException
assessMarkedJobs in interface IJobManagerManifoldCFExceptionpublic IJobDescription[] getAllJobs() throws ManifoldCFException
getAllJobs in interface IJobManagerManifoldCFExceptionpublic IJobDescription createJob() throws ManifoldCFException
createJob in interface IJobManagerManifoldCFExceptionpublic void deleteJob(Long id) throws ManifoldCFException
deleteJob in interface IJobManagerid - is the job's identifier. This method will purge all the records belonging to the job from the database, as
well as remove all documents indexed by the job from the index.ManifoldCFExceptionpublic IJobDescription load(Long id) throws ManifoldCFException
load in interface IJobManagerid - is the job's identifier.ManifoldCFExceptionpublic IJobDescription load(Long id, boolean readOnly) throws ManifoldCFException
load in interface IJobManagerid - is the job's identifier.readOnly - is true if a read-only object is desired.ManifoldCFExceptionpublic void save(IJobDescription jobDescription) throws ManifoldCFException
save in interface IJobManagerjobDescription - is the job description.ManifoldCFExceptionpublic boolean checkIfReference(String connectionName) throws ManifoldCFException
checkIfReference in interface IJobManagerconnectionName - is the name of the connection.ManifoldCFExceptionpublic boolean checkIfNotificationReference(String connectionName) throws ManifoldCFException
checkIfNotificationReference in interface IJobManagerconnectionName - is the name of the connection.ManifoldCFExceptionpublic boolean checkIfOutputReference(String connectionName) throws ManifoldCFException
checkIfOutputReference in interface IJobManagerconnectionName - is the name of the connection.ManifoldCFExceptionpublic boolean checkIfTransformationReference(String connectionName) throws ManifoldCFException
checkIfTransformationReference in interface IJobManagerconnectionName - is the name of the connection.ManifoldCFExceptionpublic IJobDescription[] findJobsForConnection(String connectionName) throws ManifoldCFException
findJobsForConnection in interface IJobManagerconnectionName - is the name of the connection.ManifoldCFExceptionpublic void clearJobSeedingState(Long jobID) throws ManifoldCFException
clearJobSeedingState in interface IJobManagerjobID - is the job ID.ManifoldCFExceptionpublic void cleanupProcessData(String processID) throws ManifoldCFException
cleanupProcessData in interface IJobManagerprocessID - is the process ID of the node we want to clean up after.ManifoldCFExceptionpublic void cleanupProcessData()
throws ManifoldCFException
cleanupProcessData in interface IJobManagerManifoldCFExceptionpublic void prepareForClusterStart()
throws ManifoldCFException
prepareForClusterStart in interface IJobManagerManifoldCFExceptionpublic void resetDocumentWorkerStatus(String processID) throws ManifoldCFException
resetDocumentWorkerStatus in interface IJobManagerprocessID - is the current process ID.ManifoldCFExceptionpublic void resetSeedingWorkerStatus(String processID) throws ManifoldCFException
resetSeedingWorkerStatus in interface IJobManagerprocessID - is the current process ID.ManifoldCFExceptionpublic void resetDocDeleteWorkerStatus(String processID) throws ManifoldCFException
resetDocDeleteWorkerStatus in interface IJobManagerprocessID - is the current process ID.ManifoldCFExceptionpublic void resetDocCleanupWorkerStatus(String processID) throws ManifoldCFException
resetDocCleanupWorkerStatus in interface IJobManagerprocessID - is the current process ID.ManifoldCFExceptionpublic void resetDeleteStartupWorkerStatus(String processID) throws ManifoldCFException
resetDeleteStartupWorkerStatus in interface IJobManagerprocessID - is the current process ID.ManifoldCFExceptionpublic void resetNotificationWorkerStatus(String processID) throws ManifoldCFException
resetNotificationWorkerStatus in interface IJobManagerprocessID - is the current process ID.ManifoldCFExceptionpublic void resetStartupWorkerStatus(String processID) throws ManifoldCFException
resetStartupWorkerStatus in interface IJobManagerprocessID - is the current process ID.ManifoldCFExceptionpublic void deleteIngestedDocumentIdentifiers(DocumentDescription[] identifiers) throws ManifoldCFException
deleteIngestedDocumentIdentifiers in interface IJobManageridentifiers - is the set of document identifiers.ManifoldCFExceptionpublic DocumentSetAndFlags getNextCleanableDocuments(String processID, int maxCount, long currentTime) throws ManifoldCFException
getNextCleanableDocuments in interface IJobManagerprocessID - is the current process ID.maxCount - is the maximum number of documents to return.currentTime - is the current time; some fetches do not occur until a specific time.ManifoldCFExceptionprotected static String makeCompositeID(String docIDHash, String connectionName)
public DocumentDescription[] getNextDeletableDocuments(String processID, int maxCount, long currentTime) throws ManifoldCFException
getNextDeletableDocuments in interface IJobManagerprocessID - is the current process ID.maxCount - is the maximum number of documents to return.currentTime - is the current time; some fetches do not occur until a specific time.ManifoldCFExceptionprotected String[] getUnindexableDocumentIdentifiers(DocumentDescription[] documentIdentifiers, String connectionName) throws ManifoldCFException
documentIdentifiers - is the set of document identifiers to consider.connectionName - is the connection name for ALL the document identifiers.ManifoldCFExceptionpublic void clearAllDocumentPriorities()
throws ManifoldCFException
clearAllDocumentPriorities in interface IJobManagerManifoldCFExceptionpublic DocumentDescription[] getNextNotYetProcessedReprioritizationDocuments(String processID, int n) throws ManifoldCFException
getNextNotYetProcessedReprioritizationDocuments in interface IJobManagerprocessID - is the process that requests the reprioritization documents.n - is the maximum number of document descriptions desired.ManifoldCFExceptionpublic void writeDocumentPriorities(DocumentDescription[] documentDescriptions, IPriorityCalculator[] priorities) throws ManifoldCFException
writeDocumentPriorities in interface IJobManagerdocumentDescriptions - are the document descriptions.priorities - are the desired priorities.ManifoldCFExceptionpublic DocumentSetAndFlags getExpiredDocuments(String processID, int n, long currentTime) throws ManifoldCFException
getExpiredDocuments in interface IJobManagerprocessID - is the current process ID.n - is the maximum number of records desired.currentTime - is the current time.ManifoldCFExceptionpublic DocumentDescription[] getNextDocuments(String processID, int n, long currentTime, long interval, BlockingDocuments blockingDocuments, PerformanceStatistics statistics, DepthStatistics scanRecord) throws ManifoldCFException
getNextDocuments in interface IJobManagerprocessID - is the current process ID.n - is the maximum number of records desired.currentTime - is the current time; some fetches do not occur until a specific time.interval - is the number of milliseconds that this set of documents should represent (for throttling).blockingDocuments - is the place to record documents that were encountered, are eligible for reprioritization,
but could not be queued due to throttling considerations.statistics - are the current performance statistics per connection, which are used to balance the queue stuffing
so that individual connections are not overwhelmed.scanRecord - retains the bins from all documents encountered from the query, even those that were skipped due
to being overcommitted.ManifoldCFExceptionprotected void fetchAndProcessDocuments(List<DocumentDescription> answers, Long currentTimeValue, Long currentPriorityValue, JobManager.ThrottleLimit vList, IRepositoryConnection[] connections, String processID) throws ManifoldCFException
ManifoldCFExceptionpublic boolean checkJobActive(Long jobID) throws ManifoldCFException
checkJobActive in interface IJobManagerjobID - is the job identifier.ManifoldCFExceptionpublic boolean checkJobBusy(Long jobID) throws ManifoldCFException
checkJobBusy in interface IJobManagerManifoldCFExceptionpublic void markDocumentCompletedMultiple(DocumentDescription[] documentDescriptions) throws ManifoldCFException
markDocumentCompletedMultiple in interface IJobManagerdocumentDescriptions - are the description objects for the documents that were processed.ManifoldCFExceptionpublic void markDocumentCompleted(DocumentDescription documentDescription) throws ManifoldCFException
markDocumentCompleted in interface IJobManagerdocumentDescription - is the description object for the document that was processed.ManifoldCFExceptionpublic DocumentDescription[] markDocumentDeletedMultiple(Long jobID, String[] legalLinkTypes, DocumentDescription[] documentDescriptions, int hopcountMethod) throws ManifoldCFException
markDocumentDeletedMultiple in interface IJobManagerdocumentDescriptions - are the set of description objects for the documents that were processed.hopcountMethod - describes how to handle deletions for hopcount purposes.ManifoldCFExceptionpublic DocumentDescription[] markDocumentDeleted(Long jobID, String[] legalLinkTypes, DocumentDescription documentDescription, int hopcountMethod) throws ManifoldCFException
markDocumentDeleted in interface IJobManagerdocumentDescription - is the description object for the document that was processed.hopcountMethod - describes how to handle deletions for hopcount purposes.ManifoldCFExceptionpublic DocumentDescription[] markDocumentHopcountRemovalMultiple(Long jobID, String[] legalLinkTypes, DocumentDescription[] documentDescriptions, int hopcountMethod) throws ManifoldCFException
markDocumentHopcountRemovalMultiple in interface IJobManagerdocumentDescriptions - are the set of description objects for the documents that were processed.hopcountMethod - describes how to handle deletions for hopcount purposes.ManifoldCFExceptionpublic DocumentDescription[] markDocumentHopcountRemoval(Long jobID, String[] legalLinkTypes, DocumentDescription documentDescription, int hopcountMethod) throws ManifoldCFException
markDocumentHopcountRemoval in interface IJobManagerdocumentDescription - is the description object for the document that was processed.hopcountMethod - describes how to handle deletions for hopcount purposes.ManifoldCFExceptionpublic DocumentDescription[] markDocumentExpiredMultiple(Long jobID, String[] legalLinkTypes, DocumentDescription[] documentDescriptions, int hopcountMethod) throws ManifoldCFException
markDocumentExpiredMultiple in interface IJobManagerdocumentDescriptions - are the set of description objects for the documents that were processed.hopcountMethod - describes how to handle deletions for hopcount purposes.ManifoldCFExceptionpublic DocumentDescription[] markDocumentExpired(Long jobID, String[] legalLinkTypes, DocumentDescription documentDescription, int hopcountMethod) throws ManifoldCFException
markDocumentExpired in interface IJobManagerdocumentDescription - is the description object for the document that was processed.hopcountMethod - describes how to handle deletions for hopcount purposes.ManifoldCFExceptionpublic DocumentDescription[] markDocumentCleanedUpMultiple(Long jobID, String[] legalLinkTypes, DocumentDescription[] documentDescriptions, int hopcountMethod) throws ManifoldCFException
markDocumentCleanedUpMultiple in interface IJobManagerdocumentDescriptions - are the set of description objects for the documents that were processed.hopcountMethod - describes how to handle deletions for hopcount purposes.ManifoldCFExceptionpublic DocumentDescription[] markDocumentCleanedUp(Long jobID, String[] legalLinkTypes, DocumentDescription documentDescription, int hopcountMethod) throws ManifoldCFException
markDocumentCleanedUp in interface IJobManagerdocumentDescription - is the description object for the document that was processed.hopcountMethod - describes how to handle deletions for hopcount purposes.ManifoldCFExceptionprotected DocumentDescription[] doDeleteMultiple(Long jobID, String[] legalLinkTypes, DocumentDescription[] documentDescriptions, int hopcountMethod) throws ManifoldCFException
documentDescriptions - are the set of description objects for the documents that were processed.hopcountMethod - describes how to handle deletions for hopcount purposes.ManifoldCFExceptionprotected DocumentDescription[] calculateAffectedDeleteCarrydownChildren(Long jobID, String[] docIDHashes) throws ManifoldCFException
ManifoldCFExceptionprotected int maxClauseProcessDeleteHashSet()
protected void processDeleteHashSet(Long jobID, HashMap resultHash, ArrayList list) throws ManifoldCFException
ManifoldCFExceptionpublic void requeueDocumentMultiple(DocumentDescription[] documentDescriptions, Long[] executeTimes, int[] actions) throws ManifoldCFException
requeueDocumentMultiple in interface IJobManagerdocumentDescriptions - is the set of description objects for the document that was processed.executeTimes - are the times that the documents should be rescanned. Null indicates "never".actions - are what should be done when the time arrives. Choices are ACTION_RESCAN or ACTION_REMOVE.ManifoldCFExceptionpublic void requeueDocument(DocumentDescription documentDescription, Long executeTime, int action) throws ManifoldCFException
requeueDocument in interface IJobManagerdocumentDescription - is the description object for the document that was processed.executeTime - is the time that the document should be rescanned. Null indicates "never".action - is what should be done when the time arrives. Choices include ACTION_RESCAN or ACTION_REMOVE.ManifoldCFExceptionpublic void resetDocumentMultiple(DocumentDescription[] documentDescriptions, long executeTime, int action, long failTime, int failCount) throws ManifoldCFException
resetDocumentMultiple in interface IJobManagerdocumentDescriptions - is the set of description objects for the document that was processed.executeTime - is the time that the documents should be rescanned.failTime - is the time beyond which a service interruption will be considered a hard failure.failCount - is the number of retries beyond which a service interruption will be considered a hard failure.ManifoldCFExceptionpublic void resetCleaningDocumentMultiple(DocumentDescription[] documentDescriptions, long checkTime) throws ManifoldCFException
resetCleaningDocumentMultiple in interface IJobManagerdocumentDescriptions - is the set of description objects for the document that was cleaned.checkTime - is the minimum time for the next cleaning attempt.ManifoldCFExceptionpublic void resetCleaningDocument(DocumentDescription documentDescription, long checkTime) throws ManifoldCFException
resetCleaningDocument in interface IJobManagerdocumentDescription - is the description of the document that was cleaned.checkTime - is the minimum time for the next cleaning attempt.ManifoldCFExceptionpublic void resetDeletingDocumentMultiple(DocumentDescription[] documentDescriptions, long checkTime) throws ManifoldCFException
resetDeletingDocumentMultiple in interface IJobManagerdocumentDescriptions - is the set of description objects for the document that was processed.checkTime - is the minimum time for the next cleaning attempt.ManifoldCFExceptionpublic void resetDeletingDocument(DocumentDescription documentDescription, long checkTime) throws ManifoldCFException
resetDeletingDocument in interface IJobManagerdocumentDescription - is the description object for the document that was cleaned.checkTime - is the minimum time for the next cleaning attempt.ManifoldCFExceptionpublic void resetDocument(DocumentDescription documentDescription, long executeTime, int action, long failTime, int failCount) throws ManifoldCFException
resetDocument in interface IJobManagerdocumentDescription - is the description object for the document that was processed.executeTime - is the time that the document should be rescanned.failTime - is the time that the document should be considered to have failed, if it has not been
successfully read until then.failCount - is the number of permitted failures before a hard error is signalled.ManifoldCFExceptionprotected static String[] eliminateDuplicates(String[] docIDHashes)
protected static HashMap buildReorderMap(String[] originalIDHashes, String[] reorderedIDHashes)
public void retryStartup(JobStartRecord jsr, long failTime, int failCount) throws ManifoldCFException
retryStartup in interface IJobManagerjsr - is the current job notification record.failTime - is the new fail time (-1L if none).failCount - is the new fail retry count (-1 if none).ManifoldCFExceptionpublic void retrySeeding(JobSeedingRecord jsr, long failTime, int failCount) throws ManifoldCFException
retrySeeding in interface IJobManagerjsr - is the current job seeding record.failTime - is the new fail time (-1L if none).failCount - is the new fail retry count (-1 if none).ManifoldCFExceptionpublic void retryNotification(JobNotifyRecord jnr, long failTime, int failCount) throws ManifoldCFException
retryNotification in interface IJobManagerjnr - is the current job notification record.failTime - is the new fail time (-1L if none).failCount - is the new fail retry count (-1 if none).ManifoldCFExceptionpublic void retryDeleteNotification(JobNotifyRecord jnr, long failTime, int failCount) throws ManifoldCFException
retryDeleteNotification in interface IJobManagerjnr - is the current job notification record.failTime - is the new fail time (-1L if none).failCount - is the new fail retry count (-1 if none).ManifoldCFExceptionpublic void addDocumentsInitial(String processID, Long jobID, String[] legalLinkTypes, String[] docIDHashes, String[] docIDs, boolean overrideSchedule, int hopcountMethod, IPriorityCalculator[] documentPriorities, String[][] prereqEventNames) throws ManifoldCFException
addDocumentsInitial in interface IJobManagerprocessID - is the current process ID.jobID - is the job identifier.legalLinkTypes - is the set of legal link types that this connector generates.docIDs - are the local document identifiers.overrideSchedule - is true if any existing document schedule should be overridden.hopcountMethod - is either accurate, nodelete, or neverdelete.documentPriorities - are the document priorities corresponding to the document identifiers.prereqEventNames - are the events that must be completed before each document can be processed.docIDHashes - are the hashes of the local document identifiers (primary key).ManifoldCFExceptionpublic void addRemainingDocumentsInitial(String processID, Long jobID, String[] legalLinkTypes, String[] docIDHashes, int hopcountMethod) throws ManifoldCFException
addRemainingDocumentsInitial in interface IJobManagerprocessID - is the current process ID.jobID - is the job identifier.legalLinkTypes - is the set of legal link types that this connector generates.docIDHashes - are the local document identifier hashes.hopcountMethod - is either accurate, nodelete, or neverdelete.ManifoldCFExceptionpublic void doneDocumentsInitial(Long jobID, String[] legalLinkTypes, boolean isPartial, int hopcountMethod) throws ManifoldCFException
doneDocumentsInitial in interface IJobManagerjobID - is the job identifier.legalLinkTypes - is the set of legal link types that this connector generates.isPartial - is set if the seeds provided are only a partial list. Some connectors cannot
supply a full list of seeds on every seeding iteration; this acknowledges that limitation.hopcountMethod - describes how to handle deletions for hopcount purposes.ManifoldCFExceptionpublic boolean[] findHopCounts(Long jobID, String[] legalLinkTypes, String[] docIDHashes, String linkType, int limit, int hopcountMethod) throws ManifoldCFException
findHopCounts in interface IJobManagerjobID - is the job identifier.legalLinkTypes - is the set of legal link types that this connector generates.docIDHashes - are the hashes for the set of documents to find the hopcount for.linkType - is the kind of link to find the hopcount for.limit - is the limit, beyond which a negative distance may be returned.hopcountMethod - is the method for managing hopcounts that is in effect.ManifoldCFExceptionpublic String[] getAllSeeds(Long jobID) throws ManifoldCFException
getAllSeeds in interface IJobManagerjobID - is the job identifier.ManifoldCFExceptionpublic void addDocuments(String processID, Long jobID, String[] legalLinkTypes, String[] docIDHashes, String[] docIDs, String parentIdentifierHash, String relationshipType, int hopcountMethod, String[][] dataNames, Object[][][] dataValues, IPriorityCalculator[] documentPriorities, String[][] prereqEventNames) throws ManifoldCFException
addDocuments in interface IJobManagerprocessID - is the process ID.jobID - is the job identifier.legalLinkTypes - is the set of legal link types that this connector generates.docIDHashes - are the local document identifier hashes.parentIdentifierHash - is the optional parent identifier hash of this document. Pass null if none.
MUST be present in the case of carrydown information.relationshipType - is the optional link type between this document and its parent. Pass null if there
is no relationship with a parent.hopcountMethod - is the desired method for managing hopcounts.dataNames - are the names of the data to carry down to the child from this parent.dataValues - are the values to carry down to the child from this parent, corresponding to dataNames above. If CharacterInput objects are passed in here,
it is the caller's responsibility to clean these up.currentTime - is the time in milliseconds since epoch that will be recorded for this operation.documentPriorities - are the desired document priorities for the documents.prereqEventNames - are the events that must be completed before a document can be queued.ManifoldCFExceptionpublic void addDocument(String processID, Long jobID, String[] legalLinkTypes, String docIDHash, String docID, String parentIdentifierHash, String relationshipType, int hopcountMethod, String[] dataNames, Object[][] dataValues, IPriorityCalculator priority, String[] prereqEventNames) throws ManifoldCFException
addDocument in interface IJobManagerprocessID - is the process ID.jobID - is the job identifier.legalLinkTypes - is the set of legal link types that this connector generates.docIDHash - is the local document identifier hash value.parentIdentifierHash - is the optional parent identifier hash of this document. Pass null if none.
MUST be present in the case of carrydown information.relationshipType - is the optional link type between this document and its parent. Pass null if there
is no relationship with a parent.hopcountMethod - is the desired method for managing hopcounts.dataNames - are the names of the data to carry down to the child from this parent.dataValues - are the values to carry down to the child from this parent, corresponding to dataNames above.priority - is the desired document priority for the document.prereqEventNames - are the events that must be completed before the document can be processed.ManifoldCFExceptionpublic void revertDocuments(Long jobID, String[] legalLinkTypes, String[] parentIdentifierHashes) throws ManifoldCFException
revertDocuments in interface IJobManagerjobID - is the job identifier.legalLinkTypes - is the set of legal link types that this connector generates.parentIdentifierHashes - are the hashes of the document identifiers for whom child link extraction just took place.ManifoldCFExceptionpublic DocumentDescription[] finishDocuments(Long jobID, String[] legalLinkTypes, String[] parentIdentifierHashes, int hopcountMethod) throws ManifoldCFException
finishDocuments in interface IJobManagerjobID - is the job identifier.legalLinkTypes - is the set of legal link types that this connector generates.parentIdentifierHashes - are the document identifier hashes for whom child link extraction just took place.hopcountMethod - describes how to handle deletions for hopcount purposes.ManifoldCFExceptionprotected DocumentDescription[] calculateAffectedRestoreCarrydownChildren(Long jobID, String[] parentIDHashes) throws ManifoldCFException
ManifoldCFExceptionprotected void processParentHashSet(Long jobID, HashMap resultHash, ArrayList list) throws ManifoldCFException
ManifoldCFExceptionpublic boolean beginEventSequence(String processID, String eventName) throws ManifoldCFException
beginEventSequence in interface IJobManagerprocessID - is the current process ID.eventName - is the name of the event.ManifoldCFExceptionpublic void completeEventSequence(String eventName) throws ManifoldCFException
completeEventSequence in interface IJobManagereventName - is the name of the event.ManifoldCFExceptionpublic void carrydownChangeDocumentMultiple(DocumentDescription[] documentDescriptions, IPriorityCalculator[] docPriorities) throws ManifoldCFException
carrydownChangeDocumentMultiple in interface IJobManagerdocumentDescriptions - is the set of description objects for the documents that have had their parent carrydown information changed.docPriorities - are the document priorities to assign to the documents, if needed.ManifoldCFExceptionpublic void carrydownChangeDocument(DocumentDescription documentDescription, IPriorityCalculator docPriority) throws ManifoldCFException
carrydownChangeDocument in interface IJobManagerdocumentDescription - is the description object for the document that has had its parent carrydown information changed.docPriority - is the document priority to assign to the document, if needed.ManifoldCFExceptionprotected long getRandomAmount()
protected void sleepFor(long amt)
throws ManifoldCFException
ManifoldCFExceptionpublic String[] retrieveParentData(Long jobID, String docIDHash, String dataName) throws ManifoldCFException
retrieveParentData in interface IJobManagerjobID - is the job identifier.docIDHash - is the document identifier hash value.dataName - is the kind of data to retrieve.ManifoldCFExceptionpublic CharacterInput[] retrieveParentDataAsFiles(Long jobID, String docIDHash, String dataName) throws ManifoldCFException
retrieveParentDataAsFiles in interface IJobManagerjobID - is the job identifier.docIDHash - is the document identifier hash value.dataName - is the kind of data to retrieve.ManifoldCFExceptionpublic void startJobs(long currentTime,
List<Long> unwaitList)
throws ManifoldCFException
startJobs in interface IJobManagercurrentTime - is the current time in milliseconds since epoch.unwaitList - is filled in with the set of job ID objects that were resumed.ManifoldCFExceptionpublic void waitJobs(long currentTime,
List<Long> waitList)
throws ManifoldCFException
waitJobs in interface IJobManagercurrentTime - is the current time in milliseconds since epoch.waitList - is filled in with the set of job ID's that were put into a wait state.ManifoldCFExceptionpublic void resetJobSchedule(Long jobID) throws ManifoldCFException
resetJobSchedule in interface IJobManagerjobID - is the job identifier.ManifoldCFExceptionprotected static Long checkTimeMatch(long startTime, long currentTimestamp, EnumeratedValues daysOfWeek, EnumeratedValues daysOfMonth, EnumeratedValues months, EnumeratedValues years, EnumeratedValues hours, EnumeratedValues minutes, String timezone, Long duration)
startTime - is the start time.currentTimestamp - is the end time.daysOfWeek - is the enumerated days of the week, or null.daysOfMonth - is the enumerated days of the month, or null.months - is the enumerated months, or null.years - is the enumerated years, or null.hours - is the enumerated hours, or null.minutes - is the enumerated minutes, or null.public void manualStart(Long jobID) throws ManifoldCFException
manualStart in interface IJobManagerjobID - is the ID of the job to start.ManifoldCFExceptionpublic void manualStart(Long jobID, boolean requestMinimum) throws ManifoldCFException
manualStart in interface IJobManagerjobID - is the ID of the job to start.requestMinimum - is true if a minimal job run is requested.ManifoldCFExceptionpublic void noteJobDeleteStarted(Long jobID, long startTime) throws ManifoldCFException
noteJobDeleteStarted in interface IJobManagerjobID - is the job id.startTime - is the job delete start time.ManifoldCFExceptionpublic void noteJobStarted(Long jobID, long startTime, String seedingVersion) throws ManifoldCFException
noteJobStarted in interface IJobManagerjobID - is the job id.startTime - is the job start time.seedingVersion - is the seeding version to record with the job start.ManifoldCFExceptionpublic void noteJobSeeded(Long jobID, String seedingVersion) throws ManifoldCFException
noteJobSeeded in interface IJobManagerjobID - is the job id.seedingVersion - is the job seeding version string to record.ManifoldCFExceptionpublic void prepareDeleteScan(Long jobID) throws ManifoldCFException
prepareDeleteScan in interface IJobManagerjobID - is the job id.ManifoldCFExceptionpublic void prepareJobScan(Long jobID, String[] legalLinkTypes, int hopcountMethod, int connectorModel, boolean continuousJob, boolean fromBeginningOfTime, boolean requestMinimum) throws ManifoldCFException
prepareJobScan in interface IJobManagerjobID - is the job id.legalLinkTypes - are the link types allowed for the job.hopcountMethod - describes how to handle deletions for hopcount purposes.connectorModel - is the model used by the connector for the job.continuousJob - is true if the job is a continuous one.fromBeginningOfTime - is true if the job is running starting from time 0.requestMinimum - is true if the minimal amount of work is requested for the job run.ManifoldCFExceptionprotected void queueAllExisting(Long jobID, String[] legalLinkTypes) throws ManifoldCFException
jobID - is the job id.legalLinkTypes - are the link types allowed for the job.ManifoldCFExceptionprotected void prepareFullScan(Long jobID, String[] legalLinkTypes, int hopcountMethod) throws ManifoldCFException
jobID - is the job id.legalLinkTypes - are the link types allowed for the job.hopcountMethod - describes how to handle deletions for hopcount purposes.ManifoldCFExceptionpublic void manualAbort(Long jobID) throws ManifoldCFException
manualAbort in interface IJobManagerjobID - is the job to abort.ManifoldCFExceptionpublic void manualAbortRestart(Long jobID, boolean requestMinimum) throws ManifoldCFException
manualAbortRestart in interface IJobManagerjobID - is the job to abort.requestMinimum - is true if a minimal job run is requested.ManifoldCFExceptionpublic void manualAbortRestart(Long jobID) throws ManifoldCFException
manualAbortRestart in interface IJobManagerjobID - is the job to abort.ManifoldCFExceptionpublic boolean errorAbort(Long jobID, String errorText) throws ManifoldCFException
errorAbort in interface IJobManagerjobID - is the job to abort.errorText - is the error text.ManifoldCFExceptionpublic void pauseJob(Long jobID) throws ManifoldCFException
pauseJob in interface IJobManagerjobID - is the job identifier to pause.ManifoldCFExceptionpublic void restartJob(Long jobID) throws ManifoldCFException
restartJob in interface IJobManagerjobID - is the job identifier to restart.ManifoldCFExceptionpublic JobSeedingRecord[] getJobsReadyForSeeding(String processID, long currentTime) throws ManifoldCFException
getJobsReadyForSeeding in interface IJobManagerprocessID - is the current process ID.currentTime - is the current time in milliseconds since epoch.ManifoldCFExceptionpublic JobDeleteRecord[] getJobsReadyForDeleteCleanup(String processID) throws ManifoldCFException
getJobsReadyForDeleteCleanup in interface IJobManagerprocessID - is the current process ID.ManifoldCFExceptionpublic JobStartRecord[] getJobsReadyForStartup(String processID) throws ManifoldCFException
getJobsReadyForStartup in interface IJobManagerprocessID - is the current process ID.ManifoldCFExceptionpublic void inactivateJob(Long jobID) throws ManifoldCFException
inactivateJob in interface IJobManagerjobID - is the ID of the job to inactivate.ManifoldCFExceptionpublic void removeJob(Long jobID) throws ManifoldCFException
removeJob in interface IJobManagerjobID - is the ID of the job to remove.ManifoldCFExceptionpublic void resetStartDeleteJob(Long jobID) throws ManifoldCFException
resetStartDeleteJob in interface IJobManagerjobID - is the job id.ManifoldCFExceptionpublic void resetNotifyJob(Long jobID) throws ManifoldCFException
resetNotifyJob in interface IJobManagerjobID - is the job id.ManifoldCFExceptionpublic void resetDeleteNotifyJob(Long jobID) throws ManifoldCFException
resetDeleteNotifyJob in interface IJobManagerjobID - is the job id.ManifoldCFExceptionpublic void resetStartupJob(Long jobID) throws ManifoldCFException
resetStartupJob in interface IJobManagerjobID - is the job id.ManifoldCFExceptionpublic void resetSeedJob(Long jobID) throws ManifoldCFException
resetSeedJob in interface IJobManagerjobID - is the job id.ManifoldCFExceptionpublic void deleteJobsReadyForDelete()
throws ManifoldCFException
deleteJobsReadyForDelete in interface IJobManagerManifoldCFExceptionpublic void finishJobs()
throws ManifoldCFException
finishJobs in interface IJobManagerManifoldCFExceptionpublic JobNotifyRecord[] getJobsReadyForInactivity(String processID) throws ManifoldCFException
getJobsReadyForInactivity in interface IJobManagerprocessID - is the process ID.ManifoldCFExceptionpublic JobNotifyRecord[] getJobsReadyForDelete(String processID) throws ManifoldCFException
getJobsReadyForDelete in interface IJobManagerprocessID - is the process ID.ManifoldCFExceptionpublic void finishJobResumes(long timestamp,
List<IJobDescription> modifiedJobs)
throws ManifoldCFException
finishJobResumes in interface IJobManagertimestamp - is the current time in milliseconds since epoch.modifiedJobs - is filled in with the set of IJobDescription objects that were resumed.ManifoldCFExceptionpublic void finishJobStops(long timestamp,
List<IJobDescription> modifiedJobs,
List<Integer> stopNotificationTypes)
throws ManifoldCFException
finishJobStops in interface IJobManagertimestamp - is the current time in milliseconds since epoch.modifiedJobs - is filled in with the set of IJobDescription objects that were stopped.stopNotificationTypes - is filled in with the type of stop notification.ManifoldCFExceptionprotected static Integer mapToNotificationType(int jobStatus, boolean noErrorText)
protected void noDocPriorities(Long jobID) throws ManifoldCFException
ManifoldCFExceptionpublic void resetJobs(long currentTime,
List<IJobDescription> resetJobs)
throws ManifoldCFException
resetJobs in interface IJobManagercurrentTime - is the current time in milliseconds since epoch.resetJobs - is filled in with the set of IJobDescription objects that were reset.ManifoldCFExceptionpublic JobStatus getStatus(Long jobID) throws ManifoldCFException
getStatus in interface IJobManagerjobID - is the job ID.ManifoldCFExceptionpublic JobStatus[] getAllStatus() throws ManifoldCFException
getAllStatus in interface IJobManagerManifoldCFExceptionpublic JobStatus[] getRunningJobs() throws ManifoldCFException
getRunningJobs in interface IJobManagerManifoldCFExceptionpublic JobStatus[] getFinishedJobs() throws ManifoldCFException
getFinishedJobs in interface IJobManagerManifoldCFExceptionpublic JobStatus getStatus(Long jobID, boolean includeCounts) throws ManifoldCFException
getStatus in interface IJobManagerjobID - is the job ID.includeCounts - is true if document counts should be included.ManifoldCFExceptionpublic JobStatus[] getAllStatus(boolean includeCounts) throws ManifoldCFException
getAllStatus in interface IJobManagerincludeCounts - is true if document counts should be included.ManifoldCFExceptionpublic JobStatus[] getRunningJobs(boolean includeCounts) throws ManifoldCFException
getRunningJobs in interface IJobManagerincludeCounts - is true if document counts should be included.ManifoldCFExceptionpublic JobStatus[] getFinishedJobs(boolean includeCounts) throws ManifoldCFException
getFinishedJobs in interface IJobManagerincludeCounts - is true if document counts should be included.ManifoldCFExceptionpublic JobStatus getStatus(Long jobID, boolean includeCounts, int maxCount) throws ManifoldCFException
getStatus in interface IJobManagerincludeCounts - is true if document counts should be included.jobID - is the job ID.maxCount - is the maximum number of documents we want to count for each status.ManifoldCFExceptionpublic JobStatus[] getAllStatus(boolean includeCounts, int maxCount) throws ManifoldCFException
getAllStatus in interface IJobManagerincludeCounts - is true if document counts should be included.maxCount - is the maximum number of documents we want to count for each status.ManifoldCFExceptionpublic JobStatus[] getRunningJobs(boolean includeCounts, int maxCount) throws ManifoldCFException
getRunningJobs in interface IJobManagerincludeCounts - is true if document counts should be included.maxCount - is the maximum number of documents we want to count for each status.ManifoldCFExceptionpublic JobStatus[] getFinishedJobs(boolean includeCounts, int maxCount) throws ManifoldCFException
getFinishedJobs in interface IJobManagerincludeCounts - is true if document counts should be included.maxCount - is the maximum number of documents we want to count for each status.ManifoldCFExceptionprotected JobStatus[] makeJobStatus(String whereClause, ArrayList whereParams, boolean includeCounts, int maxCount) throws ManifoldCFException
whereClause - is the where clause for the jobs we are interested in.ManifoldCFExceptionprotected static ClauseDescription buildOutstandingClause() throws ManifoldCFException
ManifoldCFExceptionprotected static ClauseDescription buildProcessedClause() throws ManifoldCFException
ManifoldCFExceptionprotected void buildCountsUsingIndividualQueries(String whereClause, ArrayList whereParams, int maxCount, Map<Long,Long> set2Hash, Map<Long,Long> set3Hash, Map<Long,Long> set4Hash, Map<Long,Boolean> set2Exact, Map<Long,Boolean> set3Exact, Map<Long,Boolean> set4Exact) throws ManifoldCFException
ManifoldCFExceptionprotected void buildCountsUsingGroupBy(String whereClause, ArrayList whereParams, Map<Long,Long> set2Hash, Map<Long,Long> set3Hash, Map<Long,Long> set4Hash, Map<Long,Boolean> set2Exact, Map<Long,Boolean> set3Exact, Map<Long,Boolean> set4Exact) throws ManifoldCFException
ManifoldCFExceptionprotected void addWhereClause(StringBuilder sb, ArrayList list, String whereClause, ArrayList whereParams, boolean wherePresent)
public IResultSet genDocumentStatus(String connectionName, StatusFilterCriteria filterCriteria, SortOrder sortOrder, int startRow, int rowCount) throws ManifoldCFException
genDocumentStatus in interface IJobManagerconnectionName - is the name of the connection.filterCriteria - are the criteria used to limit the records considered for the report.sortOrder - is the specified sort order of the final report.startRow - is the first row to include.rowCount - is the number of rows to include.ManifoldCFExceptionpublic IResultSet genQueueStatus(String connectionName, StatusFilterCriteria filterCriteria, SortOrder sortOrder, BucketDescription idBucketDescription, int startRow, int rowCount) throws ManifoldCFException
genQueueStatus in interface IJobManagerconnectionName - is the name of the connection.filterCriteria - are the criteria used to limit the records considered for the report.sortOrder - is the specified sort order of the final report.idBucketDescription - is the bucket description for generating the identifier class.startRow - is the first row to include.rowCount - is the number of rows to include.ManifoldCFExceptionprotected void addBucketExtract(StringBuilder sb, ArrayList list, String columnPrefix, String columnName, BucketDescription bucketDesc)
protected boolean addCriteria(StringBuilder sb, ArrayList list, String fieldPrefix, String connectionName, StatusFilterCriteria criteria, boolean whereEmitted) throws ManifoldCFException
ManifoldCFExceptionprotected boolean emitClauseStart(StringBuilder sb, boolean whereEmitted)
protected void addOrdering(StringBuilder sb, String[] completeFieldList, SortOrder sort)
protected void addLimits(StringBuilder sb, int startRow, int maxRowCount)