Ingestion scheduler¶

Requirements¶

Default implementation of the scheduler requires the following files to be executed during the connector installation:

• core.sql (See: Core SQL reference)

• configuration/app_config.sql (See: App config SQL reference)

• configuration/connector_configuration.sql (See: Connector configuration reference)

• scheduler/scheduler.sql (See: Ingestion scheduler reference)

Overview¶

The scheduler task takes care of triggering the ingestion of resources at appropriate times according to their configuration. This task is not started by the SDK itself and needs to be created and resumed, for example, during finalize configuration step. There are two ways of achieving this: using the procedure called PUBLIC.CREATE_SCHEDULER() from SQL or by calling SchedulerCreator#createScheduler() directly from the Java code.

The default implementation will create the scheduler task using the expression provided in connector_configuration, under the global_schedule key. When the default scheduler task is executed it searches for all the enabled resource ingestion definitions that have their ScheduleType in configuration set to GLOBAL and their corresponding ingestion processes. Each of the processes is then updated to IN_PROGRESS status. This status will be updated again to SCHEDULED after ingestion iteration is finished. Then for each of them OnIngestionScheduledCallback is executed. This callback can be completely custom and can be implemented using SQL or Java. The default implementation of this callback does nothing, however the SDK also provides an implementation of this callback using the Task reactor module. This implementation retrieves the data about resources from the database and puts a work item containing this data in the Task Reactor queue.

When the work item is finished another callback called OnIngestionFinishedCallback is executed. This callback changes the process state back to SCHEDULED once the ingestion is done.