Scheduler | Spiderman

Manage and schedule crawling tasks

Hierarchy

EventEmitter
- Scheduler

Index

Constructors

constructor

Properties

Constructors

constructor

new Scheduler(options?: SchedulerOptionsInterface): Scheduler

Overrides EventEmitter.__constructor
- Defined in src/modules/scheduler/scheduler.class.ts:21
Parameters
- Default value options: SchedulerOptionsInterface = {}
Returns Scheduler

Properties

Readonly stats

stats: Statistics = new Statistics()

Static defaultMaxListeners

defaultMaxListeners: number

Static Readonly errorMonitor

errorMonitor: keyof symbol

This symbol shall be used to install a listener for only monitoring 'error' events. Listeners installed using this symbol are called before the regular 'error' listeners are called.

Installing a listener using this symbol does not change the behavior once an 'error' event is emitted, therefore the process will still crash if no regular 'error' listener is installed.

Methods

addListener

addListener(event: string | symbol, listener: (...args: any[]) => void): this

Inherited from Scheduler.addListener
- Defined in node_modules/@types/node/events.d.ts:62
Parameters
- event: string | symbol
- listener: (...args: any[]) => void
  - - (...args: any[]): void
    - Parameters
      - Rest ...args: any[]
      Returns void
Returns this

Protected Abstract classifyData

classifyData(data: object): DataEntity

- Defined in src/modules/scheduler/scheduler.class.ts:131
Classify a data object

Parameters
- data: object
  
  Data object
Returns DataEntity

Data entity

Protected Abstract classifyUrl

classifyUrl(url: string): UrlEntity

- Defined in src/modules/scheduler/scheduler.class.ts:124
Classify a URL

Parameters
- url: string
  
  URL
Returns UrlEntity

URL entity

emit

emit(event: string | symbol, ...args: any[]): boolean

Inherited from Scheduler.emit
- Defined in node_modules/@types/node/events.d.ts:72
Parameters
- event: string | symbol
- Rest ...args: any[]
Returns boolean

eventNames

eventNames(): Array<string | symbol>

Inherited from Scheduler.eventNames
- Defined in node_modules/@types/node/events.d.ts:77
Returns Array<string | symbol>

getMaxListeners

getMaxListeners(): number

Inherited from Scheduler.getMaxListeners
- Defined in node_modules/@types/node/events.d.ts:69
Returns number

listenerCount

listenerCount(type: string | symbol): number

Inherited from Scheduler.listenerCount
- Defined in node_modules/@types/node/events.d.ts:73
Parameters
- type: string | symbol
Returns number

listeners

listeners(event: string | symbol): Function[]

Inherited from Scheduler.listeners
- Defined in node_modules/@types/node/events.d.ts:70
Parameters
- event: string | symbol
Returns Function[]

off

off(event: string | symbol, listener: (...args: any[]) => void): this

Inherited from Scheduler.off
- Defined in node_modules/@types/node/events.d.ts:66
Parameters
- event: string | symbol
- listener: (...args: any[]) => void
  - - (...args: any[]): void
    - Parameters
      - Rest ...args: any[]
      Returns void
Returns this

on

on(event: string | symbol, listener: (...args: any[]) => void): this

Inherited from Scheduler.on
- Defined in node_modules/@types/node/events.d.ts:63
Parameters
- event: string | symbol
- listener: (...args: any[]) => void
  - - (...args: any[]): void
    - Parameters
      - Rest ...args: any[]
      Returns void
Returns this

once

once(event: string | symbol, listener: (...args: any[]) => void): this

Inherited from Scheduler.once
- Defined in node_modules/@types/node/events.d.ts:64
Parameters
- event: string | symbol
- listener: (...args: any[]) => void
  - - (...args: any[]): void
    - Parameters
      - Rest ...args: any[]
      Returns void
Returns this

pause

pause(): void

- Defined in src/modules/scheduler/scheduler.class.ts:87
Pause crawling. Finish all running tasks and prevent new tasks from being added

Returns void

prependListener

prependListener(event: string | symbol, listener: (...args: any[]) => void): this

Inherited from Scheduler.prependListener
- Defined in node_modules/@types/node/events.d.ts:75
Parameters
- event: string | symbol
- listener: (...args: any[]) => void
  - - (...args: any[]): void
    - Parameters
      - Rest ...args: any[]
      Returns void
Returns this

prependOnceListener

prependOnceListener(event: string | symbol, listener: (...args: any[]) => void): this

Inherited from Scheduler.prependOnceListener
- Defined in node_modules/@types/node/events.d.ts:76
Parameters
- event: string | symbol
- listener: (...args: any[]) => void
  - - (...args: any[]): void
    - Parameters
      - Rest ...args: any[]
      Returns void
Returns this

rawListeners

rawListeners(event: string | symbol): Function[]

Inherited from Scheduler.rawListeners
- Defined in node_modules/@types/node/events.d.ts:71
Parameters
- event: string | symbol
Returns Function[]

removeAllListeners

removeAllListeners(event?: string | symbol): this

Inherited from Scheduler.removeAllListeners
- Defined in node_modules/@types/node/events.d.ts:67
Parameters
- Optional event: string | symbol
Returns this

removeListener

removeListener(event: string | symbol, listener: (...args: any[]) => void): this

Inherited from Scheduler.removeListener
- Defined in node_modules/@types/node/events.d.ts:65
Parameters
- event: string | symbol
- listener: (...args: any[]) => void
  - - (...args: any[]): void
    - Parameters
      - Rest ...args: any[]
      Returns void
Returns this

resume

resume(): void

- Defined in src/modules/scheduler/scheduler.class.ts:97
Resume crawling

Returns void

scheduleUrl

scheduleUrl(url: string, duplicateCheck?: boolean): void

- Defined in src/modules/scheduler/scheduler.class.ts:136
Schedule a URL to be scraped

Parameters
- url: string
- Default value duplicateCheck: boolean = true
Returns void

scheduleUrlEntity

scheduleUrlEntity(urlEntity: UrlEntity): void

- Defined in src/modules/scheduler/scheduler.class.ts:143
Schedule a URL entity to be scraped

Parameters
- urlEntity: UrlEntity
Returns void

setMaxListeners

setMaxListeners(n: number): this

Inherited from Scheduler.setMaxListeners
- Defined in node_modules/@types/node/events.d.ts:68
Parameters
- n: number
Returns this

start

start(initUrls?: string[]): Promise<void>

- Defined in src/modules/scheduler/scheduler.class.ts:77
Start crawling

Parameters
- Default value initUrls: string[] = []
  
  Initial URLs
Returns Promise<void>

stop

stop(gracefully?: boolean): Promise<void>

- Defined in src/modules/scheduler/scheduler.class.ts:111
Stop crawling

Parameters
- Default value gracefully: boolean = true
Returns Promise<void>

Static listenerCount

listenerCount(emitter: EventEmitter, event: string | symbol): number

Inherited from Scheduler.listenerCount
- Defined in node_modules/@types/node/events.d.ts:44
deprecated

since v4.0.0

Parameters
- emitter: EventEmitter
- event: string | symbol
Returns number

Hierarchy

Index

Constructors

Properties

Methods

Constructors

constructor

Parameters

Default value options: SchedulerOptionsInterface = {}

Returns Scheduler

Properties

Readonly stats

Static defaultMaxListeners

Static Readonly errorMonitor

Methods

addListener

Parameters

event: string | symbol

listener: (...args: any[]) => void

Parameters

Rest ...args: any[]

Returns void

Returns this

Protected Abstract classifyData

Parameters

data: object

Returns DataEntity

Protected Abstract classifyUrl

Parameters

url: string

Returns UrlEntity

emit

Parameters

event: string | symbol

Rest ...args: any[]

Returns boolean

eventNames

Returns Array<string | symbol>

getMaxListeners

Returns number

listenerCount

Parameters

type: string | symbol

Returns number

listeners

Parameters

event: string | symbol

Returns Function[]

off

Parameters

event: string | symbol

listener: (...args: any[]) => void

Parameters

Rest ...args: any[]

Returns void

Returns this

on

Parameters

event: string | symbol

listener: (...args: any[]) => void

Parameters

Rest ...args: any[]

Returns void

Returns this

once

Parameters

event: string | symbol

listener: (...args: any[]) => void

Parameters

Rest ...args: any[]

Returns void

Returns this

pause

Returns void

prependListener

Parameters

event: string | symbol

listener: (...args: any[]) => void

Parameters

Rest ...args: any[]