Class Scraper

Scraper class

Hierarchy

Scraper

Implements

ScraperInterface

Index

Constructors

constructor

Properties

Methods

Constructors

constructor

new Scraper(options?: ScraperOptionsInterface): Scraper

- Defined in src/modules/scraper/scraper.class.ts:19
Parameters
- Default value options: ScraperOptionsInterface = {}
Returns Scraper

Properties

Protected Readonly proxies

proxies: HttpProxy[]

Protected Readonly userAgents

userAgents: string[]

Methods

Protected Abstract parse

parse(resBody: string, meta: ParsingMeta): Promise<ParsingResult>

- Defined in src/modules/scraper/scraper.class.ts:91
Parse result from the response body

Parameters
- resBody: string
- meta: ParsingMeta
Returns Promise<ParsingResult>

Protected process

process(url: string): Promise<ParsingResult>

- Defined in src/modules/scraper/scraper.class.ts:63
Process a URL. You can override this method to change its default behavior

Parameters
- url: string
Returns Promise<ParsingResult>

run

run(url: string): Promise<ScrapingResult>

Implementation of ScraperInterface.run
- Defined in src/modules/scraper/scraper.class.ts:39
Run

Parameters
- url: string
Returns Promise<ScrapingResult>