Scraper Plugin

The Scraper plugin stores rules for web scraping and gives them to a client through the RPC plugin. A JavaScript client in the repository applies the rules to a page and returns the collected data.

Features

A scraper is a named set of actions for one type of page, for example a LinkedIn profile.
An action finds an element and does one operation: get, click, wait, scrollTo or scrollBottom.
Actions make a tree. A child of a get or wait action works inside the element of its parent action. A child of a click, scrollTo or scrollBottom action searches the full document.
RPC methods read the scrapers, read the action tree, and change or remove an action.
A backend page shows the actions as a DGS table.

Requirements

Jimbo as the system plugin of the backend.
The RPC plugin. It publishes the methods that have the @rpc tag.
The Theme package. The plugin class extends Festi\Theme\Plugin\DisplayPlugin.
A MySQL database. The repository has install dumps only for MySQL.
jQuery in the page where the JavaScript client runs.

The repository has no init.php. The class loader of the project must find scraper\ActionValuesObject in domain/ActionValuesObject.php.

Installation

Install the plugin with the Festi CLI:

./vendor/bin/festi-plugin --mode install --dump ./dumps/ \
    --plugin Scraper --repository [email protected]:FestiPlugins/php_festi_plugin_scraper.git

This command was not run for this page.

The command does these steps:

It adds the repository to the plugins directory as a git submodule.
It copies install/install.mysql.sql into the dumps directory as a new migration.
It adds the plugin to festi_plugins in the same migration.
It applies the migration.

Add --path if config.php is not in the current directory.

After the installation, call the RPC method syncRpcMethods. The RPC plugin then registers the methods of this plugin. Refer to the RPC plugin page for the procedure.

What the install folder contains

File	Content
`install/install.mysql.sql`	The tables `scrapers` and `scraper_actions`.
`install/demo.mysql.sql`	An example scraper with the identifier `linkedin` and its actions. The CLI does not apply this file.

The install dump does not add the URL rule of the backend page. The annotation of onDisplayScraperActions() declares the rule:

Plugin	Pattern	Method	Area
`Scraper`	`~^/manage/scrapers/actions/$~`	`onDisplayScraperActions`	`backend`

Do not use festi-plugin --mode url for this rule. That mode adds a new method to the plugin class, and the method is already there.

Add the rule in a migration of the project. Create the file with ./vendor/bin/festi-make-migration --dump ./dumps/. Apply it with ./vendor/bin/festi-migrate --dump ./dumps/.

Data model

Table `scrapers`

Column	Description
`name`	The name of the scraper. It is unique.
`ident`	The identifier that a client sends, for example `linkedin`. It is unique.
`url_expression`	A regular expression for the page URL. The plugin and the client do not read this column.
`status`	`active` or `disabled`.

Table `scraper_actions`

Column	Description
`id_scraper`	The scraper that owns the action.
`id_parent`	The parent action. `NULL` for an action of the first level.
`selector_type`	`css`, `xpath` or `regexp`. The client supports only `css`.
`selector`	The selector of the element. The client gives it to jQuery.
`action`	`get`, `click`, `wait`, `scrollTo` or `scrollBottom`.
`data_key`	The key of the value in the result.
`data_type`	`text`, `html`, `img` or `array`.
`data_regexp`	The client does not read this column.
`container_name`	A name for a container action. The client does not read this column.
`condition_type`	`count`, `eq`, `noteq`, `less` or `greater`.
`condition_value`	The number for the condition.
`is_required`	If set, the action gets no element when the selector finds no element. A `get` action reads no value. A `wait` action stops after 100 checks. A `click` or `scrollTo` action fails.
`note`	A free text note.
`order_index`	The position of the action in the list from `getScraperActions`.

Backend page

URL	Area	Section	Method
`/manage/scrapers/actions/`	`backend`	`none`	`onDisplayScraperActions`

The page shows the DGS scraper_actions with the actions list, insert, edit, info and remove. The annotation sets @userType user.

The repository has no DGS for the table scrapers.

RPC methods

RPC method	Parameters	Result
`getScrapers`	none	The rows of `scrapers` with the status `active`.
`getScraperActions`	`ident` (string)	The action tree of the scraper. Each action has the key `child` with its child actions.
`updateScraperAction`	`action` (object)	Saves the action through the entity manager and returns it.
`removeScraperAction`	`idAction` (int)	Removes the action and returns `true`. The child actions stay and lose their parent.

removeScraperAction throws NotFoundException if the action is not in the table.

The object for updateScraperAction uses these keys: id, id_scraper, id_parent, action, selector_type, selector, data_key, data_type, data_regexp, container_name, condition_type, condition_value, is_required, note, order_index, modifiers.

Example of a request:

{"jsonrpc":"2.0", "method":"getScraperActions", "params":["linkedin"], "id":1}

Authentication

The RPC plugin reads the access token from the HTTP header X-Authorization. Use this header in production.

The RPC plugin also accepts the token in the GET parameter token. Use the GET parameter only for development. A token in a URL can stay in server logs and in the browser history.

JavaScript client

The client is in static/js/. The plugin does not add these files to a page. Load them in this order:

action/ActionError.js
action/AbstractAction.js
action/ClickAction.js, action/GetAction.js, action/ScrollBottomAction.js, action/ScrollToAction.js, action/WaitAction.js
scraper.js

The example uses the jQuery JSON-RPC client from the RPC plugin page:

let rpc = new jQuery.JsonRpcClient({
    ajaxUrl: 'https://' + request['host'] + '/',
    headers: { 'X-Authorization': request['token'] }
});

rpc.call(
    'getScraperActions', ['linkedin'],
    function (response) {
        let scraper = new Scraper();

        scraper.parse(response).then(function (result) {
            console.log("RESULT", result);
            console.log("ERRORS", scraper.getErrors());
        });
    },
    function (error) {
        console.error("RPC::getScraperActions", error);
    }
);

This example was not run for this page.

Scraper.parse(actions) applies the action tree to the current document and returns a promise with the result. Scraper.getErrors() returns the error messages with the action identifier as the key.

Actions

Action	Operation
`get`	Finds the element and reads its value.
`click`	Clicks the element.
`wait`	Looks for the element each 300 ms. It stops when the element is there or after 100 checks.
`scrollTo`	Scrolls the page to the element.
`scrollBottom`	Scrolls the page to the bottom.

Values of the `get` action

`data_key`	`data_type`	Result
set	`text`	The trimmed text of the element in `result[data_key]`.
set	`html`	The inner HTML of the element in `result[data_key]`.
set	`img`	The attribute `src` of the element in `result[data_key]`.
set	`array`	A list in `result[data_key]`. The client applies the child actions to each found element and adds one item for each element.
empty	empty	No value. The element is the container for the child actions.

An action with text, html or img writes its value only if it has no child actions.

Conditions

The client compares the number of found elements with condition_value:

`condition_type`	The action continues if the number of elements is
`count`, `eq`	equal to the value
`noteq`	not equal to the value
`less`	less than the value
`greater`	more than the value

Events

The plugin has no events.

A headless browser can open a page and run the client in it. The repository has no code for these tools:

Known limitations

The DGS select for action has the options scrollto, scrollbottom and exist. The table and the client have scrollTo and scrollBottom, and they have no exist.
ActionValuesObject has the column modifiers. The install dump does not create this column.
The client supports only the selector type css.
getScraperActions also returns the actions of a scraper with the status disabled.