Installation Guide

The installation steps assume that you have the following requirements installed:

• Python 2.7+ or 3.4+

• pip and setuptools Python packages. Nowadays pip requires and installs setuptools if not installed.

You can install new_frontera using pip.

To install using pip:

pip install new_frontera[option1,option2,...optionN]

Options

Each option installs dependencies needed for particular functionality.

• sql - relational database,

• graphs - Graph Manager,

• logging - color logging,

• tldextract - can be used with :setting:`TLDEXTRACT_DOMAIN_INFO`

• hbase - HBase distributed backend,

• zeromq - ZeroMQ message bus,

• kafka - Kafka message bus,

• distributed - workers dependencies.

• s3 - dependencies required for seeds addition from S3 share,

• redis - RedisBackend dependencies,

• strategies - built-in crawling strategy dependencies.