Crawlab Python SDK

Python SDK for Crawlab

Installation

pip install crawlab-sdk

Usage

CLI

crawlab-cli

Scrapy Integration

In settings.py, add the following:

ITEM_PIPELINES = {
    'crawlab.CrawlabPipeline': 300
}

Save Scraped Items

from crawlab import save_item

scraped_items = [
    {
        'name': 'item1',
        'value': 'value1'
    },
    {
        'name': 'item2',
        'value': 'value2'
    }
]

for item in scraped_items:
    save_item(item) 

Development

Pre-requisites

pip install poetry

Install dependencies

poetry install