feat: Add AiCrawler with AI-powered HTML extraction

[Mantisus]

Description

• Adds AiCrawler - a new HTTP crawler that parses pages with parsel and uses pydantic-ai as the layer for LLM interaction.
• AiHtmlDistiller is a protocol for distillers that clean HTML and convert it to a compact format (e.g., cleaned HTML, Markdown) for an LLM.
  • AiCleanHtmlDistiller removes comments, noisy attributes, and scripts, returning a compact HTML version.
  • AiSkeletonDistiller extends AiCleanHtmlDistiller by truncating text and collapsing repeated siblings.
• AiHtmlExtractor is a protocol for extractors that turn a page into structured data using a distiller and an LLM.
  • AiDirectExtractor sends the distilled page to an LLM together with a Pydantic schema describing the target data and returns the validated result.
  • AiSelectorExtractor asks the LLM for CSS selectors once and caches them in a KeyValueStore, so later pages are extracted without an LLM call.

Issues

• Closes: Add support for AI/LLM-based HTML parsing (selectors) #1593

Testing

• Added new unit tests for AiCrawler, AiCleanHtmlDistiller, AiSkeletonDistiller, AiDirectExtractor, and AiSelectorExtractor.