At Pixee Play, we’ve been working for several months on Pixee Scraper, an internal R&D platform designed to streamline the extraction of supplier product catalogs (B2B) and the monitoring of competitor prices (B2C). As I write this (April 2026, v3.10), the platform is running 19 active scrapers—14 suppliers and 5 competitors—and feeds our PIM, Pixee PIM, with tens of thousands of enriched product listings. This post details the technical choices and the real obstacles we encountered: it’s an R&D update, not a marketing piece. Some issues have been resolved; others remain open.

The initial problem: 19 sites, 19 different technologies

Each supplier or marketplace has its own technical stack, its own anti-bot, and its own data structuring approach. In 2026, we manage sites on:

PrestaShop 1.6, 1.7, and 8.x (B2B and B2C, with or without login)
Sylius PHP (Symfony e-commerce framework)
Sana Commerce (JS-heavy SPA)
Joomla + VirtueMart (with Algolia API output)
Drupal 10+
Oxatis (FR, JSON-LD structured data)
Legacy ASP.NET
Proprietary REST APIs (Ubaldi Pro with JWT)
Custom Svelte SSR (TopAchat, for MPN cross-check)

The naive approach—one Python script per site—quickly leads to technical debt: code duplication, impossible updates, and no centralized monitoring. Our goal was to build a shared framework that handles 80% of the complexity, allowing each scraper to focus on the remaining 20% of specific tasks.

Architecture: a DRY framework at the heart of the project

The src/common/ provides the shared building blocks:

BasePipeline : eliminating ~170 lines of boilerplate per scraper. CLI argparse (full/incremental/resume/discover), signal handling, progress bars, state management, DB lifecycle, catalog rebuild, error tracking. 15 out of 15 scrapers inherit from it.
BaseDbStore and BaseJsonStore[T]: PostgreSQL persistence + atomic JSON, upsert on natural key. 14 out of 14 scrapers migrated.
BasePlaywrightSession, BaseHttpxSession, BaseCurlSession: network layer abstraction. Login, cookies, rate limiting, exponential retry. 12 out of 15 scrapers have migrated.
ProxyManager : 5 providers (none / static_list / tailscale / brightdata / endpoint_pool) with 4 routing strategies (round-robin, weighted, failover, least_failures).
AdaptiveRateLimiter : Dynamically adjusts the interval between requests using EWMA of HTTP response times. If the site slows down, we scale back without manual intervention. Opt-in per scraper.

The tangible benefit: adding a new scraper now takes 2 to 4 days of development, compared to 2 to 3 weeks at the start of the project.

The real battle: modern anti-bot

In 2026, scraping a major B2C site (Amazon, LDLC, ManoMano, Cdiscount) without getting blocked requires a multi-pronged approach. Here are our layered defenses.

1. Three levels of browser stealth

Playwright alone is easily detected (CDP leaks, fingerprint navigator.webdriver, missing plugins). We’ve modeled three variants, activatable by the scraper:

safe: Standard Playwright, sufficient for B2B sites with login (we’re authenticated, no need to hide).
stealth: rebrowser-playwright, a fork that patches CDP leaks (Runtime.enable, object mutations).
camoufox: Camoufox, an anti-detection Firefox fork with canvas/WebGL noise and fingerprint rotation. Slower to launch but passes Cloudflare Turnstile in cases where stealth fails.

2. TLS fingerprinting with curl_cffi

Some WAFs (Oxatis in particular) detect non-browser HTTP clients via the TLS ClientHello: order of cipher suites, extensions, ALPN. Python httpx and requests have a distinct signature from Chrome/Firefox. The workaround: curl_cffi, which impersonates the TLS fingerprint of Chrome/Firefox/Safari via a patched libcurl.

Our BaseCurlSession wraps this layer. A scraper can switch from httpx to curl_cffi by changing its parent class—everything else (proxy, cookies, rate limiter) remains the same.

3. Residential proxies and rotation

Data center IPs (AWS, OVH, Hetzner) get blocked after just a few requests on reputable sites. Our stack supports:

Tailscale as a residential exit network (routing via our physical machines)
BrightData / endpoint pools with 4 selection strategies
Automatic failover when an endpoint exceeds a failure threshold
Drag-and-drop ordering of endpoints in the dashboard (manual priority per scraper)

4. What remains open

Datadome remains our toughest challenge. Sites protected by Datadome (certain French marketplaces) still require custom solutions. This is an active R&D topic for v3.11.

The headache of changing selectors

The front-end teams of the target sites update their CSS every 3 to 6 months. A selector .product-price__value becomes [data-testid="price-amount"] overnight. Without proactive detection, your scrapers silently return empty data for days. We’ve developed two tools to address this issue.

Extraction health scoring

Each extracted product receives a quality score: completeness of critical fields (price, name, SKU), consistency of values (non-zero price, consistency with history), presence of optional fields. A rolling window calculates the average over the last N runs, and a score-drop detection alerts when a scraper falls below the historical threshold—exported as Prometheus metrics.

Selector Doctor: automatic DOM diagnosis

When a degradation is detected, a diagnostic tool automatically analyzes the target DOM:

Comparison with a reference snapshot taken when the site was healthy
Text search (search for “Price” or “€” in the DOM to locate the element)
Application of known CMS patterns (PrestaShop, Sylius, and Shopify have typical structures)
Suggestions for new CSS selectors with a confidence score

The operator receives a semi-automated PR with candidate selectors—they validate or adjust them. Resolution time for a broken site: from 2–4 hours to 15–30 minutes.

The ML-assisted creation wizard

Adding a new site involves detecting its CMS, sitemap URL structure, presence or absence of a login, and anti-bot level. We trained a small ML model on 32 analyzed sites, with 5-fold cross-validation:

CMS detection (PrestaShop / Sylius / Shopify / Drupal / Magento / custom): 55% accuracy (small dataset, room for improvement)
Session requirement detection (login required or not): 58% accuracy
Probable anti-bot score: heuristic (Cloudflare cookies, Akamai headers, reCAPTCHA)

The wizard offers a pre-filled template (7 CMS templates: PrestaShop 1.6, 1.7, Sylius, Drupal, Joomla, Oxatis, ASP.NET): selectors, login patterns, extraction strategy. The operator adjusts rather than writing everything from scratch.

Accuracy is still modest—which is to be expected with only 32 samples. Each new site analyzed contributes to the dataset. The goal isn’t perfect precision, but rather to quickly identify the right starting point.

Adaptive extraction profiles

A full crawl of a catalog of 40,000 products takes 8 to 12 hours. But we don’t need to re-extract everything every time. We’ve defined 4 profiles:

full: all data (price, specifications, images, descriptions, attached PDFs)
price_specs: price + technical specs (no images or descriptions)
price_only: price only — 3x to 5x faster execution time
auto: full on the first run, price_only thereafter (detection via DB)

In practice: the first crawl of a supplier is full, subsequent weekly runs switch to price_only to update only the prices. Specs are refreshed once a month in price_specs.

The PDF pipeline: from scraping to RAG

11 of our 15 B2B scrapers extract attached documents (datasheets, manuals, certificates, technical specifications). These PDFs are invaluable for catalog enrichment—they contain the comprehensive technical specifications that product pages summarize. The pipeline, on the Pixee PIM side, orchestrated by Celery Hourly schedule:

HTTP download of the PDF (2 retries with exponential backoff)
SHA256 deduplication: if the PDF has already been processed for another product, we reuse it
S3 upload with versioning
OCR via Docling (IBM) — PDF conversion to structured Markdown
Semantic chunking + embedding
Injection into Qdrant (vector database) for RAG
Indexing of the enriched product listing in Pixee PIM

Result: a search for a technical attribute (“IP67,” “PoE+,” “Cat 6a”) finds products whose product page did not mention it but whose datasheet contained it. This radically improves the quality of results for complex technical specifications.

The dashboard: FastAPI + HTMX, zero JS builds

The Scraper Hub is the control center: start/stop runs, real-time monitoring, cron scheduling, queue (max 6 simultaneous scrapers, auto-draining FIFO), exports (CSV, JSON, streaming Excel), CRUD proxies, settings per scraper.

Strong technical choice: FastAPI + HTMX + Jinja2 + Tailwind CDN. No React, no Vue, no JS build step. Dynamic interactions (drag-and-drop, filters persisted in sessionStorage, partial refresh) are handled in HTMX with a few lines of vanilla JS.

Why this choice? For an internal tool with a small number of users, the advantages are clear:

Minimal boilerplate: a FastAPI route that returns HTML, a template that iterates
No client/server state synchronization issues: the single source of truth is the database; HTML is generated on every request
Time-to-ship: a new page with a form + table = 30 min
No build pipeline to maintain, no Node.js version to keep up with, no npm vulnerabilities

HTMX offers an excellent balance of quality and simplicity for admin UIs.

Legality and ethics: non-negotiable

Scraping has an ethical framework. We document it in our project practices:

Respect for robots.txt: even when we could technically ignore it
Aggressive rate limiting: delays between requests, adaptive throttling, never unintentional DoS
No personal data: we extract data from public product catalogs; never from named customer reviews or personal information
GDPR compliance: the databases we populate are proprietary, with a designated DPO on the Product Manager side
Contractual relationship: for B2B sites requiring login, we systematically obtain the supplier’s authorization (it’s in their interest: we import their catalog into our PIM for mutual benefit)
B2C marketplace TOS: We monitor public prices at a reasonable frequency without fully replicating their catalog

Current figures (April 2026)

19 active scrapers (14 B2B suppliers + 5 B2C competitors)
274 automated tests (shared framework + scraper-specific tests)
17 shared modules in src/common/
~150,000 product listings in the database to date, the majority of which have been enriched by the RAG PDF pipeline
2,328 PDF assets already ingested (Docling OCR + Qdrant indexing)
3 stealth levels (safe, stealth, camoufox) with per-scraper selection
Health extraction + Selector Doctor active on all 19 scrapers

What remains in R&D

Several fronts are open, which is normal for a project at this stage:

Datadome: we do not yet have a systematic solution that bypasses this protection in a server environment. Benchmarking of commercial solutions is underway.
Semantic field detection: transitioning from static CSS selection to LLM-guided extraction (GPT-4 vision on screenshots) for highly dynamic sites. Experimental.
Cross-source consolidation: when the same product (same EAN) is found across 4 different suppliers, intelligently merge attributes with quality arbitration.
Self-healing selectors: Selector Doctor makes suggestions, but ideally it should apply them automatically after a confidence threshold is met. Architecture details still under consideration.

Conclusion

Pixee Scraper is neither a commercial product nor a demo—it is an internal R&D platform that allows us to industrialize our scraping and catalog enrichment expertise and make it available to our customers via Pixee PIM. The technical choices (async Python, multi-tier Playwright, curl_cffi TLS, HTMX dashboard, RAG pipeline) are well-considered and tested in production with real-world volumes. If you need supplier catalog extraction or structured, compliant competitive price monitoring, let’s discuss it—the foundation exists, and we adapt it to each client’s specific context. Contact Pixee Play for a technical discussion.

Pixee Scraper: R&D for a B2B/B2C multi-scraper platform in 2026