If a scraper breaks every week, I first separate source changes, extraction failures and delivery failures. A redesign may need a parser fix; an empty dataset may reflect a real change at the source. Neither diagnosis follows from an HTTP 200 or a completed job alone.
Before commissioning ongoing maintenance, agree which sources, fields and delivery windows matter. The useful outcome is an operating agreement: someone can detect a problem, contain its effects, repair the relevant component and show that the data is usable again. No maintenance plan can guarantee that an external website will remain available or unchanged.
Diagnose the failure before replacing the scraper
Start with a failed run and a comparable successful run. Record source URL, capture time, parser version, expected fields, actual output and downstream status. Retain only the evidence needed for diagnosis, with agreed access and retention rules.
- Navigation: did the job reach the expected page and pagination state?
- Extraction: are the required fields present, or did the selector match something else?
- Meaning: did currency, units, date format or the definition of a field change?
- Delivery: was valid output rejected, delayed or written twice after a retry?
Prefer an official API, export or suitable data provider when it covers the need. If a source becomes inaccessible or changes its access conditions, pause and reassess it; more retries do not resolve that decision.
Keep parser repairs separate from collection and delivery
Keep navigation, extraction, normalization and storage behind clear boundaries. Prefer stable labels and repeated structures over long chains of generated CSS classes, then version the selector rules. The Puppeteer architecture guide develops those implementation boundaries.
capture -> extract -> validate -> stage -> deliver
|
+-> quarantine and reviewSave permitted, sanitized examples with expected output so a parser change can be checked without writing to production. For a Scrapy implementation, spider contracts can check callback output counts and required fields. Those checks do not establish that a price is correct or a downstream write succeeded.
A temporary timeout may justify a bounded retry. A repeatable parser mismatch needs investigation. The queues, workers and retries guide covers execution design; maintenance defines who acts when a source stops meeting its data contract.
What a scraper maintenance agreement should cover
Use a named source inventory and separate a repair to agreed behavior from a request for new behavior. This is a proposed scope checklist, not a fixed support package or a promised response time.
| Change or incident | Agree before work starts | Evidence for closure |
|---|---|---|
| Layout changes on an existing source | Which templates and fields are covered; who fixes the parser | Old and changed samples produce the agreed output |
| Empty fields or incomplete pagination | Which fields block delivery and who reviews exceptions | Expected coverage is checked against source samples |
| Source unavailable or access conditions changed | Who pauses collection, informs operations and reviews alternatives | A documented decision to resume, replace or retire the source |
| Missed delivery window | How to label stale data and whether recovery is required | Recovered records are reconciled, or the gap is explicitly accepted |
| New source, field or destination | Separate estimate and acceptance criteria | New behavior is approved before it enters routine maintenance |
Also name the business owner, technical owner and backup contact. Define support hours, escalation route and the distinction between acknowledging an incident and restoring delivery. Agree these for the actual dependency; do not assume round-the-clock coverage.
Monitor usable data, not just successful jobs
Track required-field completeness, source coverage, age of the last accepted data, rejected records and unexpected duplicates. Compare like-for-like runs: the same source, filters and collection window. A volume drop is a signal to investigate, not proof of a broken scraper.
Hypothetical example: a catalog switches from a unit price to a pack price. The selector still returns a number and every job finishes. A check against source examples and units reveals the mismatch. Pause the affected output, establish which records are suspect and agree whether the historical data can be corrected.
The Adslyfy case shows runs, statuses and screenshot evidence that operators can inspect. It illustrates operational visibility; it does not demonstrate scraper repair times or guaranteed data quality.
How to accept a scraper repair
I would agree these checks before closing an incident. They are proposed acceptance criteria, not results from a repair performed for a client.
- Reproduce the fault with a retained example and identify the affected source, parser version and time window.
- Check the fix against the changed page and previously valid examples, including missing values and pagination boundaries.
- Compare required values with their source meaning: units, currency, dates and entity identity.
- Run a limited live check where access is still permitted. Keep doubtful output quarantined.
- Reconcile delivery and recovery. A current page cannot reconstruct a historical value that was never captured.
- Record the deployed version, evidence, remaining gaps, owner and rollback or pause procedure.
For exports, agree the accepted schema and freshness using the CSV, API and dashboard delivery guide. If records feed a CRM, apply the separate CRM ownership and retry rules before replaying writes. Repairing extraction does not authorize overwriting business edits.
What drives scraper maintenance costs?
The number of sources alone is a weak estimate. Distinct page templates, change frequency, browser execution, critical fields, delivery deadlines, evidence retention and recovery effort all affect the work. Inheriting undocumented code also requires diagnosis before a recurring scope can be priced.
Separate initial stabilization, ongoing monitoring and repairs, infrastructure or vendor charges, and new development. Compare a recurring support arrangement with separately approved interventions against your delivery dependency and the team's ability to handle incidents. Neither model removes source risk.
The software quote comparison checklist helps make inclusions, exclusions and ownership explicit. Ask which source changes are covered, what triggers a new estimate and who pays for recovery. There is no defensible fixed price without that scope.
Start with one unstable source and a clear handover
My internal platforms and integrations work can start by reviewing an existing data flow. I would first identify the failure boundary, define the source contract and propose a repair or a smaller operating scope. A rewrite is only useful if the evidence supports it.
If your team depends on a scraper it cannot maintain confidently, send me the sources, required fields, delivery frequency and a sanitized failed run. That gives us a concrete basis for discussing responsibilities, acceptance and the next piece of work.