On this page
We built our first internal scraping workflow on Zapier in 2022. The shape was the obvious one: a Google Form dumped a URL into a Google Sheet, a Zap watched the sheet, a Webhooks by Zapier step called our scraping endpoint, another step parsed the JSON, a final step appended the result back to the sheet. It worked for about six months. Then it stopped working, not because Zapier broke, but because our scraping needs grew past the shape Zapier is good at.
This is the honest post-mortem. Zapier is a very good tool. It is the best tool on the planet for one specific job: gluing two apps together when the schema of the glue is fixed. The moment the glue needs to think — pick which tool to call, retry intelligently, branch on intermediate results, wait a long time for a crawl — Zapier is the wrong shape and fighting it is expensive in both money and engineering time.
We replaced the scraping leg with Stekpad called over MCP from Claude, and with direct API calls from our backend. Zapier still runs for the app-to-app glue on either side. This post explains why we drew the line there, what the new flow looks like, and where Zapier still wins.
What broke, exactly
Three things went wrong at roughly the same time.
Per-task pricing got expensive. Our scraping Zap fired roughly 3,000 times a month. Each run used 4 tasks (trigger, webhook, formatter, sheet append). That is 12,000 tasks a month, which bumped us from the Professional plan at 29 €/month to the Team plan at 103 €/month. The scraping itself cost less than the glue wrapping it.
Webhooks by Zapier timed out on long crawls. A Zap step has a hard 30-second runtime limit, and a webhook response needs to return inside that window. Our crawls take 2 to 20 minutes. We tried firing the crawl, returning an empty response, and polling from a second Zap — that added two more tasks per run (now 6 tasks, 18,000 tasks a month) and the polling was flaky whenever Zapier's scheduler drifted.
There is no native scraping verb. Zapier's action catalog has "Webhooks", "Code by Zapier", and "Formatter", but there is no first-class scrape or crawl or extract. You rebuild it yourself with the primitives and you debug it when a site changes. That is fine when you have one site; we had forty.
None of this is Zapier's fault. Zapier is a glue layer, not a scraping tool. Using it as a scraping tool is like using a chef's knife as a hammer: it kind of works, and it kind of destroys the knife.
The replacement, in three lines
Here is the entire scraping leg now, in Python, in three lines:
from stekpad import Stekpad sp = Stekpad(api_key="stkpd_live_...")run = sp.scrape("https://example.com/product/42", formats=["markdown", "json"])sheets.append_row(run.json)That is it. One SDK call, one dataset row, one append to the sheet. No webhook timeout. No polling. No task counter ticking up four times per URL. The credit cost is 1 credit per scrape, or 0.0045 € on the Starter pack. For 3,000 scrapes a month, that is 13.50 € total — less than a seventh of what we were paying Zapier to glue the call.
If the crawl is long-running, the API returns a run_id immediately and either polls or fires a webhook at the end:
crawl = sp.crawl( url="https://example.com", limits={"max_pages": 500, "max_depth": 4}, scrape_options={"formats": ["markdown"]}, dataset={"type": "table", "name": "Example site crawl"}, webhook_url="https://myapp.com/hooks/stekpad", webhook_events=["run.completed"],)print(crawl.run_id) # returns in under a secondThe HTTP call returns in under a second with a run_id. The actual crawl runs in the background for however long it takes. When it finishes, Stekpad POSTs {run_id, status: "completed", dataset_id, rows_count} to myapp.com/hooks/stekpad. No 30-second timeout. No polling task loop. See the crawl API reference for the full webhook payload.
Why MCP was the right shape for the thinking part
For the scraping leg, a plain SDK call is enough. For the thinking leg — the part where a growth ops person wants to ask "can you research these 50 companies and tell me which ones are hiring for sales roles" — a plain SDK call is not enough. You need a planner. You need something that reads the prompt, decides which Stekpad verb to call, loops over the list, handles errors, and returns a clean result.
You have two options. You write the planner yourself in Python with LangChain or the Anthropic SDK, or you install an MCP server into Claude Desktop and let Claude be the planner. We tried both. MCP won for three reasons.
First, the chat IS the UI. Our growth person types the prompt in Claude Desktop, watches Claude plan and call tools, and copies the result into wherever. Zero code shipped. Zero "ops engineer" hours. She onboarded in ten minutes and has been shipping enrichment runs every day since.
Second, no canvas, no drag-and-drop. Zapier and n8n make you draw a graph. Every new workflow is a new graph you maintain. MCP workflows have no graph. You describe the goal in English, Claude picks the tools, and the "workflow" lives in the chat transcript. If the goal changes next week, you retype it. There is nothing to maintain.
Third, it survives tool additions. When we shipped two new enrichers last month (dns and whois), every MCP-based workflow got smarter automatically because Claude saw the new tools in the server's capability list and started calling them when the prompt warranted. In Zapier, we would have had to manually update every Zap that could benefit.
Pre-wired graphs rot. Agent-driven calls heal themselves when the toolset grows. We stopped maintaining graphs and started maintaining prompts.
A real example side by side
Old Zapier workflow: "When a row is added to the 'Prospects' sheet, scrape the website column, enrich with email, push to HubSpot."
- Trigger: New row in Google Sheets
- Action: Webhooks by Zapier — POST to our scraping endpoint, wait for response
- Action: Code by Zapier — parse JSON, reshape to HubSpot contact schema
- Action: HubSpot — Create or Update Contact
Four tasks per row. Webhook times out if the scrape takes longer than 30 seconds. Retry logic is manual. Error handling is "Zapier emails you when a Zap fails". Cost at 3,000 rows/month: 12,000 tasks, 103 €/month on the Team plan.
New Stekpad + Zapier split:
- Zapier trigger: New row in Google Sheets
- Zapier action: Webhooks by Zapier — POST to our own
/enrich-prospectendpoint, return immediately - Our endpoint: calls
sp.scrape(url, schema), thensp.enrich(row, pipeline), then HubSpot SDK
Zapier does 2 tasks per row (6,000 tasks/month). Our endpoint handles the scraping with the Stekpad SDK and a webhook for long crawls. Cost: Zapier Starter plan at 19 €/month + Stekpad credits around 15 €/month = 34 €/month. The webhook timeout problem is gone. The per-task bloat is gone. Retry logic is in our code where it belongs, with real observability.
What Zapier still wins
This is the honest part. Zapier is still the right tool for four jobs.
App-to-app triggers. When you need to watch Typeform, Stripe, Calendly, HubSpot, Pipedrive, Slack and 8,000 other apps for events, Zapier is unmatched. Nobody else has that catalog. We still use it for every "when a new X happens in Y" trigger.
Cross-team no-code flows. When a marketer needs to wire "new form submission → add to mailing list → DM the sales rep in Slack" with zero engineering time, Zapier is still the fastest path. Do not rebuild that in code unless you are going to do it a lot.
Transactional glue with tiny per-item cost. A workflow that runs 50 times a month and does one scrape per run fits inside Zapier's free or starter tier without pain. The per-task cost only bites at scale.
Error notifications. Zapier's "this Zap is halted" email is a genuinely good monitoring signal for non-critical flows. We still use it on five of our own Zaps.
The rule we use now: Zapier for the triggers and the glue on either side of scraping. Stekpad (via SDK or MCP) for the scraping leg itself. Neither tool tries to be the other. Both tools are better for it. See the MCP explainer for growth teams for the longer version of the "when to use which" argument.
The cost math, for one year
We ran the numbers on a year of our own usage. Take them with the usual grain of salt — your workload is different.
Old stack (Zapier as scraper + glue): 103 €/month × 12 = 1,236 €. All in one plan. Scraping and glue bundled together, neither well served.
New stack (Zapier glue + Stekpad scraping): Zapier Starter at 19 €/month × 12 = 228 € + Stekpad Builder pack at 29 € every two months (10,000 credits lasts us about eight weeks) × 6 = 174 € = 402 € total for the year.
Roughly a third of the cost, and the stack is faster, more observable, and not constantly fighting a webhook timeout. We would happily spend the 1,236 € if we had to, but there is no reason to.
Next steps
- Read the Stekpad REST API overview for the five verbs and the three surfaces.
- Install the Stekpad MCP server in Claude Desktop if you want the agent-driven flow.
- Read MCP explained for growth teams for the non-engineering-team version of this argument.
- Browse the enrichment pipeline with Claude guide for a longer worked example that combines
scrape,extractand four enrichers in one chat.