Integrating Vendor Portals With No Public API (Playwright + Postgres)

TL;DR Many e-commerce and quick-commerce vendor portals expose sales, settlements and inventory only through a logged-in web UI. Each connector drives a real browser with Playwright to get through login and CAPTCHA, then runs fetch() inside the authenticated page against the portal's own report endpoints — the same requests its dashboard makes. A workflow routes each source to its handler, which normalises and upserts rows into Postgres. About 40 platforms are covered this way, and on a real remittance sample every payment reconciled to the rupee.

01 · The problemThe integration was a person with a spreadsheet

Brands that sell on marketplaces and quick-commerce apps live in their vendor portals: sales, settlements, returns, deductions, inventory. Very few of those portals offer a public API.

So the "integration" was a person. Log in to each portal, click through to the right report, download a CSV, clean it up, paste it into a spreadsheet. Every day, for every platform, for every brand.

That process is slow, error-prone and scales linearly with the number of platforms — which is exactly the number that keeps growing.

02 · The intuitionThe dashboard already is an API

Every chart and export button on a portal is backed by a network request the browser makes on the user's behalf. Open the network tab while clicking "Download report" and you'll see it: a structured request returning structured data.

So the plan became: get a legitimately authenticated browser session — with the organisation's own credentials — then make the same requests the dashboard makes, from inside that session. No scraping rendered HTML, no brittle CSS selectors standing between us and the data.

03 · The solutionFive steps from login to Postgres

Authenticate with a real browser. Playwright drives the portal's own login flow with the organisation's stored credentials and handles CAPTCHA along the way.
Fetch from inside the page. Once logged in, the connector runs fetch() in the page context against the portal's report endpoints — found through the browser's network tab — and pulls reports as CSV with the session's cookies attached automatically.
Route to a handler. A workflow node picks the right handler for each source. Each handler knows how to parse and normalise its platform's reports.
Upsert into Postgres. Normalised rows are upserted, so re-running a sync is always safe and never duplicates data.
Extend with a migration. A new platform is a new handler plus an Alembic enum migration that registers the source.

04 · ImplementationThe details that make it hold up

The in-page fetch

The core trick is tiny. Because the request runs inside the logged-in page, the portal's own session applies — no token juggling, no cookie export.

async def pull_report(page, report_url: str) -> str:
    # Runs inside the logged-in page, so the portal's own session applies.
    return await page.evaluate(
        """async (url) => {
            const res = await fetch(url, { credentials: "include" });
            if (!res.ok) throw new Error(`HTTP ${res.status}`);
            return await res.text();
        }""",
        report_url,
    )

Simplified — the real connectors wrap this with per-platform parsing.

Credentials per organisation

Each organisation's portal logins are stored per organisation in a dedicated integrations-credentials table, so a sync always runs as the brand it's syncing for.

Hardening against the real world

Portals aren't built to be automated. Sessions expire, CAPTCHAs appear, layouts shift. Across Datavio's scraping and integration work that meant getting good at the unglamorous parts — CAPTCHA handling, proxy management and browser fingerprinting — which decide whether a connector that works on Tuesday still works on Friday.

Idempotent by design

Upserts make every run safe to repeat. If a sync fails halfway, you run it again; if a portal re-issues a report, rows update instead of duplicating.

Also built on thisThe same connectors feed the document-AI debit-note pipeline, which pulls PDFs from a vendor portal and matches every line back to an invoice.

05 · ResultsWhat changed

Before

A person logs in to each portal, downloads CSVs and stitches spreadsheets together — daily, per platform, per brand.

After

Connectors sync each portal automatically and land normalised, deduplicated rows in Postgres.

~40 platform integrations live in the connector SDK, each a self-contained handler.
On a real remittance sample, the pipeline reconciled all 38 payments — 982 invoice line items — to the rupee, including 41 invoices that had been split across multiple payments.
Clients stopped spending hours downloading and stitching reports by hand.

Those reports feed the penny-level payment reconciliation, which traces every payment back to the invoices, debit notes and TDS behind it.

06 · LessonsWhat I'd keep doing

Treat every portal as an unreliable dependency: design for expired sessions and surprise CAPTCHAs from day one.
Prefer the portal's own data endpoints over scraping rendered HTML — structured in, structured out.
Make runs idempotent. The best retry strategy is "just run it again".
One handler per source keeps a growing list of platforms from becoming one giant special case.

Wrangling data out of platforms that don't want to share it? Happy to talk shop.

Email me LinkedIn