Performance and limits

Read this before your first full extract. It is the difference between a load measured in minutes and one measured in hours.

What a real extract costs

Measured against a tenant holding roughly 196,000 tickets:

A full ticket row is roughly 10 KB, so a 500-row page is about 5 MB, and a complete unprojected tenant load runs into gigabytes. With a sensible projection it is a fraction of that.

At 47 seconds, nightly loads are comfortable and hourly ones are entirely feasible.

What actually helps

Use $select. This matters more than everything else on this page combined. Beyond the saving in bytes, projecting down to the columns you need lets the database eliminate work, which measurably shortens every page.

Ask for $count once, not per page. It scans the whole dataset each time.

Split very large loads by event. $filter=FaireventInternalId eq <id> cuts total work further and parallelises naturally — one worker per event.

Reuse the token. It is valid for an hour, longer than most extracts take.

Hard limits

There being no rate limit is not an invitation to run twenty parallel extracts — you are reading a live production system that is also selling tickets. Two to four parallel workers, split by event, is a reasonable ceiling.

Incremental loads

There is no delta mode yet, and the four datasets differ in what they offer:

Where a watermark exists you can filter on it and fetch only what changed, which is worth doing for TicketUsages — it is the dataset that grows fastest during an event.

The limitation that applies to any watermark approach: a hard-deleted row leaves nothing behind to filter on, so it will not appear as a change. Periodic full reloads remain the only way to catch deletions. Given the 47-second figure above, a nightly full reload of Tickets is usually the simpler and safer design. Building the extract shows both patterns.