API reference · Developer guide

Process a batch of SEO scans reliably

Run a list of pages with a bounded worker pool, a submission ceiling, and checkpoints that let you resume existing jobs after an interruption.

Start with the budget and workload

Site-wide audits and content refreshes need a repeatable way to process many URLs. This starter keeps two jobs in flight by default, writes state locally, and saves each completed report separately. It accepts up to three worker slots.

PagesScan tierWorker slotsAssumed seconds per page

150 credits for a full new batch

75 minutes under these assumptions

The timing estimate uses your assumed end-to-end duration, worker slots, and the documented default scan request limit. Shared account activity, queue delays, retries, and result retrieval can increase it. This planner submits nothing.

HTTP request limits, active-job limits, queued-job limits, and credits are separate constraints. Scan submissions can enter the server queue when active slots are occupied; a full queue or a request burst can still produce a 429. Read the rate-limit reference before increasing concurrency.

Make each page a stable work item

Save this as targets.json. Give each page a unique ID that will also name its report file. Choose the tier once for this batch.

targets.json

{
  "depth": "lite",
  "pages": [
    {
      "id": "api-comparison",
      "url": "https://api.on-page.ai/best-seo-api",
      "keyword": "best seo api",
      "region": "US"
    },
    {
      "id": "copywriting",
      "url": "https://blog.on-page.ai/seo-copywriting/",
      "keyword": "seo copywriting",
      "region": "US"
    }
  ]
}

These two Lite scans have a full-batch cost of 3 credits. Standard includes internal-link and originality sections; Deep adds deeper analysis and may include speed measurements. Choose the report fields your application actually needs.

The worker fingerprints the inputs and refuses to reuse a state file with a changed page list or tier. Use a new state filename for an intentional new analysis.

Run a resumable JavaScript or Python worker

ONPAGE_SUBMIT_LIMIT defaults to zero. Set it explicitly to allow rows that may require a paid POST. The limit includes retries of uncertain submissions, even when the API may replay an existing job for free. Known job IDs can always be checked without a new submission.

Save the worker below. Use Node.js 20+. Start with the dry run to inspect the full-batch cost and the current submission ceiling.

Inspect the plan without submitting

node batch-scans.mjs targets.json batch-state.json --dry-run

batch-scans.mjs

import { readFileSync, writeFileSync, openSync, closeSync, fsyncSync, renameSync, mkdirSync, existsSync, unlinkSync } from "node:fs";
import { resolve, join } from "node:path";
import { createHash, randomUUID } from "node:crypto";

const [inputPath, stateArgument = "batch-state.json"] = process.argv.slice(2);
if (!inputPath) throw new Error("Usage: node batch-scans.mjs targets.json [state.json] [--dry-run]");
const plan = JSON.parse(readFileSync(inputPath, "utf8"));
const prices = { lite: 1.5, standard: 2, deep: 3 };
if (!Object.hasOwn(prices, plan.depth) || !Array.isArray(plan.pages) || !plan.pages.length) {
  throw new Error("Set depth to lite, standard, or deep and provide a nonempty pages array.");
}
const ids = new Set();
const pages = plan.pages.map((p) => {
  if (typeof p.id !== "string" || !/^[a-zA-Z0-9_-]{1,64}$/.test(p.id) || ids.has(p.id)) throw new Error("Page IDs must be unique and use letters, numbers, underscores, or hyphens.");
  ids.add(p.id);
  const url = new URL(p.url);
  if (!["http:", "https:"].includes(url.protocol) || url.username || url.password || typeof p.keyword !== "string" || !p.keyword.trim() || typeof p.region !== "string" || !p.region.trim()) throw new Error("Each page needs a public URL, keyword, and region.");
  return { id: p.id, body: { url: p.url.trim(), keyword: p.keyword.trim(), region: p.region.trim() } };
});
const config = { depth: plan.depth, pages };
const fingerprint = createHash("sha256").update(JSON.stringify(config)).digest("hex");
const integer = (value, fallback, max) => {
  const n = value === undefined ? fallback : Number(value);
  if (!Number.isInteger(n) || n < 0 || n > max) throw new Error("Invalid worker or submission limit.");
  return n;
};
const workers = integer(process.env.ONPAGE_WORKERS, 2, 3);
if (workers < 1) throw new Error("Use 1-3 workers.");
const submitLimit = integer(process.env.ONPAGE_SUBMIT_LIMIT, 0, pages.length);
if (process.argv.includes("--dry-run")) {
  console.log(JSON.stringify({ pages: pages.length, depth: plan.depth, fullBatchCredits: pages.length * prices[plan.depth], workers, submissionCeiling: submitLimit, invocationCreditCeiling: submitLimit * prices[plan.depth] }, null, 2));
  process.exit(0);
}
const apiKey = process.env.ONPAGE_API_KEY;
if (!apiKey) throw new Error("Set ONPAGE_API_KEY.");
const statePath = resolve(stateArgument), lockPath = statePath + ".lock";
const outputDir = statePath + ".reports";
// One process owns this state file. Remove a stale lock only after checking its PID.
const lock = openSync(lockPath, "wx", 0o600);
writeFileSync(lock, String(process.pid)); closeSync(lock);
function atomicJson(path, value) {
  const temp = path + ".tmp";
  const fd = openSync(temp, "w", 0o600);
  try { writeFileSync(fd, JSON.stringify(value, null, 2)); fsyncSync(fd); }
  finally { closeSync(fd); }
  renameSync(temp, path);
}
let stop = false;
process.on("SIGINT", () => { stop = true; });
process.on("SIGTERM", () => { stop = true; });
const sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms));
try {
  mkdirSync(outputDir, { recursive: true, mode: 0o700 });
  const state = existsSync(statePath) ? JSON.parse(readFileSync(statePath, "utf8")) : {
    fingerprint, depth: plan.depth,
    items: pages.map(p => ({ ...p, key: randomUUID(), jobId: null, submittedAt: null, status: "pending" })),
  };
  if (state.fingerprint !== fingerprint) throw new Error("The input changed. Use the original targets file, or a new state filename for a new batch.");
  const save = () => atomicJson(statePath, state);
  const update = (item, patch) => { Object.assign(item, patch); save(); };
  save();
  let submissions = 0, cursor = 0;
  async function request(path, options, deadline) {
    for (let attempt = 0; attempt < 5 && !stop; attempt++) {
      const remaining = deadline - Date.now();
      if (remaining <= 0) throw new Error("Wait limit reached; rerun to resume.");
      const res = await fetch("https://api.on-page.ai" + path, {
        ...options, headers: { Authorization: "Bearer " + apiKey, "Content-Type": "application/json", ...options?.headers },
        signal: AbortSignal.timeout(Math.max(1, Math.min(30_000, remaining))),
      });
      const body = await res.json();
      if (res.ok) return body;
      if (![429, 503].includes(res.status)) throw new Error("HTTP " + res.status + ": " + (body.error?.code ?? "REQUEST_FAILED"));
      const header = res.headers.get("Retry-After"), seconds = Number(header);
      const delay = header !== null && Number.isFinite(seconds) && seconds >= 0 ? Math.max(1, seconds) * 1000 : Math.min(30_000, 1000 * 2 ** attempt);
      if (delay >= deadline - Date.now()) throw new Error("Rate limited beyond this wait window; resume later.");
      await sleep(delay);
    }
    throw new Error("Paused after retries or interruption. Resume with the same state file.");
  }
  async function processItem(item) {
    if (["done", "failed", "cancelled"].includes(item.status)) return;
    const deadline = Date.now() + 5 * 60_000;
    try {
      if (!item.jobId) {
        if (item.submittedAt && Date.now() - item.submittedAt >= 24 * 60 * 60_000) throw new Error("Uncertain submission is outside the idempotency window. Check job history before resubmitting.");
        if (submissions >= submitLimit) return;
        submissions++;
        // Persist the key and first attempt before sending any paid request.
        update(item, { status: "submitting", submittedAt: item.submittedAt ?? Date.now(), error: null });
        const path = plan.depth === "standard" ? "/v1/scan" : "/v1/scan/" + plan.depth;
        const submissionDeadline = Math.min(deadline, item.submittedAt + 24 * 60 * 60_000);
        const accepted = await request(path, { method: "POST", headers: { "Idempotency-Key": item.key }, body: JSON.stringify(item.body) }, submissionDeadline);
        if (typeof accepted.job_id !== "string" || !accepted.job_id.startsWith("job_")) throw new Error("Submission response lacked job_id. Preserve this state file.");
        update(item, { jobId: accepted.job_id, status: "polling" });
      }
      const jobPath = "/v1/jobs/" + encodeURIComponent(item.jobId);
      while (!stop) {
        const job = await request(jobPath, {}, deadline);
        if (["failed", "cancelled"].includes(job.status)) { update(item, { status: job.status, error: job.error?.code ?? job.status }); return; }
        if (job.status === "completed") {
          const report = await request(jobPath + "/result", {}, deadline);
          const file = join(outputDir, item.id + ".json");
          atomicJson(file, report);
          update(item, { status: "done", reportFile: file, error: null });
          return;
        }
        if (!["queued", "waiting", "running", "processing"].includes(job.status)) throw new Error("Unknown job status; inspect the job before continuing.");
        await sleep(5_000);
      }
      update(item, { status: "paused", error: "Interrupted; resume with the same state file." });
    } catch (error) { update(item, { status: "paused", error: error.message }); }
  }
  async function worker() {
    while (!stop && cursor < state.items.length) {
      const item = state.items[cursor++]; await processItem(item);
    }
  }
  await Promise.all(Array.from({ length: workers }, worker));
  console.log(JSON.stringify(state.items.map(({ id, status, jobId, error }) => ({ id, status, jobId, error })), null, 2));
  if (state.items.some(item => item.status !== "done")) process.exitCode = 1;
} finally { unlinkSync(lockPath); }

Run up to two possible new scans

export ONPAGE_API_KEY="your-api-key"
export ONPAGE_WORKERS="2"
export ONPAGE_SUBMIT_LIMIT="2"
node batch-scans.mjs targets.json batch-state.json

To resume known jobs without allowing any POST submissions:

Resume existing jobs only

ONPAGE_SUBMIT_LIMIT=0 node batch-scans.mjs targets.json batch-state.json

File	Purpose
`batch-state.json`	The original request, idempotency key, first-attempt time, job ID, and progress for every page.
`batch-state.json.lock`	The process ID of the worker that owns this state file. Prevents two local processes from submitting the same batch concurrently.
`batch-state.json.reports/`	One JSON report per completed page. The response format is chosen by the API from the scan tier.

The API key stays in the environment. Each scan key is saved before submission; each job ID is saved before polling. A report is written before its item is marked done. Requests use timeouts, bounded retries, and the same key and body on submission retries.

The script exits with code 1 when any item remains pending, paused, failed, or cancelled. Inspect the printed summary and saved state; an incomplete batch does not mean successful reports were lost.

Resume from what is already known

Situation	Recovery
Interrupted with a saved job ID	Rerun with the same targets and state file. The worker polls that job instead of submitting another scan.
POST may have succeeded, but no ID was received	Within the idempotency window, explicitly allow a submission slot and retry using the saved key. Outside that window, the worker pauses for a job-history check.
429 or 503 response	The worker respects numeric Retry-After, or uses bounded exponential backoff when no usable interval is supplied. It pauses when the wait would exceed its deadline.
Failed or cancelled job	The item remains terminal and is skipped on rerun. Investigate before placing that page into an intentionally new batch.
State lock after an abrupt stop	Read its PID and confirm that worker is no longer running before removing the stale lock. Never remove a lock owned by a live worker.

Ctrl+C requests a graceful pause; allow in-flight requests to return so their job IDs can be saved. Keep state and reports on persistent storage. This starter handles process interruptions with atomic file replacements; it is not a distributed queue or a guarantee against storage loss.

Use webhooks when the integration grows

A larger application can replace polling with verified completion webhooks. Save submitted job IDs in a database, durably enqueue a completion event, then retrieve and store the report in a worker.

Associate each submitted job with its batch item and client account.
Verify the signature and delivery timestamp before accepting the event.
Persist the event and acknowledge it promptly.
Retrieve the report using the owning account's API key.
Use a unique key for the downstream action so retries cannot create duplicate reports or client tasks.

Keep a reconciliation job that checks submitted jobs with no recorded completion. A missed webhook should not leave a batch permanently unfinished. Polling and webhook workers must update the same durable job record.

Questions before you scale

Can two workers share this state file?

The worker pool inside one process shares a checkpoint safely. Two independent processes cannot share the file: the lock prevents it. For multiple hosts, move the state into a transactional database with leases or a durable queue, and retain unique idempotency keys per logical scan.

Does the submission ceiling guarantee my account's total spend?

It limits possible new scans initiated by this invocation. Other processes, integrations, or users can spend credits independently. The dry-run estimate is a planning aid; account-wide controls belong in your application and billing setup.

What if a job takes more than five minutes?

The worker pauses that item and preserves its ID. Rerun to open another polling window. The API job can continue after the local wait ends.

Why not submit every URL immediately?

A bounded pool limits queued work and makes progress easier to recover. It also leaves account capacity for other integrations. Increase throughput based on measured behavior and your account's limits, rather than treating an HTTP request allowance as the number of jobs that can finish each minute.

How do I turn the saved reports into something useful?

Feed each report into the entity-coverage parser, the information-gain research queue, or the client PDF generator. Keep the page's job ID with the derived output so it can be traced to its analysis.