Sync

All OpenAlex data is free; keeping up to date is the paid service. Anyone can download the whole database for free in the quarterly snapshot. Getting changes as they happen is daily sync, part of the Member+ and Partner plans. The main way to get updates is the daily snapshot; for small numbers of updates, the API’s sync filters work too (why updates are paid).

This page is the reference for how the snapshot changes over time: when releases happen, what a release means, and how to keep a local copy in sync with OpenAlex, including deletions and merges.

Release cadence

PlanWhat you get
Free (everyone)New full releases of the public snapshot (s3://openalex/data/), released quarterly on the second Wednesday of January, April, July and October (UTC). Each release replaces the bucket contents in place.
Daily sync (Member+ and Partner plans)A daily full snapshot: the complete database, rebuilt and published every day (dated folders in the openalex-snapshots staging bucket). Plus the sync filters on the API.

The authoritative history of what shipped when is RELEASE_NOTES.txt, at the root of the public bucket: one dated entry per release. Each quarterly release carries a quarter’s worth of bug fixes and improvements across the database, so entries are not itemized; for individual data changes see the data documentation. Check the file (and the manifest date) rather than guessing from file timestamps.

Public release dates

The public snapshot is published automatically on the second Wednesday of January, April, July and October. The release is built from that day’s data and lands in the bucket during the day (UTC); the combined manifest’s date field shows the release date once it is live.

ReleaseDate
2026 Q4Wednesday 2026-10-14
2027 Q1Wednesday 2027-01-13
2027 Q2Wednesday 2027-04-14
2027 Q3Wednesday 2027-07-14

If a release build fails its validation checks, it is retried on the following days until it passes, so a release can occasionally land up to a week late. The bucket keeps serving the previous release in the meantime.

Note: There is no announcement feed for releases yet — watch RELEASE_NOTES.txt or the combined manifest’s date field. If a release is more than a week overdue, contact us.

What a release is: how partitions work

Records are partitioned by updated_date, and each partition holds only the records that last changed on that date. A release doesn’t add a new dated copy of the whole database — it updates the one copy in place, moving changed records into newer partitions.

Imagine launching OpenAlex with 1,000 Authors, all created on 2024-01-01:

/data/jsonl/authors/
├── manifest.json
└── updated_date=2024-01-01 [1000 Authors]

If 50 of those Authors are updated on 2024-01-15, they move out of the old partition and into the new one; 50 brand-new Authors would land in the same new partition:

/data/jsonl/authors/
├── manifest.json
├── updated_date=2024-01-01 [950 Authors]
└── updated_date=2024-01-15 [100 Authors]

Consequences:

  • The bucket is always the complete current database — the union of all partitions, exactly once per record. No partition is a “base file”; no folder is cumulative.
  • To update a copy made from the snapshot dated X, download every partition with updated_date after X and upsert its records by id. A published partition never gains records, so you never need to re-download one you already have. A record that changes after the day’s snapshot is written goes into the next day’s partition, so its updated_date can fall on the day before its partition’s date.
  • Copies from snapshots dated before 7 November 2026 need one more partition, once. Until then, authors, sources, institutions and a few smaller entities rebuilt after the day’s snapshot was written landed in that day’s partition, which then grew in the next day’s snapshot. For the first update of such a copy, download the partitions on or after X. If you already updated a copy with “after X” from a snapshot dated before 7 November 2026, download once, for every entity except works, every partition dated on or after the date of the copy you started from. Works partitions never grew.
  • A few work fields don’t move updated_date: fwci, the percentile value and cited_by_percentile_year (recomputed every night) and has_content; and a change made to every record at once is announced with what to reload (RELEASE_NOTES.txt, or the field’s own page). Common attributes has the list.
  • Old partitions shrink as their records move forward. If you sync files rather than upserting records, use --delete so vacated files disappear (see below).

Keeping in sync with OpenAlex

There are four ways to keep a copy of OpenAlex current. They differ in freshness, in how much infrastructure they demand, and in how much of the database they suit — pick by what you’re mirroring:

MethodFreshnessBest forAvailability
Public snapshotQuarterly (release dates)Full mirrors where quarterly is fresh enoughFree
Daily snapshotDailyThe main way to get updates; full-database mirrorsMember+ and Partner
API sync filtersContinuousSmall numbers of updates: keeping a subset fresh (e.g. one institution’s works) with no bulk infrastructureMember+ and Partner
Unpaywall Data FeedDaily (works only, legacy format)Existing integrations that speak the Unpaywall schemaSubscription

The two snapshot methods move the whole database in bulk; the API-filter method needs nothing but HTTP requests, which makes it the right fit when you only care about a slice of OpenAlex.

The public snapshot (free)

Re-sync from each quarterly release using the manifest (per entity or combined):

  1. Download the manifest — aws s3 cp s3://openalex/data/jsonl/works/manifest.json . --no-sign-request
  2. Find the partitions to download — every updated_date in the manifest’s file list after the date of the release you last ingested. If that release is from before 7 November 2026 (2026-10-14 or earlier), take the partitions on or after its date instead, once: its own partition gained records later (why).
  3. Download those partitions.
  4. Verify consistency — re-download the manifest; if unchanged since step 1, no records moved mid-download.
  5. Upsert by id into your store.

If you mirror files with aws s3 sync, always pass --delete — otherwise files vacated by records moving to newer partitions linger and you get duplicates. Full commands in the download recipe.

The daily snapshot (Member+ and Partner)

This is the main way to keep up to date. Member+ and Partner subscribers get the complete database, rebuilt and published every day: each day’s copy in its own dated folder in the staging bucket, partitioned by updated_date just like the public snapshot (access details).

Because every daily copy is both complete and partitioned by change date, it covers every sync rhythm:

  • Incremental sync, on your schedule. Grab today’s copy and download the partitions dated after the date of the copy you last synced — whether that was yesterday, the 15th of last month, or whatever day you fancy. There’s no fixed release calendar to wait on. A copy from before 7 November 2026 needs a one-time catch-up.
  • Full rebuild, as often as you like. Rebuild your entire database from scratch every day if you want — each dated folder is the whole thing.
  • Deletions handled. Each day’s copy is the complete current corpus, so records that were deleted or merged away are simply absent — reconcile against it and they fall out of your mirror. For works, each copy also names its deletions explicitly in deleted_ids.csv.gz (see below).

Tracking only some fields

A night’s works partition can be huge (68 million works, about 205 GB of Parquet on 22 November 2026) because updated_date moves whenever anything in a work changes. Downloading full works wastes bandwidth when you only track a few fields, like citation counts and work type.

Instead, read the Parquet copy in the same dated folder (parquet/ instead of jsonl/). Parquet stores each field as its own column, so a reader that asks for a few columns downloads only those; on 22 November 2026, id, cited_by_count, and type were about 0.4% of that night’s Parquet bytes (about 0.7 GB). Compare those values with yours and update what differs.

This DuckDB query works with the AWS profile from the snapshot page when the secret uses CHAIN 'process'. Replace both dates with today’s folder and the partitions you need:

INSTALL httpfs; LOAD httpfs; INSTALL aws; LOAD aws;
CREATE SECRET openalex (TYPE s3, PROVIDER credential_chain, CHAIN 'process', PROFILE 'openalex', REGION 'us-east-1');
SELECT id, cited_by_count, type
FROM read_parquet('s3://openalex-snapshots/full/2026-11-22/parquet/works/updated_date=2026-11-22/*.parquet');

Deletions and merges still come from deleted_ids.csv.gz and merged_ids.csv.gz.

API sync filters (Member+ and Partner)

Some people would rather get updates from the API than from a snapshot. We support that: for small numbers of updates, the sync filters make the API itself a sync mechanism, on every entity type:

  • from_created_date: records created since a date
  • from_updated_date: records updated since a date

to_created_date and to_updated_date (on or before a date) close the window, on every entity type too.

These are the one exception to “every filter is free”. They tell you what changed, and keeping up to date is the service the plans pay for, so like the daily snapshot they need a Member+ or Partner plan (older Premium and Institutional contracts include them too; Member doesn’t). The same goes for sorting works by created_date or updated_date (only works sort by these dates). Without a plan, the API answers 403 with "Plan upgrade required".

https://api.openalex.org/works?filter=from_updated_date:2026-07-30&corpus=all

On works, add corpus=all. Works queries cover only the core corpus unless you ask for more, while the snapshot holds every work. A mirror that polls without corpus=all never hears about changes to expansion works, which were between 4% and a quarter of a night’s work updates in November 2026.

Poll with these and upsert the results by id: no snapshot downloads, no bulk infrastructure, just HTTP requests. It’s also the freshest view of the data, since the API updates continuously rather than on a release schedule.

Poll by day, with a day of overlap. updated_date and created_date are in UTC, written without an offset (2026-11-15T12:55:33.944570). Most records change in the nightly build: a night’s changed works share one updated_date, stamped when that night’s build ran, and they reach the API a few hours later. So a poll that asks only for changes since its last run, an hour ago, finds none of them. Ask instead from the day before your last successful poll (on 15 November, from_updated_date:2026-11-14) and upsert by id; records you already have just come back again.

The real superpower is that they combine with any other filter, which makes this the method for keeping a subset of OpenAlex synced. Most mirrors don’t need the whole database: an institution tracking its own research output, say, only needs its own works, and standing up snapshot infrastructure for that is wild overkill. One polling loop covers it:

https://api.openalex.org/works?filter=institutions.id:I27837315,from_updated_date:2026-07-30&corpus=all

Trade-offs: paging through the API is slower per record than bulk files, so this method fits modest change volumes; for full-database mirrors, use the daily snapshot. And the filters only surface records that exist: deleted or merged-away records just stop appearing (deleted IDs 404; merged-away IDs usually resolve to the survivor), so reconcile your ID set against a snapshot periodically if deletions matter to you. Filter mechanics: Sync filters in the API reference.

The Unpaywall Data Feed (subscribers)

Unpaywall subscribers can receive daily change updates in the legacy Unpaywall record format — works with Crossref DOIs only. It delivers changes to the same underlying OpenAlex data, and exists for the ecosystem of integrations that already speak the Unpaywall schema; for anything new, use one of the OpenAlex-native methods above. Details: the Unpaywall Data Feed.

Free data, paid services

All OpenAlex data is CC0 and free: anyone can download the whole database, forever, at no cost. What’s paid is the service of keeping up to date. Building and publishing the full database every day, and answering what-changed queries on the API, costs us real money every day, so the people who need daily updates pay for it, and that keeps the data free for everyone else. It’s the model the Principles of Open Scholarly Infrastructure (POSI) encourage: keep the data open, sell services, not data.

So every way of getting updates is paid: the daily snapshot (the main route), the API sync filters (for small numbers of updates), and the Unpaywall Data Feed. Plans and the whole model: Pricing.

Deletions and merged entities

Records don’t just get created and updated — they also disappear: works get merged when they’re found to be duplicates, author profiles get merged or deleted, and bogus records get removed.

How that shows up today:

  • In the API: a merged-away work, author, source, institution or funder ID usually resolves to the record it was merged into: you get the surviving record (its id is the surviving ID), and filters on the old ID match the surviving record. This is best effort, not a permanence promise; see Merged entities. A deleted ID returns 404. (Special case: works of removed author profiles point to the null author A9999999999, and A5317838346 marks deleted authors.)
  • In the snapshot: the record is gone from the current release — it doesn’t appear in any partition, and the vacated file space disappears from the manifest. This is true of every daily copy too, so reconciling against a snapshot picks up deletions.
  • In deleted_ids.csv.gz (works): every daily snapshot, and the public bucket since the 2026-09-23 release, names its deleted works explicitly — see below.
  • In merged_ids.csv.gz (works, authors, sources, institutions): every merged-away ID with the ID it now resolves to — see below.

The works deletion log: deleted_ids.csv.gz

Works only, for now. Works are the first entity type with an explicit deletion log; every other entity type still relies on the reconcile methods below to detect deletions.

Each snapshot’s works directory carries the log at {format}/works/deleted_ids.csv.gz (same file in both format trees, next to manifest.json). It’s a gzip-compressed CSV with a header and two columns, and it’s big: 343 MB compressed and 102.9 million rows in the daily snapshot of 7 November 2026 (about 4.5 GB unpacked), so load it into your database or DuckDB rather than reading it line by line:

work_id,deleted_date
https://openalex.org/W4245566371,2026-08-14
  • work_id — the deleted work’s ID, in the same URL form as the works data files.
  • deleted_date — the date the work disappeared from the corpus. (For works deleted before the log existed and backfilled into it, this is the date the deletion was detected, not the original deletion date.)

The file is cumulative — each release carries the full history, so you don’t need to collect it daily. Apply it as: remove these IDs from your copy of works. That makes mirror deletions a direct lookup instead of a full ID-set diff (the reconcile methods below still work and remain the belt-and-braces option). It’s a ledger, not an append-only log: in the rare case a deleted work is restored, its row is removed and the work reappears in a partition — so always treat the latest file as the whole truth rather than diffing it against an older copy.

Deletion is final. A deleted ID stays deleted, and work IDs are never reused — a deleted ID will never point to a different work.

Where it is today: in every daily snapshot since 2026-08-15 (written as an uncompressed deleted_ids.csv up to the 2026-09-24 snapshot; gzip-compressed only from 2026-09-25), and in the free public bucket since the 2026-09-23 release.

Merged IDs: merged_ids.csv.gz

Each snapshot’s works, authors, sources and institutions directories carry merged_ids.csv.gz ({format}/works/merged_ids.csv.gz, {format}/authors/merged_ids.csv.gz, {format}/sources/merged_ids.csv.gz and {format}/institutions/merged_ids.csv.gz, next to each manifest.json): every merged-away ID and the live record it now resolves to. Where it is today: in every daily snapshot since 2026-10-27 (institutions since 2026-11-17); the free public bucket gets it with the next quarterly release (13 January 2027), since the 2026-10-14 release came out before the file existed. Use it to repair stored IDs in bulk, the way the API resolves them one at a time. A merged-away work also appears in deleted_ids.csv.gz (it left the corpus); merged_ids.csv.gz tells you where it went.

id,merge_into_id,merge_date
https://openalex.org/A5107411446,https://openalex.org/A5107863564,2026-10-11
  • id — the merged-away ID, in the same URL form as the entity files.
  • merge_into_id — the live record it resolves to today. Chains are already followed: if A was merged into B and B later into C, the row says A → C.
  • merge_date — the date of the merge the row reflects.

The file is cumulative and the whole truth each day: a merged-away ID that later comes back into use drops out, and a row’s merge_into_id changes if the surviving record is itself merged. Apply it as: replace each id in your data with its merge_into_id. The API gives the same pairs one request at a time: a list response whose filter names merged IDs lists them in meta.merged_ids (see Merged entities). It covers the merges in our merge ledgers, including the pre-2026 legacy-data/merged_ids/ files. Deleted records (with no surviving record) are not in it: those IDs are gone. The works and authors files run to tens of millions of rows, so load them into a database or DuckDB rather than reading them line by line. Like the API redirects, the file is best effort, not a promise that any particular ID will keep resolving.

Where it is today: in every daily snapshot from 2026-10-27 (works added that evening); all three in the free public bucket from the next quarterly release.

Tracing merges: location IDs move to the surviving work

The direct route is works/merged_ids.csv.gz, which names each merged work’s survivor. Location IDs are a second trail:

When two works are found to be duplicates and merged, the surviving work absorbs the other’s locations. Each location carries an internal id — a stable handle for the underlying harvested record, like doi:10.7717/peerj.4375, pmid:29456894, or pmh:oai:europepmc.org:4724910 — and that handle survives the merge. So between releases you can see a location id that used to sit on work A now sitting on work B: that’s the merge trail.

Combining the two signals gives you the full picture of records leaving the corpus:

You observeIt means
Work A in deleted_ids.csv.gz, and A’s location ids now appear under work BA was merged into B — repoint anything referencing A to B
Work A in deleted_ids.csv.gz, and its location ids appear nowhereA was removed (bogus or unsupportable record)

You can also spot a merge before it lands: if two live records share the same DOI (works) or ISSN-L (sources), the one with the more recent updated_date is the canonical record that keeps receiving updates; the other has stopped updating and is on its way out. Picking the most recently updated record is a reliable rule for choosing the survivor.

Picking up deletions in a mirror

Reconcile against a full release periodically (daily-snapshot subscribers can do this any day):

  • Works: delete every ID in deleted_ids.csv.gz from your copy — cheap and exact.
  • File mirror: aws s3 sync ... --delete, then rebuild — the synced tree is the current corpus.
  • Database mirror (all entities): diff your ID set against the release’s ID set (stream IDs from the part files, or compare per-partition record_counts in the manifest) and delete local records that no longer exist upstream.

Note: The pre-Walden snapshot published a merged_ids/ directory mapping merged IDs to their survivors. That mechanism ended with the 2025 Walden cutover (historical files preserved under legacy-data/, not updated). Its successors are deleted_ids.csv.gz for works deletions and merged_ids.csv.gz for works, authors, sources and institutions merges, which also carries the legacy rows. What does not exist today: deletion logs for authors, sources, institutions, or any other entity type. If that matters to your pipeline, tell us about your use case.

Point-in-time and reproducibility

The public bucket holds only the current release: once a release lands, the previous state is gone, so cite the combined manifest’s date (or the RELEASE_NOTES.txt entry) in anything you need to reproduce, and archive your own copy if you need to re-run against it. Member+ and Partner plans get closer to point-in-time: the staging bucket keeps each day’s full snapshot in its own dated folder.

Last updated

View as Markdown