Fulltext
OpenAlex keeps its own cached copy of the full text for more than 60 million open-access works, as PDF and as machine-readable TEI XML parsed by Grobid. Every work that has one links to it directly: has_content says which formats exist, and content_urls holds the download links, served from the content API at content.openalex.org. Any API key works, including a free one; downloads are metered against your daily usage like any other call, and the free allowance covers a decent batch every day (see Example costs).
From there it’s a question of scale: one file at a time from the API, a few million with the CLI, or the complete archive synced to your own bucket.
| Format | Files | Size |
|---|---|---|
| ~73M | ~113 TB | |
| TEI XML | ~71M | ~1.5 TB |
Download options
Option 1: API (up to ~10K files)
Download files one at a time from the content API. Each download is metered against your API usage (Example costs). A HEAD request (the file’s size and type, no bytes) is free.
https://content.openalex.org/works/{work_id}.pdf?api_key=YOUR_KEY
| Extension | Format |
|---|---|
.pdf | PDF (default) |
.grobid-xml | TEI XML |
curl "https://content.openalex.org/works/W3038568908.pdf?api_key=YOUR_KEY"
curl --compressed "https://content.openalex.org/works/W3038568908.grobid-xml?api_key=YOUR_KEY"
TEI XML is served as Content-Type: application/xml with Content-Encoding: gzip, so most HTTP clients (requests, httpx, fetch, browsers) hand you plain XML with no extra work. Plain curl doesn’t decode content encodings unless you pass --compressed; without it you’ll get the raw gzip bytes and need to gunzip them yourself. The compressed bytes are what’s stored, so if your client ever sees a body starting with the gzip magic bytes 1f 8b, decompressing it yields the same XML.
Finding downloadable works:
Use the has_content filter to find works with available content:
# Works with PDFs from 2024
https://api.openalex.org/works?filter=has_content.pdf:true,publication_year:2024
# CC-BY works with PDFs
https://api.openalex.org/works?filter=has_content.pdf:true,best_oa_location.license:cc-by
You can also check the content_urls field on any work object to see if content is available.
A free API key’s daily allowance covers a batch of downloads every day (how many: Example costs). Good for research projects, building small corpora, or sampling.
Option 2: OpenAlex CLI (up to a few million files)
For larger downloads, use the OpenAlex CLI. It handles parallel downloads, retries, checkpointing, and resume automatically.
pip install openalex-official
Download PDFs for a specific topic:
openalex download \
--api-key YOUR_KEY \
--output ./climate-pdfs \
--filter "topics.id:T10325,has_content.pdf:true" \
--content pdf
Download works with Creative Commons licenses:
openalex download \
--api-key YOUR_KEY \
--output ./cc-pdfs \
--filter "best_oa_location.license:cc-by,has_content.pdf:true" \
--content pdf
Download metadata + PDFs + TEI XML:
openalex download \
--api-key YOUR_KEY \
--output ./my-corpus \
--filter "topics.id:T10325,has_content.pdf:true" \
--content pdf,xml
Standard usage rates apply per file; the accompanying metadata list calls are nearly free. Throughput is bandwidth-bound (PDFs average ~5 MB): expect roughly 20,000–50,000 files per hour on a fast home connection, several times that from a cloud VM — so a million files is a day or two from the cloud, or a few days from home.
Option 3: Complete archive sync
For the complete archive, the PDF sync service gives you read access to our storage buckets for your subscription year, so you can copy every file to your own infrastructure — and keep receiving new ones as they arrive. Files are stored on Cloudflare R2, which is fully S3-compatible: PDFs in openalex-pdfs, TEI XML in openalex-grobid-xml.
PDF sync is sold as a paid add-on to the Member+ or Partner plan: those plans include daily sync, and a new PDF is only usable with fresh metadata. What you’re buying is the ongoing sync service, not the documents themselves (see Pricing for the reasoning). Contact sales to get started.
How it works: we generate R2 API credentials with read-only access. The buckets also hold files no work uses (repeat downloads and superseded parses, about 60% of each bucket), so copy the files the manifest lists rather than the whole bucket. With rclone:
# once: save your credentials as an rclone remote
rclone config create openalex s3 provider=Cloudflare \
access_key_id=YOUR_KEY_ID secret_access_key=YOUR_SECRET no_check_bucket=true \
endpoint=https://a452eddbbe06eb7d02f4879cee70d29c.r2.cloudflarestorage.com
# copy the manifest (sync also removes any file we stop publishing)
rclone sync openalex:openalex-pdfs/_manifest/content_index ./manifest
# list the PDFs it names, then download them (rclone skips files you already have)
duckdb -c "COPY (SELECT DISTINCT pdf_uuid || '.pdf' FROM 'manifest/*.parquet' WHERE pdf_uuid IS NOT NULL) TO 'pdf_files.txt' (HEADER false, QUOTE '')"
rclone copy openalex:openalex-pdfs ./pdfs --files-from-raw pdf_files.txt --no-traverse --transfers 64
For TEI XML, do the same with grobid_xml_id || '.xml.gz' and the openalex-grobid-xml bucket. To stay current, run the last three commands daily after 04:00 UTC (the manifest is rebuilt from 03:30 UTC) — they pick up new files as they arrive.
The files the manifest lists come to ~113 TB of PDFs and ~1.5 TB of XML, so plan on a cloud VM with multi-gigabit networking: at 1–2 Gbit/s sustained, expect 5–10 days for the initial copy. A home connection is not a realistic option for the whole archive (a few hundred Mbit/s would take months).
We add files every day. We plan to move files no work uses out of the buckets, and will email PDF sync clients the date before we do. As a sensible default, validate each file on ingest: check the leading %PDF magic bytes, skip implausibly small objects, and run the two checks under PDF quality and limitations.
Mapping work IDs to files
Files in the archive bucket are named by UUID, not work ID. A work’s PDF and its TEI XML are stored as separate objects with different UUIDs — the file names alone won’t tell you which PDF and which XML belong to the same work:
openalex-pdfs/
├── 3a07228e-de2a-4c37-955d-b1411a498328.pdf
├── 7b12f4a1-8c9d-4e5f-a2b3-c4d5e6f7a8b9.pdf
└── ...
openalex-grobid-xml/
├── 5e6ea859-62f6-40ab-9f7e-6472aab3a19d.xml.gz
├── 523b7cb3-17f3-4e2f-b81c-867fd8802182.xml.gz
└── ...
The manifest (for archive sync)
Use the manifest we publish in the openalex-pdfs bucket itself:
s3://openalex-pdfs/_manifest/content_index/
It’s one Parquet file, manifest.parquet, regenerated in full once a day from 03:30 UTC (read it after 04:00 UTC, when the rebuild is done), that maps works to their file UUIDs. Until 23 November 2026 it lists one PDF and one XML per work. From 23 November 2026 it lists every file we hold, one row per location, so a work can have several rows; filter on is_primary for one row per work. The new layout is already at _manifest/content_index_preview/, which stays until 30 November.
| Column | Example | Object |
|---|---|---|
openalex_id | W2741809807 | the OpenAlex work ID |
pdf_uuid | 3a07228e-de2a-4c37-955d-b1411a498328 | openalex-pdfs/{pdf_uuid}.pdf |
grobid_xml_id | 5e6ea859-62f6-40ab-9f7e-6472aab3a19d | openalex-grobid-xml/{grobid_xml_id}.xml.gz |
native_id | 10.1038/s41586-020-2649-2 | the location’s DOI or repository (OAI-PMH) ID |
native_id_namespace | doi | doi or pmh |
updated_date | 2026-06-20 | when the location’s source record last changed |
version | publishedVersion | publishedVersion, acceptedVersion or submittedVersion (from 23 November) |
is_primary | true | the one file per work the content endpoint serves (from 23 November) |
UUIDs are stored without their file extension — append .pdf and .xml.gz to build the object keys. A row may have a PDF but no XML (GROBID can’t parse every PDF), so grobid_xml_id is sometimes empty.
Query it directly with any Parquet reader — no need to load it into a database:
import duckdb
# Look up one work
duckdb.sql("""
SELECT * FROM 'manifest/*.parquet'
WHERE openalex_id = 'W2741809807'
""")
The join is then entirely local, with no API calls:
- Join
openalex_idto your works snapshot to attach metadata. - Join
pdf_uuid/grobid_xml_idto your downloaded files to attach the PDF and TEI XML.
The file is replaced in full every day, so copy it with rclone sync (above), which also removes any file we stop publishing.
The content endpoint (for per-work fetches)
If you’re fetching files one at a time through the content API rather than syncing the bucket, you don’t deal with UUIDs at all — the endpoint resolves the work ID to the underlying object server-side:
https://content.openalex.org/works/{work_id}.pdf
Work objects from the API expose this directly via has_content and content_urls:
"has_content": { "pdf": true, "grobid_xml": true },
"content_urls": { "pdf": "https://content.openalex.org/works/W1775749144.pdf",
"grobid_xml": "https://content.openalex.org/works/W1775749144.grobid-xml" }
Filter your candidate set with has_content.pdf:true (available in the snapshot and the API; content_urls is API-only), then fetch each work by ID. This is convenient for small-to-medium sets, but at archive scale prefer the manifest above — the join is offline and costs nothing.
Warning: Do not map work IDs to archive objects via
locations[].pdf_url. That field is the original publisher URL, not the archive object — keying off it produces a lossy, inferred match. Use the manifest or the content endpoint instead.
PDF quality and limitations
has_content.pdf: true means we hold a copy of the work’s PDF. It doesn’t promise you can extract text from it. Scanned PDFs have no text layer, and some publisher PDFs embed fonts with no Unicode map, so text tools return glyph codes instead of words. Those files are faithful copies of what the publisher serves.
A small share of the PDFs we hold, about 1 in 100, were damaged when we fetched them (files we’ve fetched recently don’t show it): bytes went missing, so pages come out blank or the file won’t open. They still start with %PDF-, so a header check won’t catch them. Two cheap checks on each file catch most of them:
- the file ends with
%%EOF(look in its last few kilobytes); - the number after the last
startxrefis smaller than the file’s size in bytes.
For the rest, the sign is a text extraction that comes back empty for a file that should have text. If you find a damaged file, please send the work ID to support@openalex.org.
TEI XML quality and limitations
GROBID (project, docs) is the state of the art for converting scholarly PDFs to structured TEI XML. But PDF parsing is genuinely hard, and a meaningful share of files will contain errors — missing or duplicated references, occasional self-references (the paper’s own DOI picked up from its header or footer), wrong or partial header metadata, and the odd truncated section. You can see how GROBID performs field-by-field in the project’s own benchmarks.
GROBID also can’t parse every PDF. It doesn’t do OCR, so scanned or image-only PDFs produce little or nothing useful, and unusual or malformed PDFs can fail outright. Filter on has_content.grobid_xml:true to limit your set to works where we have a parse.
The XML comes in two shapes, because the archive was built with more than one GROBID version. Most files are standard TEI: a <TEI> root with <teiHeader> and <text><body>. An older minority instead wraps a lowercase <tei> element inside <html><body>, with the section <div>s hanging directly off <text> and no <body> element inside the TEI. An extractor that hardcodes one shape will silently return nothing for the other, so match on local element names case-insensitively and fall back from text/body/div to text/div.
We pass GROBID’s output through unchanged, including any errors it made. You can catch some with simple guard clauses — for example, drop any reference whose DOI matches the work’s own DOI, and cross-check GROBID’s header metadata against the OpenAlex Work record (which draws on Crossref and other sources independently of GROBID).
A lot of people today actually want Markdown rather than XML for LLM and RAG pipelines. We don’t have Markdown parses yet, though they’re on the roadmap. In the meantime you can roll your own by downloading the PDFs we provide and running them through one of the newer PDF→Markdown tools like Marker, Docling, or MinerU.
Licensing
The PDFs retain their original copyright. OpenAlex does not grant any additional rights to the content.
To check the license for a specific work, use the best_oa_location.license field in the API:
https://api.openalex.org/works?filter=has_content.pdf:true,best_oa_location.license:cc-by
This returns works that have downloadable PDFs and are licensed under CC BY.
Last updated
View as Markdown