How do I read JSONL files?

The OpenAlex snapshot delivers data in the JSON Lines text format (JSONL): each line of a file is one JSON object representing a single record.

Since the decompressed snapshot runs to terabytes, this is a great advantage over plain JSON — you can process one line at a time without ever loading a whole file. The downside is that a JSONL file as a whole isn’t valid JSON, so tools expecting one big JSON document can’t read it directly. Treat each line, not the file, as the JSON object.

Python

Read gzipped part files directly — no need to decompress on disk:

import gzip, json

with gzip.open("part_0000.gz", "rt") as f:
    for line in f:
        work = json.loads(line)
        print(work["id"], work["display_name"])

From there, load records into pandas, DuckDB, or whatever structure suits. (DuckDB can also query .gz JSONL files directly: SELECT id, display_name FROM read_json_auto('part_*.gz') — and if you’d rather skip JSONL entirely, the snapshot’s Parquet copy loads straight into DuckDB, Spark, or BigQuery.)

R

Same principle with readLines and jsonlite:

library(jsonlite)

con <- gzfile("part_0000.gz", "r")
while (length(line <- readLines(con, n = 1)) > 0) {
  work <- fromJSON(line)
  # do something with work$id, work$display_name, ...
}
close(con)

The important thing in any language: parse each line as its own JSON object.