Using Unpaywall data safely

Some fields in Unpaywall records (and OpenAlex records generally) include text from external sources like Crossref. If you’re simply viewing an API response in a browser or reading a spreadsheet from the Simple Query Tool, this causes no problems. But if you write programs that process this text or include it on web pages, be aware of a class of bug called a code injection vulnerability.

When you mix text someone else gives you with code or markup that gets interpreted, the interpreter sees your code and that text the same way — certain sequences can change your application’s behavior. A real example: the title of this article is:

<em>Traje</em>'s Future: Gendered Paths in Guatemala

Insert that directly into a web page and the <em> tag italicizes the word rather than displaying as text. That seems harmless — maybe even an improvement — but you’ve given up control of how your application behaves: the author of that text can now do anything your markup can do. HTML solves this with character entity references; to display correctly, the title should be converted to:

<em>Traje</em>'s Future: Gendered Paths in Guatemala

This isn’t unique to Unpaywall or to HTML — it applies any time you include someone else’s text in code or markup. Cross-site scripting and SQL injection are two especially important variants. And while most discussion of these involves malicious attackers, it’s very easy to hit accidentally, as in the example above.

The general solution: encode the text so it can’t be interpreted as part of your program, or use features like prepared statements that do it for you. In most cases there’s a library for your language and output format. (When OpenAlex creates an API response, Python’s json package encodes all text so it can’t alter the JSON structure.)

Unpaywall doesn’t remove or alter text that looks like HTML, SQL, or any other language — you get the text we get. The official metadata is worth preserving even when it contains oddities, complete rather than arbitrarily (and possibly ineffectively) sanitized. Handling it safely is the consumer’s job — now you know how.