yceffort
◆ SERIES 2 POSTS

OG Scraping Server Design Notes

What should you decide first when building a link preview server? Following the principles from the reasoning behind the runtime choice to SSRF, encoding, and cache stampedes, and running every piece of code written along the way

README

On paper, a link preview is a three-line feature. Open the URL, parse the HTML, pull out the og: tags. In production, though, those three lines break in quite a few places, and many of them are lumped together behind a single number, "the error rate is high", where they are hard to see.

This series is a set of design notes that follows, in order, the places where those three lines break. Part 1 splits failures by cause, examines where the runtime choice actually makes a difference, and goes on to raise coverage with User-Agent and encoding handling and to set up caching and target numbers. Part 2 deals with the risk of a server opening a URL on behalf of a user, and looks at how it gets broken before how to block it.

The focus is on why the design ends up the way it does rather than on usage, and every piece of code shown was run on Node.js v24.14.1. In the process I found that four of the things I had first written down plausibly were wrong, and I left them in place instead of deleting them. That contrast is also what I most wanted to say in this series.

02 ITEMS

All posts

in order
01

Building an OG Scraping Server in Node.js (1): From Runtime Choice to Error Rate and Latency

#web-scraping#backend#nodejs
The "10% error rate" of a link preview server mixes five kinds of failure with different causes. This post works out the four points where runtime choice actually matters, lowers the error rate by fixing User-Agent and encoding handling, and verifies the "P95 under one second" target by working backwards from the cache hit rate with a two-million-run simulation. Along the way it shows that Node built-in TextDecoder turns CP949 extension characters into different characters without raising an error.
33 min read
02

Building an OG Scraping Server in Node.js (2): How SSRF Gets Through

#security#nodejs#backend
A feature where the server opens a URL the user handed it is, by itself, the textbook setup for SSRF. I ran six whitelist bypasses and the five defensive principles that block them on Node. Strip IPv4-mapped by hand and hex notation gets through, and undici lookup hook is never called when the host is an IP literal.
25 min read