Every system I have worked on here broke somewhere unglamorous, and none of them broke because of traffic. The community platform did not struggle because too many people read a guide. It got hard to change because a guide has a title, a title has to exist in more than one language, and nobody had decided what "exists" means when three of the seven translations are empty.
That is the shape of scale in a studio like ours. We build for people who are not served well by Korean-only software: a community platform, an English-first marketplace, a retail consulting tool that runs in seven languages, and an application suite for international students. The load is not requests per second. It is the number of content shapes multiplied by the number of locales multiplied by the number of surfaces each one renders on. Get that wrong and the fourth language costs more than the third, which already cost more than the second. That curve is what kills small teams, long before concurrency does.
The data model sets the price of your next language
There are three ways to store a translatable field, and the choice is effectively permanent by the time you have real content in the table.
| Shape | Adding a locale costs | Cheap | Expensive |
|---|---|---|---|
A column per language (title_en, title_ko) |
a migration on every translatable table, plus every query and serializer that names those columns | sorting, filtering, indexing, reading the schema | everything after the second language |
| A translations table keyed by (row, field, locale) | one row insert | adding locales, partial translation, auditing coverage | reads: a join per translatable field, and any ordering by a translated value |
| A JSONB column keyed by locale | one key in an object | reads, because a single row carries every locale | sorting and filtering, which need an expression index per locale you actually sort on |
We use the third for content, and I would defend it, but the bill is real. Ordering guides alphabetically for a Korean reader means an index on (title->>'ko') and a separate one for every other locale that gets its own sorted list. Postgres will not use one index for all seven. So the rule we settled on is that a locale gets an expression index when a screen sorts by that field in that locale, and not before. Three indexes exist today. The other four locales sort by recency, which needs no index per locale at all.
The column-per-language shape is the one that looks best in a code review and ages worst. It is fine at two languages. At seven it means seven columns per translatable field, a serializer with a seven-branch switch, and a migration every time a client asks for Vietnamese.
A fallback chain is a function, not a convention
The bug that taught us this rendered an English title above a Korean body on the same card. Both values were present, both were correct, and the result looked broken. A different row had a translation stored as an empty string rather than null, so the "is it translated" check passed and the card rendered a blank heading.
Resolution now lives in one function that every serializer calls. It walks the requested locale, then the locale the content was authored in, then English, and it returns both the value and the locale it resolved to. The UI can then say what the reader is looking at instead of quietly pretending. A silent fallback is a bug that we cannot see and the user can, which is the worst possible split.
That costs us something. Every translated field on the wire is now an object rather than a string, so payloads are fatter and the frontend types are noisier. We took it, because the alternative was a class of bug that only ever gets reported as "the site looks weird".
Search is where a multilingual database stops pretending
Postgres full-text search wants a language configuration per column. to_tsvector('english', ...) stems, so a search for "banking" finds a guide titled "Banks and transfers". Stock Postgres ships no Korean configuration, so the honest option is simple, which does not stem and does not split agglutinative forms. A reader who types 회원 가입 as two tokens and a guide that wrote 회원가입 as one do not meet.
There is a third case that neither configuration touches: a reader whose keyboard is Latin script searching for a Korean brand in romanization. Every tsvector in the table is useless to them.
So we run two matchers, not one.
-- one tsvector per language configuration, each with its own GIN index
ALTER TABLE guide ADD COLUMN search_en tsvector GENERATED ALWAYS AS (
to_tsvector('english', coalesce(title->>'en','') || ' ' || coalesce(body->>'en',''))
) STORED;
ALTER TABLE guide ADD COLUMN search_ko tsvector GENERATED ALWAYS AS (
to_tsvector('simple', coalesce(title->>'ko','') || ' ' || coalesce(body->>'ko',''))
) STORED;
CREATE INDEX guide_search_en_idx ON guide USING gin (search_en);
CREATE INDEX guide_search_ko_idx ON guide USING gin (search_ko);
-- trigrams over every locale of the title, for romanization and typos
CREATE INDEX guide_title_trgm_idx ON guide USING gin (title_all gin_trgm_ops);
Two matchers produce two scores that are not comparable. ts_rank and trigram similarity live on different scales, and any weighted blend of them is a number somebody made up in an afternoon. We do not blend. The tsvector results come first in their own rank order, and the trigram query only runs when the tsvector query returns fewer rows than fill a page. Its results are appended under a visible break labelled as close matches. That rule is also arbitrary, but it is arbitrary in a way we can explain to a reader and change in one place.
Writes pay for all of this. Generated columns are computed on every insert and update, and both GIN indexes are touched at the same time. For guides that is obviously the right trade: the library is a little over thirty guides, the most read one has passed 1,100 views, and each guide is edited a handful of times in its life. Marketplace listings have the opposite profile, since a seller adjusts price and photos repeatedly in the first hour after posting. So listings index title and category and leave the description out of the tsvector entirely.
A marketplace is read shaped until the second it is not
Browsing dominates. Almost every request against the marketplace is somebody scrolling a category, and only a thin slice of them post, edit, or buy. That ratio makes caching the category listing the obvious first optimization, and it is correct right up until a seller posts an item, refreshes, does not see it, and posts it again. Now there are two listings, a confused seller, and a support message. The cache was doing exactly what it was told. The product was wrong.
The version that works: the cached list is the anonymous one, and an authenticated seller's response merges their own recent rows straight from Postgres on top of it. That extra query is bounded by definition, because one person's recent listings are always few. Everyone else sees a list that can be stale up to the TTL.
We chose a short TTL over event-driven invalidation deliberately. Event-driven invalidation is more correct and it is also a distributed system: a publisher, a subscriber, a retry policy, a dead-letter path, and a failure mode where one dropped message leaves a category page wrong indefinitely with nothing to alert on. A TTL is wrong for a known, bounded window and then fixes itself. At our size that is better engineering, and the day it stops being better we will know, because the complaint will be specific.
Counters are writes wearing a read's clothes
Guides carry views and likes, and those counts are not decoration: they are how the platform decides what to surface. A guide that reaches a thousand readers is the one that belongs at the top of its category.
The naive implementation is one statement.
UPDATE guide SET views = views + 1 WHERE id = $1;
On the most popular guide in the library, that statement puts every concurrent reader behind a row lock on the single hottest row in the table, and it produces a dead tuple per view for autovacuum to clean up later. The read path has quietly become the write bottleneck, and it gets worse exactly as a guide gets more successful.
Increments go to Redis. A periodic task flushes them into Postgres, and a read returns the stored value plus the live delta. If Redis dies we lose a flush window of view counts. For a view counter that is a fine thing to lose. The line I hold is that this pattern is allowed for anything nobody would file a complaint about being slightly off, and never for anything attached to money or eligibility. No counter that touches an order lives in Redis alone.
Owning the database means owning the boring five minutes
Production runs on hardware we own, in Docker, behind a tunnel: Postgres, Redis, the API, the background workers. That buys predictable cost and total control, and it charges for both on the day the machine loses power.
After an unclean shutdown Postgres replays its write-ahead log, and before it can do that it fsyncs the entire data directory. On our disks that takes minutes. Every other container has already started by then, because restart policies start everything at once and do not honor dependency ordering on boot. The API opens a database connection during app initialization, blocks there, and never binds its port. The tunnel in front of it has no origin to route to. From outside, the whole site is a gateway error and the database looks dead.
It is not dead. It is working, and it says so in its own log, one line at a time, with an elapsed counter that climbs.
The mistake I made the first time is the part worth writing down. I restarted it. The Postgres image stops on SIGINT, which is a fast shutdown, which aborts the recovery already in progress. The next start begins recovery from the beginning. Every restart reset the clock, and from the outside "recovering normally" and "stuck in a loop" look identical if the only thing you are watching is a health endpoint.
Three changes came out of that hour.
- The database healthcheck got a start period long enough to cover a genuine recovery, so it reports itself as starting rather than unhealthy while it works. Any watchdog that reacts to unhealthy is now structurally unable to make things worse.
- The API entrypoint waits for the database in a bounded retry loop instead of opening a connection at import time and hanging. A process that exits with a clear message is diagnosable. A process hung inside an import is not.
- Boot is a script with an explicit order, not a pile of restart policies. Database first, wait until it accepts connections, then the API and workers, then everything that is not production. Nothing else starts on boot at all.
Underneath all three is a rule about behaviour rather than code: while a database is recovering, the correct action is to read the elapsed counter twice and confirm it is climbing, then do nothing. Doing nothing is the hardest operational skill to build, because every instinct while the site is down is to type something.
The work that does not belong inside a request
The student application suite produces a submission-ready document: a filled resume and self-introduction, printable and downloadable as a PDF. Students do not generate it once. They fix a date, regenerate, notice a phone number, regenerate.
Document rendering is CPU bound. Inside the request, one student's download holds a worker process for its entire duration, and the worker pool is small and fixed. Enough concurrent downloads and every other request on the site queues behind documents, including the ones that have nothing to do with documents. The site does not fall over. It becomes uniformly slow, which is harder to diagnose, because there is no error anywhere to point at.
Anything CPU bound and anything that calls a third party goes to a queue. The request records a job and returns, and the artifact appears when it is ready. That is more moving parts and more code than rendering inline, and it is the difference between one slow feature and one slow product.
The document is also deterministic: the same form payload produces the same file every time. So the artifact is keyed on a hash of the payload, and a student who regenerates without having changed anything gets the file we already made. That is less a performance trick than an admission about how people actually use forms.
The trade I would make again, and the one I would not
I would take JSONB translations with an explicit fallback every time. It costs an expression index per sorted locale and a fatter payload on the wire. What it bought is that adding the sixth and seventh language to the phone shop tool was content work, not a schema migration plus a sweep through every query in the codebase. That is the whole point: make the second language cheap so the seventh is possible.
The one I would not repeat, at least not this early, is putting the primary production database on hardware we own before anybody's job description included being awake for it. The engineering was sound and the fixes above were the right fixes. The cost that never appeared in any estimate is that recovery time is now a person, and a small team has a small number of people. Starting the same stack tomorrow I would argue for self-hosting everything except the database, and I would be arguing against my own preference.
Both trades come from the same habit: find the thing that gets more expensive every time you add one more of it, and pay for that curve while it is still a schema decision rather than a migration with a downtime window.
