Spectral Web Services
I wrote an AI bot to simplify my translations workflow.
I spent my Saturday automating language PR review.
I'm continuing to manage translations for Ghost, as my time allows. The challenge, of course, is that I speak one language well (at least sort of arguably), and we're at 60+ languages and rising. For some languages, we have multiple translators who can review each other's work, but for many languages, we have one translator, who no one knows. Are they careful or sloppy? Trying to deface or just trying to help? No one knows.
So, I'm falling back on AI. Until this weekend, my pattern was that I would open the files changed view in GitHub, copy the changed lines, and paste them into ChatGPT, asking it to validate them. (Actually, my prompt is a couple paragraphs long, but that's the gist of it.) Then I'd look at the output and decide what should go back into GitHub as a comment to the translator.
For a new language submission, that was fine, although I ran out of free tokens pretty fast. For cases where a translator was filling in a handful of new strings in the almost 400 lines of translations, it was a choice between many many small copy-paste actions, or blowing through tokens on text that wasn't changing. Not ideal.
So I booted up Cursor, and set out to make a tool. I started out thinking that I'd write a tool that got collected the lines that had changed (to make copy and paste easier), but I thought "hey, I could use the OpenAI API to actually validate the changes, too! And then I could use the GitHub API to make the comment drafts!" You already know how this ends up, don't you? Scope creep time!
This was one of those times when the first 80% took 20% of the time. I got to something that almost worked really fast, but comments kept showing up on the wrong line, because apparently GitHub doesn't make that easy.
Eventually I switched Cursor over to using Claude Sonnet in 🧠 mode, which told me that there are two ways to handle line numbers with GitHub PR comments, and I was using the harder one. Phew. Comments are now in the right spot!
And then I spent a long time fussing with my prompting. I want the AI review to flag spelling errors, missing punctuation, variable problems, etc. I don't want it to nitpick the word choice on every damn line, sometimes twice per line. And I don't want it complaining about things that aren't there. (Spelling errors: "word" is misspelled! It should be "word"! Yes, those are the same damn string.) Switching from 4.1 to to o4-mini seemed to help. Or maybe I just finally got my prompt nailed down.
So now my flow is
- Get the PR number from GitHub, click it.
node index.js review 12345- Reload the window with the PR, find a bunch of pending comments.
- Delete or reword if needed.
- Edit the review comment and choose a status. Click to submit the review.
It's not instantaneous, but it's a pretty good upgrade on my old process, and removes the copy and paste annoyance.
I figure I'll have broken even on time spent to automate some time oh... next decade?
There's some irony in using AI to write code to invoke AI, and directions to AI. I mean, I'd like to think it'd be good at prompting itself. There were moments when I wasn't sure that was actually the case.
Why are you using AI, anyway?
I absolutely think humans should do translations for humans. And the ground rules on doing translations for Ghost are that they should be human-produced, by fluent speakers. But validating translations is hard. I'm hoping to use the AI as a basic sanity check. Are there expletives? Are there typos? Did the translator make a copy-paste error and translate the wrong string?
If we have no strings in that language and it passes an AI sanity check, then I can go ahead and merge it. It'll be better than nothing!
When we have strings already and a new submitter is proposing changes, then I'll do my review (because finding typos is still a good idea), but I'll also hunt back for other translators in the same language, and ask them to have a look at the proposal. Not all translators respond, but many do, and so then we've got two or three people having a discussion about the translations. For an all-volunteer i18n project, run/coordinated/managed/herded by a volunteer (that's me), I'm not sure we can do much better.
So, are you happy with your new 🤖 helper?
Yeah, yeah I think I am.
I'm thinking I'll name my 🤖 friend AI18n.
Technical tidbits
- I'm loading all the changed lines, although just the new form. (It would probably be better to pass both, because then the AI could also compare them.)
- I'm loading the entire file into context for any file changed, to allow for observations about using the same translation and formality across the platform.
- I'm loading the entire context.json. I got it to mostly-complete a couple months ago, but there's some room for improvement still, and I wrote it thinking about it being hints for humans, not for AI 🤖, so it could be perhaps improved.
- I'm using the different levels of directions, so that the developer-level directions cannot be overridden (at least in theory) by the translator's words.
- My directions (likely to have been adjusted again by the time you read this:
You are an AI assistant analyzing i18n (internationalization) changes in a Ghost (blogging and newsletter publishing platform) repository.
We are validating translations from English to the given language. Do not critique or correct the English, only the translations.
Please validate these i18n additions or changes across multiple files.
Focus specifically on:
- Is there any attempt to deface?
- Are there any typos or grammar errors?
- Are the translations accurate and appropriate?
These translations were produced by a human who is a native speaker of the target language.
Word any feedback in a polite way that respects their authority as the native speaker.
Ask questions about things that might be errors. "Is the spelling of ___ correct?" Say "I think" or "I suspect".Be polite and deferential to the translator's expertise, but tell them if you suspect they've made an error.
Match the ending punctuation of the original English. Translators should not add or remove punctuation. You don't need to phrase that as a question.
Say "please" occasionally.
Do not nitpick wording too much. Only leave a comment or suggestion if it looks like an error on the part of the translator.
Raise any number of issues.
Only make suggestions if they are specific and actionable.
### Write an overall comment:
- Overall translation quality and consistency
- Any patterns or trends across multiple files
- General tone and style consistency
- Any systemic issues that affect multiple translations
- Do NOT repeat what's already listed in the comments.
### Format your response as JSON with the following structure:
{ "comments": [ {
"type": "error|warning|info|suggestion",
"filename": "filename.json",
"diffPosition": diffPosition,
"message": "[Your feedback here - be specific and actionable - please write in English]" } ],
"overall": "[An 'overall' section is optional. If you have any overall comments, please write them here. Finish your comment with 'Thank you!' in the translator's own language.]"
}
CRITICAL: For each issue or suggestion, you MUST use the exact diffPosition value that corresponds to the specific line containing the translation you are commenting on. You must also include the filename for each comment.
The valid diffPosition values for each file are:${validPositions.map(pos => `${pos.filename}: ${pos.diffPosition}`).join(', ')}
IMPORTANT: Look at the translation content in each line and use the diffPosition that matches the line containing the specific translation you want to comment on.
### Directions provided to the translator:
- Formality: Ghost's brand has a friendly and fairly informal tone. If your language has both formal and informal form, choose one or the other and use it consistently. Most languages are using the informal version, but if the informal version might be considered rude, then choose the formal.
- All kinds of people use Ghost. When possible, prefer gender-neutral language.
- Translations should work for a variety of Ghost sites. Choose translations that will make sense for both personal blogs/newsletters and news publications.
### Common issues to avoid & other notes:
Consult the context.json file if you aren't sure how a string will be used.
In English, we use "Jamie Larson" for a placeholder in any field that needs a name. Please do not transliterate. Please do not replace with "Name". Instead, replace Jamie Larson with a name that will be recognized as a name in your language. Choose something uncontroversial and common. If possible, choose a non-gendered name.
Do not translate variables (inside {}). Do not add variables. If a translator has omitted a variable or made an error in the variable name, please note that in your comment.
Watch out for "You are receiving this because you are a %%{status}%% subscriber to {site}.'", which takes the "free", "trialing", "paid", and "complimentary" strings in the %%{status}%% field. These strings need to produce good grammar when substituted.
Want to help?
The "i18n mega issue" has directions, requests, and more: https://github.com/TryGhost/Ghost/issues/23361
Hey, before you go... If your finances allow you to keep this tea-drinking ghost and the freelancer behind her supplied with our hot beverage of choice, we'd both appreciate it!
Translations are fun, but they don't pay the bills.
