We Need The Discourse Translation Plugin

[mDav (Trey Beyónce)]

If this global multi-language community forum is to succeed and thrive together, we need a method to translate posts from its original language, into a users home language.

There exists a plugin which allows for this functionality:

Discourse Translator

• Who thinks this will be a benefit to our community? Who doesn’t? And why?
• Are there any barriers to the implementation of a translation plugin? (@Site Admins)
• Any reason why we should not test such a plugin?
• If this succeeds and becomes a popular feature, enough to use more than the free tier of the translation services, who will pay for it?
• Other thoughts?

[ezekielf]

With the old forum dropping a url into Google Translate worked well enough, but this doesn’t work well with Discourse. Given that, I think adding this plugin would be very beneficial.

[Map_HeRo]

I believe an integrated translator could be a great step forward to reduce the language barrier for community members with different language background, specially for those not fit enough in english to engage themselves in the international part of the forum. Anyhow it would only make sense if

• it is easy to handle (one click)
• delivers useful translations
• would not slow down performance

in other words: a practical real world solution.

I have to admit that I do not have any experience with such plugins. So far I use DeepL whenever I need an translation occasionally, just copying the text on screen, paste it to the DeepL window and get the instant translation (as far as the language is available). This works well and the translations are better than any others I have seen so far but working through the posts of an extensive topic and then preparing a reply in the same way is a time consuming job … in other words: not really a practical real world solution.

@mDav

If this succeeds and becomes a popular feature, enough to use more than the free tier of the translation services, who will pay for it?

… very good question …

[nukeador (Rubén Martín)]

Hi @mDav and thanks for the request.

This is already part of the current languages and location proposal

Proposal - Language and location based content and categories

Español

Esta propuesta es el resultado de consolidar <a href="https://community.openstreetmap.org/t/handling-categories-and-languages/269/">comentarios muy diversos</a> sobre idiomas, lugares y categorías. Por favor, queremos leer tu opinión en los comentarios de abajo. Reconocemos que OpenStreetMap es una comunidad muy diversa, multicultural y multilingüe, que no puede traducirse fácilmente en una simple categorización u organización del foro. Algunos de los retos que …

We want to eventually experiment with it, probably once we have other big topics sorted out (specially the transition from Help OSM and the old forums).

[mDav (Trey Beyónce)]

Thanks for the pointer @nukeador

It wasn’t clear from the proposal language if this plugin was an active to-do item or just a “maybe”. Can you confirm it is a proper to-do in the roadmap for this discourse site?

@nukeador

We want to eventually experiment with it, probably once we have other big topics sorted out (specially the transition from Help OSM and the old forums).

The issue I see with adding the plugin far down the line is that the ability to translate posts will affect how people naturally organize topics and discussions. And, the way that topics become organized at the outset will be the ones which stick forever (some more thoughts on this here).

Considering it is relatively trivial to install plugins would you consider implementing this before the transition?

[nukeador (Rubén Martín)]

@mDav

It wasn’t clear from the proposal language if this plugin was an active to-do item or just a “maybe”. Can you confirm it is a proper to-do in the roadmap for this discourse site?

The proposal is still in the discussion phase, no decision has been taken yet.

@mDav

Considering it is relatively trivial to install plugins would you consider implementing this before the transition?

See my reply on the other topic:

Proposal - Language and location based content and categories

Testing the translate plugin is something we would like to do eventually, right now we have still a few big things to sort out with the Help OSM and old forums transition. We already know that a translator plugin is not going to replace language specific spaces, as some have already commented here. Thinking from a major European-language perspective it’s easy to think that a machine translation can solve it, but there are a ton of African and Asian languages where machine translation is not doi…

[cquest (Christian Quest)]

I’ve installed this plugin on https://forum.openfoodfacts.org/ using the Google translate API (not my choice).

It works pretty well:

• the globe icon appears when a post is not written in your selected interface language
• the translations are stored to reduce the calls to the translation API
• translation appears when you click on the globe, it does not show automatically if already available

A fork of this plugin supporting DeepL exists: GitHub - literatecomputing/discourse-translator
DeepL provides a very good quality, but supports less languages (24) than competitors.

All translation API have some costs: around 20$/MB for Google or DeepL (500K free per month)

[TomH (Tom Hughes)]

@cquest

All translation API have some costs: around 20$/MB for Google or DeepL (500K free per month)

This is the real question - from an operations point of view I am sure there are no objections to installing the plugin except that we somehow need to estimate what usage is likely to be so that we can work out the likely cost and seek a budget for it.

[mDav (Trey Beyónce)]

Thanks for sharing this @cquest! I signed up for the forum and tested it out, it works great! ps openfoodfacts is a super cool project!

@TomH

This is the real question - from an operations point of view I am sure there are no objections to installing the plugin except that we somehow need to estimate what usage is likely to be so that we can work out the likely cost and seek a budget for it.

There’s a really nice post on the Discourse forums on how to estimate translation costs, using a few SQL queries to estimate how many users and average length of posts etc.

How to estimate the cost of translation using the translator plugin

Sure we are at the beginning stages of the forums here so it’s hard to estimate with little data, but perhaps something similar can be run on the previous forum databases we have access to.

At least for DeepL, they offer to sign up for the free plan which doesn’t charge at all even if the free 500k characters is over (it just denies any more requests once monthly limit is reached). I see this as a way to test out the plugin so we can have a more nuanced discussion around it, with no risk.

What would you need to be willing to implement this @TomH?

[TomH (Tom Hughes)]

@mDav

What would you need to be willing to implement this @TomH?

It’s not really up to me as I’m not the lead on this project. I was just commenting with my OWG hat on as to what we would be concerned about.

[cquest (Christian Quest)]

@mDav

There’s a really nice post on the Discourse forums on how to estimate translation costs, using a few SQL queries to estimate how many users and average length of posts etc.

Using the old forum posts, we can try to evaluate if the cost would be really high and non sustainable for the OSMF, but with 20$/MB of text (that’s quite a lot of text) I think it should not be a problem.

[westnordost]

Regarding the DeepL fork, I do wonder why it exists at all because the discourse translator already supports adding different providers, so why is it not simply a Pull Request on the main branch? So maybe in that fork, some unsaintly hackery is involved?

Here is the comparison between the two branches: Turns out there is no hackery and I do wonder why this is not simply a PR

Comparing discourse:main...literatecomputing:main ·...

[westnordost]

I’ve created a test PR at the fork as a pretext to see if it is still maintained and more importantly to ask why it hasn’t been merged upstream yet (as the issue tracker is disabled here too):

add missing supported languages

[cquest (Christian Quest)]

Here is test on my migration test instance where I installed the plugin:

[dcbx (Daniel)]

i have to say: I’m quite impressed by the quality of the automated translation! after a short reading, it seems the two slang words “Bock” and “Schalter” (in this particular context) are the only issues…
But that shows some basic problem as well: some “meta semantics” expressed via slang or common local saying might be removed in the translation process.

[westnordost]

Which translation engine is this? It’s really good!

I researched into the different translation services a bit and put it into another thread:

Discourse Translator - Pricing and Language Coverage

Forking off from We Need The Discourse Translation Plugin , I researched a bit into the different translation services that could be used with the discourse translation plugin. Language coverage Actually, I was surprised that coverages of all services are pretty good. Part of this may be due to the Americas and large parts of Africa having English, French, Spanish or Portuguese as sole official or at least co-official languages. DeepL [deepl] The most obvious missing language is Arabic, a…

[cquest (Christian Quest)]

I’ve installed the DeepL modified plugin and DeepL translations are really very good as far as I’ve seen with the languages I understand (french, english, spanish).

The goal is not to test translation quality by itself, but its integration into Discourse UX.

On another discourse I’m administering, I’ve setup Google Translate.

[Nop]

I wouldn’t worry about the unability to translate particular slang words. I mean if you visit a remote corner of your own country where you are are native speaker you might also have a hard time understanding everything people are telling you in their regional dialect. Wouldn’t expect an auto-translate to do better than myself in my native language.

As the goal is to remove the language barrier and enable cross-lingual discussions (probably mostly on tagging and handling real-world stuff in OSM) I would go for the best translation quality so we don’t have automated misunderstandings.

It is better if a language is not supported - that’s a clear and easily understood information - than if many more are supported but there is frequent mistranslation and miscommunication.

I guess this warrants more testing/looking up of translator quality. From what was posted here so far, it looks like we already have a very good candidate with DeepL

[cquest (Christian Quest)]

I’ve not looked in depth at how the plugin works, but instead of selecting a single translation API, it should be possible to use multiple API:

• DeepL first for available languages
• a second one for other languages

[dcbx (Daniel)]

@Nop

I wouldn’t worry about the unability to translate particular slang words. I mean if you visit a remote corner of your own country where you are are native speaker you might also have a hard time understanding everything people are telling you in their regional dialect. Wouldn’t expect an auto-translate to do better than myself in my native language.

you’re right. good point!

@Nop

As the goal is to remove the language barrier and enable cross-lingual discussions (probably mostly on tagging and handling real-world stuff in OSM) I would go for the best translation quality so we don’t have automated misunderstandings.

@Nop

It is better if a language is not supported - that’s a clear and easily understood information - than if many more are supported but there is frequent mistranslation and miscommunication.

@Nop

I guess this warrants more testing/looking up of translator quality. From what was posted here so far, it looks like we already have a very good candidate with DeepL

Yeah, but that’s also the most expensive! Could be a real scaling issue…!