Glitchy town labels in Oklahoma

[Minh_Nguyen (Minh Nguyễn)]

If you see tofu spreading across the map of Oklahoma, please do not be alarmed. In some cases, an apparent text rendering bug in Mapnik is causing some styles like OSM Carto and Tracestrack Topo to produce missing character placeholders until the label reaches the maximum possible length. Unluckily, this bug affects the town of Hominy (𐒹𐓘́͘𐓨𐓘͘𐓵𐓣͘ in Osage), causing its label to stretch across most of northeastern Oklahoma.

As seen in these screenshots, not every Osage label is affected by this bug. It seems to affect only a particular sequence of characters.

Known affected maps include:

• OpenStreetMap Carto
• Tracestrack Topo
• Wikimedia Maps^[1]

Most maps are unaffected, subject to font availability:

• CoMaps and Organic Maps
• CyclOSM
• Humanitarian
• Magic Earth
• OpenStreetMap Americana
• OsmAnd
• Shortbread
• Transport

Maps that only show labels in English are also unaffected. Other kinds of data consumers like geocoders and routers rely on the operating system to render text, so they are also unaffected. Beyond Oklahoma, there are some text rendering issues in South and Southeast Asia that may be related, but this is unclear until we know the root cause.

In the last couple weeks since this bug arose, I’ve reverted multiple attempts to remove the Osage name from the name=* tag. (The most recent revert was just an hour ago, so some maps will need time to catch up.) This is a good reminder that you should not tag for the renderer. Never remove valid information from the map purely due to presentational concerns like fonts. This withholds or loses relevant information, disadvantaging data consumers that are already behaving correctly and impairing our efforts to clean up coverage of indigenous names in the U.S.

I understand that the tofu is annoying to anyone trying to read other parts of the map many miles away. If you’re as bothered about the bug as I am, please help us find more examples of runaway tofu on the map, so we can get a better sense of the magnitude of the problem. And if you’re able, please help the developers of OSM Carto and Mapnik diagnose the underlying bug.

Failure to render combining mark unicode characters properly.

Thank you for your patience!

---

1. This link will get your IP address blocked for this particular URL. Copy the link and manually navigate to it instead. ↩︎

[Spiekerooger]

Can I get a short ELI5 why this is not happening here: OpenStreetMap Retina Tiles Demo (osm-carto fork). Is that because the letters in that Osage name are not shown/rendered combined the way they should be combined (this here is maybe easier to decipher: OpenStreetMap Larger Text Example )? So this is incorrect (but without the bug) as it does not use the correct Osage script?

Edit: i should just have looked at the unaffected map rendering examples like Shortbread or Americana as those show the same than the osm-fork linked above. So are those examples wrong in not correctly displaying the Osage script? (Even while I once live on the KS/OK border, I’m still not familiar with Osage script) - so the problem is likely with Mapnik and Noto Sans Osage Regular not working together? Do we have a real working rendering with Noto Sans Osage Regular somewhere?

[Minh_Nguyen (Minh Nguyễn)]

The problem that I’m calling attention to is that some Mapnik-based maps are not only showing tofu but also repeating it uncontrollably.

For reference, here’s how macOS’s native Core Text rendering engine presents 𐒹𐓘́͘𐓨𐓘͘𐓵𐓣͘ in Noto Sans Osage. As you can see, it’s just six letters with some combining diacritics:

If you don’t have that or an equivalent font installed, you’d instead see a row of boxes (“tofu”) or the hexadecimal codes from Apple’s Last Resort font, but still just six base characters. Earlier, I listed some layers like Humanitarian as being “unaffected” despite showing six tofu. That isn’t totally unexpected for a writing system that’s only 20 years old, in a Unicode block that’s only 10 years old.

MapLibre powers both OSM Americana and the Shortbread layer. On a recent version of macOS, which comes with Noto Sans Osage, MapLibre’s rendition is not quite accurate. It can only horizontally offset the combining nasal dot diacritic but cannot yet use the dedicated combined glyphs that the font provides for 𐓘͘ and 𐓣͘ (as seen in the screenshot to the left). As far as I know, this is only a minor typographical issue for Osage, though it affects South and Southeast Asian writing systems much more severely. I’ve prototyped a fix for the broader issue (the screenshot to the right):

In general, Mapnik does a fine job with complex writing systems, so I want to stress that this is not a criticism of Mapnik or OSM Carto overall. I think the relatively good support for text rendering is what makes the runaway tofu such a problem in Oklahoma: users aren’t accustomed to seeing OSM Carto behave so strangely, so they incorrectly assume that there’s a defect in the data. So far, five different mappers have made this mistake with Hominy, so I hope we can finally get to the bottom of it.

As far as we know, the latest release of Mapnik gets confused by the combining diacritics on this Osage name. Unlike the Osage letters it modifies, the nasal dot comes from a part of Unicode that Western fonts have supported for a long time. (It’s the same dot that appears in Lithuanian or traditionally in Irish.) When a mapper originally added this Osage name in 2023, they chose to use the nasal dot. Later on, when I added Osage names for other places in the Osage Nation, such as Pawhuska (𐓄𐓘𐓢𐓶𐓮𐓤𐓘), I didn’t see the nasal dot on signs for those places. I may have gotten lucky, or maybe the orthography has changed sometime over the past decade.

Fortunately, @ramseraph1 verified that the latest development version of Mapnik has fixed the bug. Once the fix gets released and the operations team upgrades to that release, 𐒹𐓘́͘𐓨𐓘͘𐓵𐓣͘ should return to being a small town with a short name.

[TheNightRider]

Wow, I just saw the tofu while looking for something in Pawnee County, OK, and my initial thought was it was vandalism. Then I searched OSM for “Hominy”, found this object and your note, which brought me here.

[Tex2002ans]

@Minh_Nguyen

For reference, here’s how macOS’s native Core Text rendering engine presents 𐒹𐓘́͘𐓨𐓘͘𐓵𐓣͘ in Noto Sans Osage. As you can see, it’s just six letters with some combining diacritics

Reminds me of this similar issue with Irish’s “dotted b” (ḃ) and “dotted d” (ḋ):

• Wikipedia.org: “Irish Orthography” > Diacritics
• #205: Source Code Pro: “ḃ and ḋ (b & d with dot above)”

and when I wrote about it back in:

• 2021: Mobileread.com: “free font with no dot on i”

where I covered quite a few other “weird” cases, like:

• the infamous “dotted and dotless I” (İ + ı) in Turkish
• various “hats” (circumflexes) being put on letters in Maths

So many fonts completely bug out when doing “unexpected” accent combinations, like the:

• b̂ = “b-hat”
  • SEE IMAGE of it in 2 different fonts.

So to see this “combining dot” combined with some really new/rare Unicode characters exploding the renderer, heh, I’m not surprised!

---

Side Note #1: For those of you interested in more of this kind of stuff, also see my posts back in:

• 2022: Mobileread.com: “Two characters look like one in Sigil, but wrong on the e-reader”
  • Combining Accents going “above” and “below” letters too.

(And in many cases: “Above” is fine. “Below” is fine. “Above AND Below” = explodes!!!)

---

Side Note #2: If you found that fascinating, you may also be interested in this absolutely fantastic talk:

• GOTO 2023: “Plain Text” by Dylan Beattie

That covers “mojibake”, which is when complete gibberish (or “blank”/“missing” squares) show up on the other end because of bad encoding.

So when you get this “missing character” � appearing, or:

• — = — = EM DASH
• ’ = ’ = RIGHT SINGLE QUOTE
• “ = “ = LEFT DOUBLE QUOTE
• ” = â€ = RIGHT DOUBLE QUOTE

You know you have a potential mess on your hands!