Migrating content from old forums

[Firefishy]

@Firefishy

5. Ouch. Not good enough, message is changed and meaning could likely be changed. It is due to over eager regex, likely very difficult to fix. Here is pseudocode version: gist:88c45733449078f8fa061838f0eb899a · GitHub

I have just reset the test import site again. I am now running a new test import with a fix for above.

The test import run should finish in around 20 hours time.

[Firefishy]

@Zverik

5. old → new: multi-level quoting is broken: text was stashed into the last quote.

5. Now Fixed Вопросы по JOSM - #1179 by evgenykatyshev - users: Russia - Test Import Site - OpenStreetMap Community

[Firefishy]

Reset again… Another test import now running with feature complete permalink (old forum links) code.

[Firefishy]

Topic Permalinks (old forum links) code confirmed working.

Reset again… Likely final changes: fixes for list items (list item inner formatting and fix for some <ul> incorrectly being converted to <ol>

Import should complete in around 12 hours.

[Firefishy]

@Zverik

One thing I see that’s wrong is links did not get made into links, they stay markdown-ed: Откаты правок - users: Russia - Test Import Site - OpenStreetMap Community I also found double-asterisk for bold that didn’t make the text bold.

Fixed: Откаты правок - users: Russia - Test Import Site - OpenStreetMap Community

Thanks to @Harry_Wood

[Firefishy]

I think we’re nearly there, final? test import run done: https://forum-import-test.openstreetmap.org/

I have also enabled logins (use login with OSM button at login) and check your imported message history.

The FluxBB Markdown (forum.osm.org) content conversion looking very good now.

Thank you to @TomH and @Harry_Wood for help fixing importer issues.

Now need to work out the steps for the real import. The import will likely mean the community site will be offline / read-only for a bit. I will properly schedule and announce the outage with at least a few days notice.

[Zverik (Ilya Zverev)]

Thank you Grant and Harry and Tom for fixing the prominent issues! I’ve posted the update to the Russian forum.

On a side note, do we keep the imported threads separate, or merge them with the already existing regional forums?

[Firefishy]

@Zverik

On a side note, do we keep the imported threads separate, or merge them with the already existing regional forums?

Post import the suitable categories (eg: regional forums) will be merged into existing community.osm.org categories.

[Firefishy]

@Matija_Nalis

It is already destroyed in HTML output of old forum, i.e. all those ? chars are real 0x3f ASCII chars, …
However, in my experience mysql itself often keeps enough information to reconstruct such messages (usually by doing the dump to a file, editing a file and changing CHARSET=xxxx where needed (utf8? utf8mb4 ?), and then importing again) - it is often connecting client that mangles it (if one sets use names to same thing as the database/table/column uses, one can usually get data out in raw form, which can then be converted. If client however does not issue correct commands on startup, it will get gibberish or ?).

I have tried, but doesn’t look like anything is recoverable. The old forum used to be hosted by an individual in the OpenStreetMap NL community. Many years ago the individual walked away from OSM without handing anything over. As part of the OSM.org operations team I was later able by brute force to get a dump of the database and re-host the site, but the unicode data loss likely happened then (bad dump?) or may have happened before then.

[daganzdaanda]

@Firefishy @TomH @Harry_Wood and everybody else who may be involved: Thank you VERY much for the immense amount of work!

The login works smoothly, and I think the posting history is preserved well. All the posts I checked looked very fine.

Will links to the old forum be changed to point to the new url, or will the old link be preserved, and then redirected to the new address?
We will need a redirect anyway for links from “outside”, but having the links inside the new community going directly to the new address will probably be a smoother user experience.

[Firefishy]

@daganzdaanda

Will links to the old forum be changed to point to the new url, or will the old link be preserved, and then redirected to the new address?
We will need a redirect anyway for links from “outside”, but having the links inside the new community going directly to the new address will probably be a smoother user experience.

All old forum links will redirect to the imported content.

The redirect is as follows

1. https://forum.openstreetmap.org/(.*) → https://community.openstreetmap.org/$1 (note the regex .* and $1)
2. community.openstreetmap.org will redirect to the correct content (importer populates the discourse permalink data)

You can test it yourself, find any URL on the old forum eg:

1. https://forum.openstreetmap.org/viewtopic.php?pid=871339
2. Take the url path + parameter /viewtopic.php?pid=871339
3. Prefix it with https://forum-import-test.openstreetmap.org eg: https://forum-import-test.openstreetmap.org/viewtopic.php?pid=871339
4. That URL will redirect to the imported content.

[Nadjita]

Went over it just this morning and I noticed that [...] or […] inside a quote is missing the opening [ all the time. We use that to point out that text was left out. Other than that, everything I was testing was looking great and worked perfectly

[Firefishy]

Could you link to examples? Old and new.

[Nadjita]

Sure:

In this thread, it appears very often. The corresponding thread in the old forums is this one.

vs

They are used frequently, but I only picked this one thread, because there’s so many in it.

[Wynndale]

Have you looked on the Wayback Machine?

[kmpoppe (Kai Poppe)]

@Firefishy

https://forum.openstreetmap.org/(.*) → https://community.openstreetmap.org/$1 (note the regex .* and $1)

Works great

[Minh_Nguyen (Minh Nguyễn)]

@Firefishy

Here is an example post: Как обозначать? (Page 5) / users: Russia / OpenStreetMap Forum
I don’t think the text can be recovered, but happy for others to try.

@Wynndale

Have you looked on the Wayback Machine?

Unfortunately, the corrupted characters were already present in this example thread by the time the Wayback Machine first indexed it in 2016.

[kmpoppe (Kai Poppe)]

@Firefishy, I just crawled the Wiki. There are 1089 links to the forum, of which 717 are unique AND start with http(s?)://forum.openstreetmap.org (there are some weird Google Translate links, I’ll have to look at).

Of those 717 links, 36 do not redirect, when the part after .org/ is put to the tester link.

Things you need not worry:

• http(s)://forum.openstreetmap.org(/index.php) ==> Yeah, that’s kind of expected
• all sorts of links that are not seperated correctly (like https://forum-import-test.openstreetmap.org/viewtopic.php?id=284||Newcastle) fixed
• Topics 19910, 999999 (probably a tester), 797497 are not present in the old forum fixed

Things I’d like you to take a look at:

• https://forum-import-test.openstreetmap.org/extern.php?action=feed&type=rss&fid=8, https://forum-import-test.openstreetmap.org/extern.php?action=feed&type=rss&fid=14, https://forum-import-test.openstreetmap.org/extern.php?fid=25&type=rss ==> I know for a fact, that you can get RSS feeds from Discourse, do you want to bother with redirecting those?
• There are 19 links where there’s a page-parameter attached (like https://forum-import-test.openstreetmap.org/viewtopic.php?id=68288&p=5) and those do not seem to get caught by the regex ==> do you want to include those in the match and just redirect to the topic as Discourse doesn’t have the page-concept at all?

HTH

Kai

[Firefishy]

I will add a workaround for the index.php issue.

Atom and RSS feeds, briefly looking into it, it looks quite complex to fix. I’ll likely add an ugly hack to point all old forum feeds to a static feed with a message asking the person to update their feed links. I don’t want burn a huge amount more time.

The &p= issue, I’ll create a regex which strips the page portion of the URL on redirect.

[kmpoppe (Kai Poppe)]

The forum... and forum.../index.php should redirect to the main page of Discourse, which I gathered would happen anyway when you shut down the forum completely.

For the RSS, I’m not too fussed, that’s like three pages and I bet nobody clicked on that link in a year or two

The old forum’s page size was 25, so you could always redirect to t/nnn/<topic-slack>/<(&p=x-1)*25+1>/ to get to the first post for that “page” (should you happen to suffer from boredom )