Migrating content from old forums

I have just reset the test import site again. I am now running a new test import with a fix for above.

The test import run should finish in around 20 hours time.

1 Like
  1. Now Fixed https://forum-import-test.openstreetmap.org/t/josm/42662/1179
5 Likes

Reset again… Another test import now running with feature complete permalink (old forum links) code.

3 Likes

Topic Permalinks (old forum links) code confirmed working.

Reset again… Likely final changes: fixes for list items (list item inner formatting and fix for some <ul> incorrectly being converted to <ol>

Import should complete in around 12 hours.

3 Likes

Fixed: https://forum-import-test.openstreetmap.org/t/topic/48223

Thanks to @Harry_Wood

1 Like

I think we’re nearly there, final? test import run done: https://forum-import-test.openstreetmap.org/

I have also enabled logins (use login with OSM button at login) and check your imported message history.

The FluxBB Markdown (forum.osm.org) content conversion looking very good now.

Thank you to @TomH and @Harry_Wood for help fixing importer issues.

Now need to work out the steps for the real import. The import will likely mean the community site will be offline / read-only for a bit. I will properly schedule and announce the outage with at least a few days notice.

16 Likes

Thank you Grant and Harry and Tom for fixing the prominent issues! I’ve posted the update to the Russian forum.

On a side note, do we keep the imported threads separate, or merge them with the already existing regional forums?

1 Like

Post import the suitable categories (eg: regional forums) will be merged into existing community.osm.org categories.

3 Likes

I have tried, but doesn’t look like anything is recoverable. The old forum used to be hosted by an individual in the OpenStreetMap NL community. Many years ago the individual walked away from OSM without handing anything over. As part of the OSM.org operations team I was later able by brute force to get a dump of the database and re-host the site, but the unicode data loss likely happened then (bad dump?) or may have happened before then.

6 Likes

@Firefishy @TomH @Harry_Wood and everybody else who may be involved: Thank you VERY much for the immense amount of work!

The login works smoothly, and I think the posting history is preserved well. All the posts I checked looked very fine.

Will links to the old forum be changed to point to the new url, or will the old link be preserved, and then redirected to the new address?
We will need a redirect anyway for links from “outside”, but having the links inside the new community going directly to the new address will probably be a smoother user experience.

3 Likes

All old forum links will redirect to the imported content.

The redirect is as follows

  1. https://forum.openstreetmap.org/(.*)https://community.openstreetmap.org/$1 (note the regex .* and $1)
  2. community.openstreetmap.org will redirect to the correct content (importer populates the discourse permalink data)

You can test it yourself, find any URL on the old forum eg:

  1. https://forum.openstreetmap.org/viewtopic.php?pid=871339
  2. Take the url path + parameter /viewtopic.php?pid=871339
  3. Prefix it with https://forum-import-test.openstreetmap.org eg: https://forum-import-test.openstreetmap.org/viewtopic.php?pid=871339
  4. That URL will redirect to the imported content.
5 Likes

Went over it just this morning and I noticed that [...] or […] inside a quote is missing the opening [ all the time. We use that to point out that text was left out. Other than that, everything I was testing was looking great and worked perfectly :+1:

1 Like

Could you link to examples? Old and new.

Sure:

In this thread, it appears very often. The corresponding thread in the old forums is this one.

image

vs

image

They are used frequently, but I only picked this one thread, because there’s so many in it.

3 Likes

Have you looked on the Wayback Machine?

Works great :slight_smile:

1 Like

Unfortunately, the corrupted characters were already present in this example thread by the time the Wayback Machine first indexed it in 2016.

3 Likes

@Firefishy, I just crawled the Wiki. There are 1089 links to the forum, of which 717 are unique AND start with http(s?)://forum.openstreetmap.org (there are some weird Google Translate links, I’ll have to look at).

Of those 717 links, 36 do not redirect, when the part after .org/ is put to the tester link.

Things you need not worry:

Things I’d like you to take a look at:

HTH :slight_smile:

Kai

5 Likes

I will add a workaround for the index.php issue.

Atom and RSS feeds, briefly looking into it, it looks quite complex to fix. I’ll likely add an ugly hack to point all old forum feeds to a static feed with a message asking the person to update their feed links. I don’t want burn a huge amount more time.

The &p= issue, I’ll create a regex which strips the page portion of the URL on redirect.

4 Likes

The forum... and forum.../index.php should redirect to the main page of Discourse, which I gathered would happen anyway when you shut down the forum completely.

For the RSS, I’m not too fussed, that’s like three pages and I bet nobody clicked on that link in a year or two :wink:

The old forum’s page size was 25, so you could always redirect to t/nnn/<topic-slack>/<(&p=x-1)*25+1>/ to get to the first post for that “page” (should you happen to suffer from boredom :rofl:)