Migrating content from old forums

@Harry_Wood Got the bbcode converter into a good state, the changes are in Pull Request upstream in: Minor lint fixes and typos by harry-wood · Pull Request #4 · nlalonde/ruby-bbcode-to-md · GitHub WIP Assorted tests and fixes encountered with FluxBB by harry-wood · Pull Request #6 · nlalonde/ruby-bbcode-to-md · GitHub

Harry also added tests and improved the importer, adding permalink support to allow the old forum links to still work once content migrated. Pull Requests here: Add a test for create_post in import scripts by harry-wood · Pull Request #18893 · discourse/discourse · GitHub and WIP import script bbcode fixes in Fluxbb importer + some fixes to all importers by harry-wood · Pull Request #18953 · discourse/discourse · GitHub

On my backlog tasks is to run a full import into a cloned version of this site to test the importer and test integrations. I have not yet completed this task, but I estimate before end of November.

11 Likes

Thanks for all that work! Would the cloned version be made available to the public for some time, so interested parties can take a look (e.g. how their regional forum was handled etc) and provide feedback if needed?

4 Likes

Any news on that?

1 Like

@nukeador : I’m really disappointed that our moderators are being pushed by the forum governance team to re-confirm the moderator selection in a super short time period, whereas the migration is now ongoing for months without much obvious progress to outsiders not watching any github repos. People on the German subforum are increasingly frustrated by the lack of any feedback wrt the current status of the migration process.

8 Likes

@mmd Nobody is being pushed, the community agreed on a timeline to review moderators on categories that were created before the policy was established, because they were simply self-selected. The agreement was to review in 3 month, December 10th (even some people flagged they needed to be reviewed even sooner).

That’s what’s happening right now, each community in this situation is self-managing to validate their moderators list.

On the migration I can’t comment since I’m not directly involved. I agree it’s going slow, the reason being not having enough people to tackle the issues arising with the content migration. @Firefishy can expand on this one, but it seems it’s being way trickier than expected to get it right.

Preview of the site is now available: https://forum-import-test.openstreetmap.org/
The preview site does NOT allow login.

NOTE: So far only around 25% of the old forum has been imported. The importer is still running and expected to finish Friday 6th Jan 2023 15:00 UTC.

The forum and discourse snapshots used are from 19th December 2022.

This is NOT the final import test, bugs will be fixed and additional import runs will be done.

There are a few known bugs:

  1. The [url]...[/url] syntax has not been converted cleanly
  2. Many users have not been merged with existing accounts, this needs further investigation
  3. 1000s of users imported, but with disabled accounts due to missing email in fluxbb database.
  4. Avatar images are currently not migrated from fluxbb. Feature currently missing from importer.
  5. Any more? Minor styling issues will likely not be addressed.
9 Likes

Crosschecking some topics with the old forum, everything looks as it should be (besides the [url] tag, but that’s ok for now).
I don’t remember if it was discussed already, but will the users’ avatars be transferred aswell?

2 Likes

The importer currently does not transfer the avatar images, I will add it to the list of known issues.

1 Like

If automated merging of users doesn’t work, I think you could at least merge them manually upon user request. I think that’s still not too bad.

Example to try out could be my second user maxheight in DE - #6 by mmd1 - users: Germany - OpenStreetMap Community

Should users edit their old accounts to make their migration easier?

Test run of importer has completed import of all posts.

The importer failed in the final step of creating permalinks (forum compatibility links).

seems like the conversion for links only works when they are given as [url=$LINK]$TEXT[/url], but not for [url]$LINK[/url], see for example

MTB-Relationen gelöscht im Raum Bad Ischl / Hallstatt - users: Austria - OpenStreetMap Community

edit: oh, sorry - just saw that you already mentioned that

Thank you, @Firefishy. Is it intentional that the country specific categories from the old forum are imported under a whole separate category structure than the country specific categories we are already using here? If so, that seems confusing. For example this new United States category vs this old United States category. Maybe the plan is to do some reorganization after importing?

2 Likes

Yes, the plan is to manually fix the duplicate categories after the import.

1 Like

I am resetting the test site now and running another import. Import will complete late Saturday.

Excellent!

Test re-import started again and this time the bbcode to md conversion is looking a lot better.

3% of test import done. Will complete Saturday late afternoon UTC.

@Firefishy

A silver lining on the horizon! :+1: Thank you very much for sharing the current state of work!

Question for me to understand:
is there a real conversion of the posts or are there adjustments on the frontend? I am referring to these BB codes here.

About the errors mentioned…

Many users have not been merged with existing accounts, this needs further investigation

This can sometimes be due to the upper-lower case of the user name, or users who write little or rarely. These can only be “parked” using the old login data until the user logs in for the first time.

1000s of users imported, but with disabled accounts due to missing email in fluxbb database.

There is no way around treating them as a kind of “unregistered user” until a user logs in with the same username as the saved one. Then there is a question as to whether the newly registered user “xy” is the same as the user of the found contributions and whether one would like to have them assigned. Here you come to the point that you also have to rely a little on trust. And if not, then not.

Translated with DeepL Translate: The world's most accurate translator (free version)

DE-Text-Version:
Ein Silberstreif am Horizont! Sehr Vielen Dank für die Mitteilung des aktuellen Arbeitsstands!

Frage zum Verständnis für mich:
findet eine echte Konvertierung der Beiträge statt oder finden Anpassungen am Frontend statt? Ich meine hier diese BB-Codes.

Zu den genannten Fehlern…

Many users have not been merged with existing accounts, this needs further investigation

Das kann manchmal an der Groß-Kleinschreibung des Usernamens liegen, oder sind User, die wenig, bis selten schreiben. Diese können nur anhand der alten Anmeldedaten “geparkt” werden, bis sich der User das erste mal neu anmeldet.

1000s of users imported, but with disabled accounts due to missing email in fluxbb database.

Da wird man nicht umhin kommen, diese, als eine Art “nicht registrierte User” zu behandeln, bis eine Useranmeldung erfolgt, die den selben Usernamen hat, wie der gespeicherte. Dann eine Frage, ob der neu angemeldete User “xy” gleich dem User der gefundenen Beiträge ist und man sich diese zuordnen lassen möchte. Hier kommt man an den Punkt, daß man auch ein stückweit auf Vertrauen setzen muß. Na und wenn nicht, dann nicht.


Greetings, Sven

1 Like

@Firefishy Seems there is an issue now with [quote=xxx]…[/quote] if those codes are inline and not in a separate line
See:

[url]…[\url] seems to be fine now.

1 Like

Hi, nice to see progress here.

The old smilies are not converted properly, eg. the cool sunglasses smilie is converted to a blue square with word “cool” in it.

grafik

2 Likes