Migrating content from old forums

Es hat nicht vermutlich viel Sinn, diese Forderungen in die unterschiedlichsten Topics einzustreuen… das Migrationsthema wird hier behandelt, das Thema der Editiermöglichkeiten hier. Dort kannst Du Deine Wünsche zielgenau platzieren … :upside_down_face:.

@Mammi71 vielleicht die Migrationsbeiträge von hier in das entsprechende Topic verschieben?

4 Likes

Manchmal ändern sich z.B. Links oder die Überschrift stellt sich als missverständlich heraus oder der Beitrag bedarf einer sonstigen Aktualisierung. Hierfür kann zwar immer wieder eine neue Antwort verfasst werden, aber das macht die Sache sehr schnell unübersichtlich, da im Zweifel das gesamte Thema durchgelesen werden muss, statt nur der erste, aktualisierte, Beitrag.

1 Like

Wellicht als je weet wat je (nieuwe) planning is, deze communiceren.
Nu is het wazig allemaal.

1 Like

The OSMF has contracted a Ruby dev to help me with the FluxBB → Discourse migration. Our work together has started this week. He was able to get the first import running locally today.

I’ll post updates as we make progress.

8 Likes

https://wiki.osmfoundation.org/wiki/Board/Minutes/2022-11#2022/Res29_Pay_Harry_Wood_for_migration_to_Discourse

4 Likes

Raised this in teh Site Design thread, but it was suggested that it would be better here, so reposting.

Was reading that thread for the first time, & saw this post: Site design and organization - #8 by cquest. I know that it was only a very old example of how things could be migrated, but I noticed the old OSM Communities listed for both Australia & New Zealand.

How would they then integrate with the new “Oceania” community that is just in the process of being created? Would everything from both old spots go together into the new one?

Thanks

Yes, that should be possible, the same way some LATAM countries have opted to get their post into the LATAM category.

4 Likes

Progress update:

  1. We now have a smaller sample dataset used for repeated fluxbb → discourse imports. A full import test can take upto 22 hours!
  2. Work is progressing on improving the BBcode (FluxBB) to Markdown (Discourse) conversion code. Code under active development. Example PR upstream
  3. We will use a dedicated server for full import test runs. Server will be accessible in a few days.

We do not yet have a definite date for the conversion / import work to be completed.

12 Likes

Is this really needed? Discourse can also handle BBcode… this is text in [b]...[/b]

Discourse handles some some bbcode yes. Actually this caused me some confusion as I was first looking at the import results! Some of that is useful (The “quote” and “code” tags for example will be allowed through unmodified) but ultimately the import needs to convert almost all the different syntax of FluxBB-flavoured bbcode over to discourse-flavoured markdown/html. There’s a few nasty edge cases and nesting possibilities to consider

…but I think I’m nearly there with it.

11 Likes

Has any progress been made since?

@Harry_Wood Got the bbcode converter into a good state, the changes are in Pull Request upstream in: Minor lint fixes and typos by harry-wood · Pull Request #4 · nlalonde/ruby-bbcode-to-md · GitHub WIP Assorted tests and fixes encountered with FluxBB by harry-wood · Pull Request #6 · nlalonde/ruby-bbcode-to-md · GitHub

Harry also added tests and improved the importer, adding permalink support to allow the old forum links to still work once content migrated. Pull Requests here: Add a test for create_post in import scripts by harry-wood · Pull Request #18893 · discourse/discourse · GitHub and WIP import script bbcode fixes in Fluxbb importer + some fixes to all importers by harry-wood · Pull Request #18953 · discourse/discourse · GitHub

On my backlog tasks is to run a full import into a cloned version of this site to test the importer and test integrations. I have not yet completed this task, but I estimate before end of November.

11 Likes

Thanks for all that work! Would the cloned version be made available to the public for some time, so interested parties can take a look (e.g. how their regional forum was handled etc) and provide feedback if needed?

4 Likes

Any news on that?

1 Like

@nukeador : I’m really disappointed that our moderators are being pushed by the forum governance team to re-confirm the moderator selection in a super short time period, whereas the migration is now ongoing for months without much obvious progress to outsiders not watching any github repos. People on the German subforum are increasingly frustrated by the lack of any feedback wrt the current status of the migration process.

8 Likes

@mmd Nobody is being pushed, the community agreed on a timeline to review moderators on categories that were created before the policy was established, because they were simply self-selected. The agreement was to review in 3 month, December 10th (even some people flagged they needed to be reviewed even sooner).

That’s what’s happening right now, each community in this situation is self-managing to validate their moderators list.

On the migration I can’t comment since I’m not directly involved. I agree it’s going slow, the reason being not having enough people to tackle the issues arising with the content migration. @Firefishy can expand on this one, but it seems it’s being way trickier than expected to get it right.

Preview of the site is now available: https://forum-import-test.openstreetmap.org/
The preview site does NOT allow login.

NOTE: So far only around 25% of the old forum has been imported. The importer is still running and expected to finish Friday 6th Jan 2023 15:00 UTC.

The forum and discourse snapshots used are from 19th December 2022.

This is NOT the final import test, bugs will be fixed and additional import runs will be done.

There are a few known bugs:

  1. The [url]...[/url] syntax has not been converted cleanly
  2. Many users have not been merged with existing accounts, this needs further investigation
  3. 1000s of users imported, but with disabled accounts due to missing email in fluxbb database.
  4. Avatar images are currently not migrated from fluxbb. Feature currently missing from importer.
  5. Any more? Minor styling issues will likely not be addressed.
9 Likes

Crosschecking some topics with the old forum, everything looks as it should be (besides the [url] tag, but that’s ok for now).
I don’t remember if it was discussed already, but will the users’ avatars be transferred aswell?

2 Likes

The importer currently does not transfer the avatar images, I will add it to the list of known issues.

1 Like

If automated merging of users doesn’t work, I think you could at least merge them manually upon user request. I think that’s still not too bad.

Example to try out could be my second user maxheight in DE - #6 by mmd1 - users: Germany - OpenStreetMap Community